Mercurial Crashing: Out of Memory

[Matt Mackall]

On Thu, 2012-09-27 at 13:53 -0400, Jensen, Aaron wrote:
> We have a Mercurial repository with some large database files (greater
> than 128MB in size).  We are running Mercurial 2.3.1 with TortoiseHg
> 2.5 on Windows 2008 with 4GB of RAM.  Recently, Mercurial has started
> crashing with out of memory errors when updating this repository. 

Just as a point of clarity, what you describe is not a 'crash', which is
an uncontrolled failure. Mercurial ran out of memory, detected the
problem, and bailed about as gracefully as a program can possibly be
expected to in such a situation. In other words, it didn't run into the
tree, it came to a stop in front of the tree and said "hey, there's a
tree in my way."

>  When I watch memory usage during the update, I see the hg.exe quickly
> balloon up to almost 3 GB of memory before crashing.  When I run
> update with the --debug flag, this is what I just before the crash:
> 
> 	updating: Common/AdmSiteAssociations.bcp 5/102 files (4.90%)
> 	getting Common/AdmSiteAssociations.bcp
> 	hg.exe : abort: out of memory

Does enabling a few gig of swap help?

> This file is 192 MB in this .hg\store\data directory, and 267 MB in the working directory.

I would expect that to work even on a 32-bit system (which has a maximum
of 2G of address space). Mercurial's delta algorithm chunks based on
linefeed bytes, does this file have lots of very short lines?

> Is this expected behavior?  Are my files too big?  If so, what is the
> limiting factor?  The internal size?  Or the working directory size?
> Is there a way I can help figure out where Mercurial is using memory
> and make things better?  Is there anything I can do to get my
> repository updated, short of installing extra RAM?

The limiting factor is that deltas chains are applied in memory for each
file. So you need enough memory to hold the base revision, all the
deltas, and the output revision in memory. That's typically 3x, but can
be much more if compression is involved.

-- 
Mathematics is the supreme nostalgia of our time.