Mercurial Crashing: Out of Memory

[Jensen, Aaron]

> Just as a point of clarity, what you describe is not a 'crash', which is an uncontrolled failure. 

Thank you, yes.


> Does enabling a few gig of swap help?

I'll talk to our operations group about that.  They manage the servers where we're getting RAM errors.


> I would expect that to work even on a 32-bit system (which has a maximum of 2G of address space). Mercurial's delta algorithm chunks based on linefeed 
> bytes, does this file have lots of very short lines?

Yes.  The files are exports of database tables.  Each line is a row.  This file has 1,814,885 rows, each between 115 and 158 bytes.  


> The limiting factor is that deltas chains are applied in memory for each file. So you need enough memory to hold the base revision, all the deltas, and the 
> output revision in memory. That's typically 3x, but can be much more if compression is involved.

This helps me understand why this file is causing issues.  We have larger files in this repository, but this file in particular compresses really well (it has lots of duplicate byte sequences) and huge chunks of it change almost daily, which explains why it likes to gobble up memory.

We definitely need to be smarter how we're handling/storing our data.  In the meantime, I'll try and get this table removed.

     <:> Aaron