hg pull runs out of memory

[Matt Mackall]

On Thu, 2012-04-26 at 10:06 +0200, Hertroys A. wrote:
> > On Wed, 2012-04-25 at 14:40 +0200, Hertroys A. wrote:
> > > Or perhaps I need more memory than our (mainly 32-bit) systems can
> > > address, in which case my options are limited to a few production
> > > servers :(
> > 
> > Indeed. When running a 32-bit version of Mercurial, the effective file
> > size limit is about 400MB, no matter how much memory you throw at the
> > machine.
> > 
> > Start with 4G of address space (not RAM!). Divide by 2: 2G for the
> > kernel and 2G for the application. Now that 2G is already chopped up
> > into pieces by where the kernel decides to place the application and all
> > its libraries in memory. You're lucky at this point if you can get a
> > single 400MB piece.. and Mercurial needs a few.
> > 
> > No amount of additional RAM or swap will help.
> 
> Thanks for confirming that, I feared the problem would be something like that.
>  
> > > Why does Mercurial need that much memory to pull in a binary file
> > > anyway? There doesn't seem to be any benefit in keeping it in memory.
> > > Can that be prevented somehow, perhaps (after they've been added to
> > > the repo, obviously)?
> > 
> > Because Mercurial is designed for dealing with _source code_ quickly.
> > And it's massively more efficient to deal with small files typical of
> > source code by reading, writing, and calculating deltas on them in
> > memory.
> 
> Of course, but isn't it kind of dumb to attempt the same with a large
> binary file?

The engineering effort to change the situation is huge and the demand is
small. I'll consider fixing it for about $100k. That same $100k of
course will buy you lots of 64-bit systems. Until then, there are more
pressing matters.

-- 
Mathematics is the supreme nostalgia of our time.