hg pull runs out of memory

[Hertroys A.]

> On Wed, 2012-04-25 at 14:40 +0200, Hertroys A. wrote:
> > Or perhaps I need more memory than our (mainly 32-bit) systems can
> > address, in which case my options are limited to a few production
> > servers :(
> 
> Indeed. When running a 32-bit version of Mercurial, the effective file
> size limit is about 400MB, no matter how much memory you throw at the
> machine.
> 
> Start with 4G of address space (not RAM!). Divide by 2: 2G for the
> kernel and 2G for the application. Now that 2G is already chopped up
> into pieces by where the kernel decides to place the application and all
> its libraries in memory. You're lucky at this point if you can get a
> single 400MB piece.. and Mercurial needs a few.
> 
> No amount of additional RAM or swap will help.

Thanks for confirming that, I feared the problem would be something like that.
 
> > Why does Mercurial need that much memory to pull in a binary file
> > anyway? There doesn't seem to be any benefit in keeping it in memory.
> > Can that be prevented somehow, perhaps (after they've been added to
> > the repo, obviously)?
> 
> Because Mercurial is designed for dealing with _source code_ quickly.
> And it's massively more efficient to deal with small files typical of
> source code by reading, writing, and calculating deltas on them in
> memory.

Of course, but isn't it kind of dumb to attempt the same with a large binary file?

I'm well aware that having two distinct methods for dealing with files has its own share of problems, but when the only method (as it is now) isn't capable of doing the job for some files (thankfully it aborts gracefully, that's _not_ obvious), isn't that worse?

I _am_ grateful for Mercurial, it works like a charm for pretty much everything. Even when it breaks, it tends to do so gracefully by rolling back the transaction (neat!). Yet, this is a stain on its reputation.

Compare that to my experiences with earlier versions of Subversion that just threw a segfault during a fresh import of an existing project. No rollback, just a repo that contained many of the files that needed to be in it, and many that weren't. It took days to fix the mess it created!

Alban.

 
alban.hertroys at apollovredestein.com
T: 

Apollo Vredestein B.V. - P.O. Box 27 - 7500 AA Enschede - The Netherlands - Chamber of Commerce number 34223268 - http://www.apollovredestein.com

The information contained in this e-mail is intended solely for the use of the individual or entity to whom it is addressed and others authorized to receive it. You are hereby notified that any disclosure, copying, distribution or action in relation to the contents of this information is strictly prohibited. If you are not the intended recipient, please delete this message and any attachments and advise the sender by return e-mail. The confidentiality of this message is not warranted. Apollo Vredestein B.V. rules out any and every liability resulting from this or any other electronic transmission.
--------------------------------------------------------------------------