Mercurial problem with large binary changesets

[Matt Mackall]

On Tue, May 29, 2007 at 03:06:49PM -0400, Francois-Denis Gonthier wrote:
> ----- KRYPTIVA PACKAGED MESSAGE -----
> PACKAGING TYPE: SIGNED
> Thank you for your scripts and tools, I was able to see the precise
> memory consumption of hg at the moment of the crash:
> 
> hg log:
> 2007-05-29 17:22:20.278055 - getting bigasslotustemplate.ntf
> 2007-05-29 17:22:20.490091 - ** unknown exception encountered, details
> follow
> 
> ps log:
> 2007-05-29 17:22:20.501208 -   PID    VSZ
> 2007-05-29 17:22:20.501705 - 22899 454572
> 
> This seem to confirm that the Lotus Notes template is the culprit.  The
> memory consumption is a more reasonable >400mb but that is still a lot
> in my mind.  The target is system has 128mb RAM, and less that 400mb
> swap, which explain why it crashes... but that is still a big memory
> consumption.
> 
> I might add that everybody in the office runs Mercurial 0.9.3.  I've
> been on #mercurial on Freenode to ask for help about this problem and
> got told to check my mercurial versions and come back.  It was once the
> case that the target system is outdated but this is no longer the case.
> 
> I'm open to any solution, as weird as they may be.

It's possible that if your template contains a huge number of LF
characters, memory usage could be especially ugly. The delta algorithm
is "line-based", even for "binary" files. Can you count them?

Alternately, you can hack mercurial/bdiff.c to report how big its
various mallocs are. Line 78 is the most likely culprit.

For big files (>1MB?), we should probably switch to a different delta
generation algorithm.

-- 
Mathematics is the supreme nostalgia of our time.