Mercurial problem with large binary changesets

[Matt Mackall]

On Wed, May 30, 2007 at 11:05:20AM -0400, Francois-Denis Gonthier wrote:
> ----- KRYPTIVA PACKAGED MESSAGE -----
> PACKAGING TYPE: SIGNED
>  > This will dump a copy of troublesome compressed chunk to disk. Then
>  > we can try to decompress it manually:
>  >
>  >>>> import zlib b = file("/tmp/busted-bin").read() a =
>  >>>> zlib.decompress(b) len(a)
>  > 688890
>  >>>> a[:50]
>  > '[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14,'
>  >
> 
>  >>> import zlib b = file("/tmp/busted-bin").read() a =
>  >>> zlib.decompress(b) len(a)
> 22130994

Okay, decompressed just fine. So that rules out integrity issues and
various other kinds of bugs.

>  >>> len(b)
> 1305692
>  >>> a[:50]
> '\x00\x00\x00\x00\x00\x00\x00\xdc\x00\x00\x00\xdc\x1a\x00\x00\x04\x00\x00+\x00\x00\x00\xa1ll\x00or%\x85\x00\x00P\x97\x00\x00^\x18\x00\x00\x1e\x00\x00\x00\x00\x00I\x1cw\x00'
>  >>>
> 
> at this point, the Python process consumes that much memory:
> 
> fdgonthier at nemo:~$ ps -o pid,vsz -p $(pgrep python)
>    PID    VSZ
> 26836  27772

And that's a reasonable number too. No idea how you're getting from
there to 400MB.

>From your earlier message:

hg log:
2007-05-29 17:22:20.278055 - getting bigasslotustemplate.ntf
2007-05-29 17:22:20.490091 - ** unknown exception encountered, details
follow

ps log:
2007-05-29 17:22:20.501208 -   PID    VSZ
2007-05-29 17:22:20.501705 - 22899 454572

What was the memory usage like right before getting the large file?

Your machine shouldn't be able to plow 400MB into swap in .2 seconds.
Do you have strict overcommit turned on in your kernel? (I assume this
is Linux?)

-- 
Mathematics is the supreme nostalgia of our time.