Mercurial problem with large binary changesets

Tue May 29 19:06:49 UTC 2007

----- KRYPTIVA PACKAGED MESSAGE -----
PACKAGING TYPE: SIGNED
Thank you for your scripts and tools, I was able to see the precise
memory consumption of hg at the moment of the crash:

hg log:
2007-05-29 17:22:20.278055 - getting bigasslotustemplate.ntf
2007-05-29 17:22:20.490091 - ** unknown exception encountered, details
follow

ps log:
2007-05-29 17:22:20.501208 -   PID    VSZ
2007-05-29 17:22:20.501705 - 22899 454572

This seem to confirm that the Lotus Notes template is the culprit.  The
memory consumption is a more reasonable >400mb but that is still a lot
in my mind.  The target is system has 128mb RAM, and less that 400mb
swap, which explain why it crashes... but that is still a big memory
consumption.

I might add that everybody in the office runs Mercurial 0.9.3.  I've
been on #mercurial on Freenode to ask for help about this problem and
got told to check my mercurial versions and come back.  It was once the
case that the target system is outdated but this is no longer the case.

I'm open to any solution, as weird as they may be.

F-D

PS:
from another attempt, here is the complete stack trace:

pushing to ssh://adorais@nemo/repos/kpp-ln
searching for changes
remote: adding changesets
remote: adding manifests
remote: adding file changes
remote: ** unknown exception encountered, details follow
remote: ** report bug details to http://www.selenic.com/mercurial/bts
remote: ** or mercurial at selenic.com
remote: ** Mercurial Distributed SCM (version 0.9.3)
remote: Traceback (most recent call last):
remote:   File "/usr/bin/hg", line 12, in ?
remote:     commands.run()
remote:   File
"/var/lib/python-support/python2.4/mercurial/commands.py", line 3000, in run
remote:     sys.exit(dispatch(sys.argv[1:]))
remote:   File
"/var/lib/python-support/python2.4/mercurial/commands.py", line 3223, in
dispatch
remote:     return d()
remote:   File
"/var/lib/python-support/python2.4/mercurial/commands.py", line 3182, in
<lambda>
remote:     d = lambda: func(u, repo, *args, **cmdoptions)
remote:   File
"/var/lib/python-support/python2.4/mercurial/commands.py", line 2291, in
serve
remote:     s.serve_forever()
remote:   File
"/var/lib/python-support/python2.4/mercurial/sshserver.py", line 40, in
serve_forever
remote:     while self.serve_one(): pass
remote:   File
"/var/lib/python-support/python2.4/mercurial/sshserver.py", line 47, in
serve_one
remote:     if impl: impl()
remote:   File
"/var/lib/python-support/python2.4/mercurial/sshserver.py", line 194, in
do_unbundle
remote:     r = self.repo.addchangegroup(fp, 'serve', self.client_url())
remote:   File
"/var/lib/python-support/python2.4/mercurial/localrepo.py", line 1849,
in addchangegroup
remote:     if fl.addgroup(chunkiter, revmap, tr) is None:
remote:   File "/var/lib/python-support/python2.4/mercurial/revlog.py",
line 1183, in addgroup
remote:     text = self.revision(chain)
remote:   File "/var/lib/python-support/python2.4/mercurial/revlog.py",
line 919, in revision
remote:     bins.append(self.chunk(r, df=df))
remote:   File "/var/lib/python-support/python2.4/mercurial/revlog.py",
line 875, in chunk
remote:     return decompress(self.chunkcache[1][offset:offset + length])
remote:   File "/var/lib/python-support/python2.4/mercurial/revlog.py",
line 66, in decompress
remote:     if t == 'x': return zlib.decompress(bin)
remote: MemoryError
remote: transaction abort!
remote: rollback completed

Giorgos Keramidas wrote:
 >
 > Hmmm that's odd.  I have a local repository with moderately 'large'
 > files in the 14-20 MB range, and cloning it over SSH never exceeds
 > 100 MB of memory usage for Python here.  The Python process is
 > usually about 14-17 MB big.  It momentarily grows to 65-70 MB but
 > then grows smaller pretty fast...
 >
 > Can you run "hg --debug push" or "hg --debug clone" and collect its
 > output to a logfile?  If you can watch the memory size of Hg with a
 > script at the same time, it would be nice.
 >
 > To make it easier to correlate timestamps, you can filter the output
 > of hg itself through:
 >
 > http://people.freebsd.org/~keramida/bin-scripts/tslog
 >
 > which is a short Perl script:
 >
 > $ hg --debug clone ssh://host/path 2>&1 | tslog
 >
 > and in another terminal, you can keep watching the size of the python
 >  process with ps(1) or a similar tool, i.e. here's a FreeBSD-specific
 >  loop to print the virtual-size of all Python processes and filter
 > the output through `tslog' mentioned above:
 >
 > $ while true ; do \ ps -o pid,vsz \ -p $(echo $(pgrep python) |
 > expand | sed -e 's/ /,/g') ; \ done 2>&1 | tslog

----- KRYPTIVA SIGNED MESSAGE -----
This email claims to have been packaged by Kryptiva.
To process this email and authenticate its origin, get
the free plugin from:
http://www.kryptiva.com/downloads

----- KRYPTIVA SIGNATURE START -----
AvWVqAAAAAIAAAABAAAAAAAATiACAQAAAAC3AQAIAAAAAwAAAAECABS3FuvxegTt63v7UWCF
iSdtKgVqvAMAFNtYnPf/czq1EyEfvnKRBT4kXaMpBAAUPeBFYOVSOO06L2Fx5nv/SqsjQBQF
ABTaOaPuXmtLDTJVv++VYBiQr9gHCQYAFNcZh+doKHiXksUamShm9lxTMLNaBwAUPBs9vVQF
3OGCTeOm9OwZ2yp2HkQRABgAAAAAAABOIEZceckACVLdAAAAAAAAEJoTAAQAAAAAAAAAggQA
nu6v8X5Pd3BEU75lidCdGFliwiGYJLyovIKU9QwafNoLe8IQQYN8W81C1naDe4WQTc9pL9G2
hcMuDahgxPe+vx3/uuAPNjOi95kYA80d3wAJf3MN+ZMUXCX3xy4XOXd19wX019RZD+f/uJZL
R3ICkbdtMHfYUF1MVXzm0UCrpqA=
----- KRYPTIVA SIGNATURE END -----