Slow push of large file over HTTP

[Martin Geisler]

Michael Tjørnemark <mtj at pfa.dk> writes:

Hejsa :)

> I have a repository with a single changeset which adds a single 60 MB
> file (zip-file). Pushing this repo over HTTP is much, much slower than
> other commands on the repository, including a similar pull - is this
> to be expected? I have recreated the problem on other machines and
> files as well, so it seems to be a general problem with pushing a
> large(ish) file.
>
> Times (all on my local machine):
> Commit file - 4 secs
> Push to empty repo using filesystem - 7 secs
> Clone from repo over HTTP - 23 secs
> Pull to empty repo over HTTP - 23 secs
> Push to empty repo over HTTP - 4 mins <-- SLOW
>
> Command to serve empty repo:
> hg serve --config web.allow_push=* --config web.push_ssl=false
>
> Command to push to empty repo:
> hg push http://localhost:8000/ --debug --time
>
> In the debug output (see below), bundling and sending takes around 10
> secs. Then there is a almost 4 min pause between "sending:
> 60342/120684 kb (50.00%)" and "remote: adding changesets". (Also it
> seems wrong that sending only goes to 50%, but that is another
> problem).

You've stumpled upon a weird corner case in Python's HTTP library. If
the server asks for authentication, then we'll only see this *after*
pushing the entire changegroup to the server! We'll then have to start
over. The progress code anticipates this and claims that you need to
send 120 MB for the 60 MB push so that the progress bar will go smoothly
from 0% to 100%. Here the push is really finished when it reaches 50%.

I'm unsure why it then stops for 4 minuttes -- I would not expect that.

> I understand that the largefiles extension might help, but this
> requires that everybody that uses the repo enables the extension, so i
> would rather avoid that. And also everything else is fast (commit,
> clone, pull), so it seems as if something is wrong with push over
> HTTP.

Versioning zip files is... unusual :) Every revision of the zip file
will take up a lot of new space since it can't be delta compressed much
against the previous version. So after 10 edits to the file, you could
end up with a repo with maybe 400 MB of history for that single file.

The largefiles extension sound like just what you need -- Unity is
actually is using it for versioning a lot of zip files.

-- 
Martin Geisler

aragost Trifork
Commercial Mercurial support
http://aragost.com/mercurial/