Network performance problems when pulling and cloning from HTTP server

[Augie Fackler]

On Nov 22, 2012, at 7:03 AM, Angel Ezquerra <ezquerra at gmail.com> wrote:

> On Wed, Nov 21, 2012 at 10:24 PM, Matt Mackall <mpm at selenic.com> wrote:
>> On Wed, 2012-11-21 at 08:02 +0100, Angel Ezquerra wrote:
>>> On Nov 20, 2012 6:55 PM, "Bryan O'Sullivan" <bos at serpentine.com> wrote:
>>>> 
>>>> On Tue, Nov 20, 2012 at 8:30 AM, Angel Ezquerra <ezquerra at gmail.com> wrote:
> 
> Matt, thanks for pitching in.
> 
> I understand now the need to get the hashes, usernames, dates and
> descriptions. As for also getting the diffs, that would only happen
> when using the --bundle option, right? That is, calling a plain "hg
> incoming" (without --bundle) should be pretty fast regardless of the
> contents of the changesets themselves. Is that correct?

Sadly no, incoming pulls a complete bundle, then throws it on the floor when done. In fact, 'hg help incoming' tells you that '--bundle FILE' lets you save the requested bundle to a file so you can apply it later.

> 
>>> In any case, may I ask why you are not surprised by this?
>> 
>> You've hit the trifecta of ways to have suboptimal performance:
>> 
>> a) Windows
>> b) large files
>> c) using a protocol designed for broadband and slower on a LAN
> 
> By protocol you mean the underlying http or mercurial's own protocol?

hg's own protocol, which can use http (or ssh) as a transport.


>>>> Do you have the ability to serve the repo that shows poor performance using "hg serve"? If so, it would be helpful to "hg serve --profile", do a pull from a client, then stop the server and share the profile dump.
>>> C:\mercurial_tests\tmp_bigfiles_not_large>hg serve --port 7000
>>> --profile --verbose
>> 
>> ..no profile. A profile looks like this:
>> 
>> $ hg serve --profile
>> listening at http://calx:8000/ (bound to *:8000)
>> [do something]
>> [hit ctrl-c]
>>   CallCount    Recursive     Total(s)    Inline(s) module:lineno(function)
>>           4            0      1.5059      1.5058   <select.select>
>>          57           46      0.0052      0.0036   <__import__>
>>           1            0      0.0025      0.0013   mimetypes:205(readfp)
>>         684            0      0.0006      0.0005   mercurial.config:20(__setitem__)
>>         665            0      0.0007      0.0005   mimetypes:78(add_type)
>>           2            0      0.0004      0.0004   <_socket.gethostbyaddr>
>>          80            0      0.0008      0.0003   mercurial.config:27(update)
>>         971            0      0.0003      0.0003   <method 'split' of 'str' objects>
>> 
> 
> Sorry, I did not know that.
> 
> Unfortunately, I cannot get --profile to work. Whenever I use Ctrl+C
> to stop the server it just exits without printing any profile
> information.
> 
> I am running this test server on our production mercurial (windows
> 2003) server, which is already using the default port. If I do:
> 
> hg server --profile
> 
> without specifying a port, the server exists immediately, saying
> "abort: cannot start server at ':8000'", and in that case it _does_
> print some profile information. However, it does not print anything if
> I set a port to something other than 8000 and then I use Ctrl+C.
> 
> I thought that perhaps --profile did not work when the --port is set.
> To verify this I stopped our production server tonight and tried again
> to run "hg server --profile" (without setting the port). The result is
> the same (i.e. no profile information is printed when I do Ctrl+C). I
> tried killing the hg process using the windows task manager but I get
> the same result (no profile printed).
> 
> What else could I do?

I just verified that ^C gives me profiler output on a Mac. I don't know enough about Windows to guess why this isn't working though.


The fact that archives are fast and pulls are slow suggests (to me) that computing deltas is being expensive for your use case, but that's only intuition talking. Figuring out a way to get profiler output would be the best outcome.

One thing that *might* give us an idea where you'd be burning some time would be 'hg bundle --all --profile foo.bundle'. That's not exactly the same as the web server, but it might show something surprising (and therefore be helpful without me having to guess about Windows things.)

> 
> Thank you for your help. I really appreciate it.
> 
> Angel
> _______________________________________________
> Mercurial mailing list
> Mercurial at selenic.com
> http://selenic.com/mailman/listinfo/mercurial