hg push of large changeset fails to Nginx with auth configured

[Augie Fackler]

On Thu, Aug 18, 2011 at 2:29 PM, Kyrra <kyrra at nlworks.com> wrote:
> I'm seeing what may be an issue with mercurial and how it does
> authentication with a push.
> hg version: 1.9.1 on client and server
> nginx version: 1.0.5
> server: ubuntu 10.04 (running python 2.6.5)
> client: windows 7 (using hg command line and tortoisehg)
>
> I have hg serve running behind nginx on my server.  nginx is
> configured with auth_basic, which then uses the proxy mechanism to
> forward the request to hg serve.
>
> It's worked well for me and a few friends for a couple months now, but
> we just hit a stumbling block.  When doing a large commit it kept
> timing out and wouldn't let my friend upload.  We ended up taking
> nginx out and letting him commit directly to hg serve and all worked
> well.  I've been doing some testing sine then to see if I could find
> the cause (or at least narrow it down).
>
> Enabling or disabling ssl on nginx made no difference.  I also messed
> with every timeout and buffer value I could in nginx, but that also
> was not the cause.  I ended up removing the auth_basic from nginx and
> uploads of large files started working just fine.
>
> So over http, I pushed a small and large change, captured the
> transfers in wireshark and compared to see what was happening.
>
> Small commit:
> 1) client opens TCP stream to server
> 2) client issues a get: "GET /testtest?cmd=batch HTTP/1.1"  with auth
> details included.
> 3) server responds (200 status) with result.
> 4) client http POST issued (no auth details included)
> 5) server responds with 401
> 6) client http POST completes upload
> 7) that TCP stream ends (close by client with RST, ACK
>
> 8) client starts new TCP stream
> 9) client sends http POST (with auth details)
> 10) client finishes POST request.
> 11) server responds with 200.  All is good.
>
>
> Large commit:
> 1) client opens TCP stream to server
> 2) client issues a: "GET /testtest?cmd=branchmap HTTP/1.1" with auth
> details included.
> 3) server responds (200 status) with results
> 4) client issues a POST (no auth details included) (includes about 4
> packets worth of post data so far)
> 5) server responds with 401 (unauthorized).
> 6) client continues to send post data
> 7) after 30 seconds, the server sends [RST] to all outstanding packets
> and resets that TCP connection. (30 seconds was measured on the server
> by printing $request_time variable to the nginx logs, which showed
> 29.501 seconds, the wireshark timestamps agree with this).
>
> 8) as soon as the client saw the first [RST] packet, it started a new TCP stream
> 9) the client issued another POST (no auth details)
> 10) the server responded with 401 again after about 15 packets.
> 11) the client continued to send data for 30 seconds until the server
> closed the connect with a [RST] again.
> 12) hg push printed the following error message "abort: error: An
> existing connection was forcibly closed by the remote host"
>
>
> I would recommend that if the few gets that hg does before doing a
> push, if they require auth_basic to be sent that HG also sends
> auth_basic details with its post command.  Having to post the data
> twice because http auth fails seems like a waste to me.

Can you try enabling ui.usehttp2 and make sure you've got hg 1.9 or
later on the server? That should allow the server to send 401s early
and transmit less data.

>
>
> config files:
> =======hgweb.config=========
> [web]
> push_ssl=false
> allow_push=*
> [paths]
> testtest=/var/hg/repos/testtest
> =========================
>
>
> =======nginx.conf (relevant sections)=========
> server {
>  listen      80;
>  server_name [hidden server name];
>
>  auth_basic           "Restricted";
>  auth_basic_user_file /var/hg/htpasswd;
>
>  location / {
>    expires epoch;
>    include /etc/nginx/proxy.conf;
>    proxy_pass http://localhost:8000;
>  }
> }
> =====================================
>
>
> =======proxy.conf=========
> proxy_redirect off;
> proxy_set_header Host $host;
> proxy_set_header X-Host $http_host;
> proxy_set_header X-Real-IP $remote_addr;
> proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
> proxy_set_header Proxy-host $proxy_host;
> client_max_body_size 400m;
> client_body_buffer_size 128k;
> proxy_buffering off;
> proxy_connect_timeout 3600;
> proxy_send_timeout 3600;
> proxy_read_timeout 3600;
> proxy_buffer_size 8k;
> proxy_buffers 8 32k;
> proxy_busy_buffers_size 64k;
> proxy_temp_file_write_size 64k;
> =====================================
>
> Thanks,
> -James
> _______________________________________________
> Mercurial mailing list
> Mercurial at selenic.com
> http://selenic.com/mailman/listinfo/mercurial
>