Tricks for taming large repo manifests and/or improving large repo clone times?

[Lawrence Stewart]

On 1/6/19 1:10 pm, Lawrence Stewart wrote:
[snip]
> PS: As an aside, I've been unable to test --stream behaviour because
> something about my nginx + uwsgi setup breaks --stream mode:
> 
> % hg clone --stream -U http://hg-beta.freebsd.org/base freebsd_base_hg
> streaming all changes
> abort: HTTP Error 502: Bad Gateway
> 
> I spent a little bit of time trying to figure out how to fix --stream
> mode but my Google fu has failed me thus far... any clues would be
> appreciated.

Turns out this was a filesystem permissions problem... oops. uwsgi on
the server was unable to write a lock file to freebsd_base/.hg/store/lock

To round out results, --stream clone times are at the end of this email.

The low latency network --stream clone time is very nice indeed, but I
suspect relying on --stream to deliver goodness is fragile given that
the server and client versions will inevitably diverge over time given
that this is a public repo and I have no control over clients. Stream
cloning also suffers from network latency slowing down the process - by
an order of magnitude in my case.

Cheers,
Lawrence


Cloning with v4.9 from a fast home broadband connection in Australia
that is ~165ms away from the hg-beta.freebsd.org server:
########################################
% time hg clone --stream -U http://hg-beta.freebsd.org/base freebsd_base_hg
no compatible clone bundles available on server; falling back to regular
clone
(you may want to report this to the server operator)
streaming all changes
143418 files to transfer, 1.27 GB of data
transferred 1.27 GB in 615.3 seconds (2.12 MB/sec)
23.220u 12.480s 10:20.43 5.7%	5+172k 3683+147704io 0pf+0w
########################################


Cloning with v4.9 from a fast corporate network on the US
West Coast that is ~3ms away from the hg-beta.freebsd.org server:
########################################
% time hg clone --stream -U http://hg-beta.freebsd.org/base freebsd_base_hg
no compatible clone bundles available on server; falling back to regular
clone
(you may want to report this to the server operator)
streaming all changes
143418 files to transfer, 1.27 GB of data
transferred 1.27 GB in 66.2 seconds (19.7 MB/sec)
23.252u 16.246s 1:11.11 55.5%	4+174k 921+147704io 0pf+0w
########################################