Tricks for taming large repo manifests and/or improving large repo clone times?

Gregory Szorc gregory.szorc at gmail.com
Fri May 31 02:50:27 UTC 2019


On Thu, May 30, 2019 at 1:20 AM Lawrence Stewart <lstewart at room52.net>
wrote:

> Hi Pierre-Yves,
>
> Thank you for your email, analysis and suggestion. Further comments
> inline...
>
> On 30/5/19 12:06 am, Pierre-Yves David wrote:
> >
> >
> > On 5/29/19 12:28 PM, Lawrence Stewart wrote:
> >> Greetings all,
> >>
> >> I maintain unofficial Mercurial seed repos [1] for the FreeBSD project's
> >> Subversion base system [2] and ports (non-base-system software) [3]
> >> repositories. They have sizable code bases and 20+ year development
> >> histories with quite a few active branches, and I'm curious if there are
> >> any tricks to make first-time cloning operations non-glacial?
> >>
> >> The server-side repos have been re-cloned in place to bring them up to
> >> the latest on-disk format as of mercurial 4.9 so they've got
> >> generaldelta and sparserevlog enabled, yet the manifest remains a beast
> >> (see end of email for some stats). A git clone from Github [4] is on the
> >> order of 1.5GiB and substantially faster to clone.
> >
> > Manifest size issues I have met in the past are usually tackled through
> > the use of sparse-revlog. However, you might need to force a delta
> > recomputation to get the full benefit. You can do it using the following
> > command:
> >
> >   $ hg debugupgraderepo --optimize re-delta-all --run
> >
> > (this will take a while)
> Wow, what an impressive difference the optimisation operation has made
> to the repo! I've included the same command outputs from my first email
> and a "du -h" for the optimised repo at the end of this email for
> comparison.
>
> To provide a bit more detail and context...
>
> The original full conversion from Subversion was done circa Feb 2014
> with whatever version of Mercurial was current at that time (I could
> probably figure out what I used if it's of interest). Since that time
> I've been running an incremental convert script every 15 mins to sync
> new changesets from svn.
>
> The FreeBSD development model frequently interleaves (in time) commits
> to the "head" (default/master) branch and stable branches as changes are
> merged back after some soak time, which given the size of the repo and
> changesets going in makes it a pretty brutal use case.
>
> Prior to re-cloning, the manifest had ballooned to:
>
> % ll -Sh freebsd_base_pre20190529_conversion/.hg/store/00manifest.d
>
> -rw-r--r--  1 mercurial  mercurial    16G May 29 02:10
> freebsd_base_pre20190529_conversion/.hg/store/00manifest.d
>
> which is what prompted me to investigate advancements in the on-disk
> format and initiate the Mercurial v4.9 re-clone which I accomplished
> locally on the server side with:
>
> % hg clone -U --pull freebsd_base freebsd_base_newpass1
>
> i.e. the 8.4GiB repo referred to in my first email was already a 2x
> improvement (presumably from generaldelta + sparserevlog?) from the old
> format pre-cloned repo (which I also still have if having access to it
> would be useful).
>
> > A bit more details:
> >
> > Looking at The statistic you pasted, I see 12222 delta against "others",
> > while you only have 249 snapshots. So I suspect your in-place pull
> > inherited bad delta that the previous format generated. The command
> > above should fix that by forcing these deltas to be recomputed toward a
> > better base.
>
> Yes that would appear to be the case. It is both unfortunate and
> surprising that both the in situ re-clone on the server side and a
> remote full clone using a modern 4.9 client on an end system via HTTP
> continue to be stuck with the inefficiencies of outdated on-disk
> formats/deltas.
>
> All the documentation I read suggested to me that the wire protocol
> should abstract disk format details sufficiently to allow a from-scratch
> clone to create a local repo which benefits from all the advancements
> available in a new client, but I guess not in this particular case, as
> my fresh clone still had an 8.4GiB manifest file.
>

The wire protocol - with the exception of `hg clone --stream`, which is
basically `tar | nc` - does attempt to abstract storage differences. So you
can e.g. have zstd compression on the server and a non-zstd client can
push/pull just fine. Or vice-versa.

The storage on the server does play a role though. Since servers are often
a CPU bottleneck in the grand scheme of things, servers tend to be rather
aggressive about minimizing the work they do. For example, the data that
servers send out over the wire tends to be the deltas that are in local
storage. That way they simply have to read a delta from storage and that's
it. No expensive computing a new delta, etc. Similarly, the client logic
for receiving the deltas performs minimal work to ensure the incoming data
(usually a delta) is "reasonable." And the threshold for "reasonable" is by
default pretty low. (The various optimizations that can be enabled in `hg
debugupgraderepo` change this logic to look for more options.)

There is definitely room to add configuration options to tweak behavior on
both the client and the server. This has been brought up a few times. We're
not opposed to adding the config options. But changing defaults -
especially in a direction that would make clones slower or increase CPU -
would be a difficult argument to win. I would particularly like a "server
mode" that performs aggressive searching on push operations (leading to
optimal deltas in storage) but does the least amount of work on pulls
(leading to minimal CPU usage).


>
> Not sure if there's some way to programmatically detect inefficient repo
> metadata and perhaps warn the user that there may be improvements to be
> had. I was completely unaware of the "hg debugupgraderepo --optimize
> re-delta-all --run" command's existence. Admittedly, it's probably a
> rare situation to have such an old enormous repo, so perhaps it's not
> worth investing any development cycles into...
>

Various functionality in storage / `hg debugupgraderepo` is still
relatively new. I believe it was the 4.9 release that added support for a
brand new way of choosing multiple levels of snapshots for delta bases
("sparse revlogs"). That change in particular was responsible for
drastically reducing the sizes of some repos. I'm willing to bet it shaved
>10% from the FreeBSD repo over what was possible before. And years before
that, Mercurial wasn't the best at choosing optimal deltas, leading to very
large manifests. Or we didn't have the more aggressive delta computation
enabled by default because it was too slow. And years before that, the
delta had to be the previous revision in storage. Depending on when the
FreeBSD repo conversion was initially performed, it likely inherited these
legacy sub-optimal-to-today's-standards settings, bloating the manifest
size.

I think having a way to programmatically detect inefficient repo storage
could be useful. But, so much burden is on server operators to have the
correct settings. Or at least server operators running optimal repos is the
easiest way to ensure optimal outcomes everywhere. I'm not sure how we'd
warn server operators about things other than say "it is a best practice to
run `hg debugupgraderepo` periodically." (A client-side message about a
sub-optimal server may not be optimal.) I believe we've talked about a
"server mode" config setting in the past. And a dedicated help page for
server operators. There's definitely room to improve things here...


>
> >> Am I missing anything obvious that would improve the situation?
> >
> > I am getting a local clone of your repo to have a look at it. I will
> > keep you posted.
>
> I will proceed with an optimisation pass on the ports repo too and
> report back afterwards.
>
> I also welcome any further suggestions on things I can/should do to
> further improve things.
>
> Cheers,
> Lawrence
>
> % du -h -d 3 freebsd_base
> 633K    freebsd_base/.hg/store/dh
> 1.2G    freebsd_base/.hg/store/data
> 1.5G    freebsd_base/.hg/store
> 512B    freebsd_base/.hg/wcache
>  26M    freebsd_base/.hg/cache
> 1.6G    freebsd_base/.hg
> 1.6G    freebsd_base
>
>
> % ll -Sh freebsd_base/.hg/store/ | head -5
> total 275508
> -rw-r--r--   1 mercurial  mercurial   123M May 30 01:58 00manifest.d
> -rw-r--r--   1 mercurial  mercurial    94M May 30 02:00 00changelog.d
> -rw-r--r--   1 mercurial  mercurial    19M May 30 02:00 00changelog.i
> -rw-r--r--   1 mercurial  mercurial    19M May 30 01:58 00manifest.i
>
>
> % ( cd freebsd_base && hg debugrevlog -m )
> format : 1
> flags  : generaldelta
>
> revisions     :    314061
>     merges    :         0 ( 0.00%)
>     normal    :    314061 (100.00%)
> revisions     :    314061
>     empty     :        34 ( 0.01%)
>                    text  :         0 ( 0.00%)
>                    delta :        34 (100.00%)
>     snapshot  :       317 ( 0.10%)
>       lvl-0   :              10 ( 0.00%)
>       lvl-1   :              32 ( 0.01%)
>       lvl-2   :              77 ( 0.02%)
>       lvl-3   :              98 ( 0.03%)
>       lvl-4   :              75 ( 0.02%)
>       lvl-5   :              21 ( 0.01%)
>       lvl-6   :               3 ( 0.00%)
>       lvl-7   :               1 ( 0.00%)
>     deltas    :    313710 (99.89%)
> revision size : 129170655
>     snapshot  :  64011354 (49.56%)
>       lvl-0   :         8255561 ( 6.39%)
>       lvl-1   :        10602667 ( 8.21%)
>       lvl-2   :        19453203 (15.06%)
>       lvl-3   :        15488620 (11.99%)
>       lvl-4   :         8245249 ( 6.38%)
>       lvl-5   :         1759706 ( 1.36%)
>       lvl-6   :          165476 ( 0.13%)
>       lvl-7   :           40872 ( 0.03%)
>     deltas    :  65159301 (50.44%)
>
> chunks        :    314061
>     0x00      :    185204 (58.97%)
>     empty     :        34 ( 0.01%)
>     0x78 (x)  :    128823 (41.02%)
> chunks size   : 129170655
>     0x00      :  14274035 (11.05%)
>     empty     :         0 ( 0.00%)
>     0x78 (x)  : 114896620 (88.95%)
>
> avg chain length  :      501
> max chain length  :     1000
> max chain reach   : 71198271
> compression ratio :     7458
>
> uncompressed data size (min/max/avg) : 11478 / 6316502 / 3067780
> full revision size (min/max/avg)     : 4705 / 2294460 / 825556
> inter-snapshot size (min/max/avg)    : 5786 / 1161113 / 181614
>     level-1   (min/max/avg)          : 12595 / 1161113 / 331333
>     level-2   (min/max/avg)          : 29707 / 1151391 / 252639
>     level-3   (min/max/avg)          : 5786 / 684039 / 158047
>     level-4   (min/max/avg)          : 7772 / 254821 / 109936
>     level-5   (min/max/avg)          : 49153 / 165992 / 83795
>     level-6   (min/max/avg)          : 24740 / 75899 / 55158
>     level-7   (min/max/avg)          : 40872 / 40872 / 40872
> delta size (min/max/avg)             : 12 / 235186 / 207
>
> deltas against prev  : 292653 (93.29%)
>     where prev = p1  : 292653     (100.00%)
>     where prev = p2  :      0     ( 0.00%)
>     other            :      0     ( 0.00%)
> deltas against p1    :  21025 ( 6.70%)
> deltas against p2    :      0 ( 0.00%)
> deltas against other :     32 ( 0.01%)
> _______________________________________________
> Mercurial mailing list
> Mercurial at mercurial-scm.org
> https://www.mercurial-scm.org/mailman/listinfo/mercurial
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mercurial-scm.org/pipermail/mercurial/attachments/20190530/a05660fa/attachment-0002.html>


More information about the Mercurial mailing list