Tricks for taming large repo manifests and/or improving large repo clone times?
Pierre-Yves David
pierre-yves.david at ens-lyon.org
Tue Jun 11 21:38:22 UTC 2019
On 5/31/19 4:50 AM, Gregory Szorc wrote:
> On Thu, May 30, 2019 at 1:20 AM Lawrence Stewart <lstewart at room52.net
> <mailto:lstewart at room52.net>> wrote:
>
> Hi Pierre-Yves,
>
> Thank you for your email, analysis and suggestion. Further comments
> inline...
>
> On 30/5/19 12:06 am, Pierre-Yves David wrote:
> >
> >
> > On 5/29/19 12:28 PM, Lawrence Stewart wrote:
> >> Greetings all,
> >>
> >> I maintain unofficial Mercurial seed repos [1] for the FreeBSD
> project's
> >> Subversion base system [2] and ports (non-base-system software) [3]
> >> repositories. They have sizable code bases and 20+ year development
> >> histories with quite a few active branches, and I'm curious if
> there are
> >> any tricks to make first-time cloning operations non-glacial?
> >>
> >> The server-side repos have been re-cloned in place to bring them
> up to
> >> the latest on-disk format as of mercurial 4.9 so they've got
> >> generaldelta and sparserevlog enabled, yet the manifest remains
> a beast
> >> (see end of email for some stats). A git clone from Github [4]
> is on the
> >> order of 1.5GiB and substantially faster to clone.
> >
> > Manifest size issues I have met in the past are usually tackled
> through
> > the use of sparse-revlog. However, you might need to force a delta
> > recomputation to get the full benefit. You can do it using the
> following
> > command:
> >
> > $ hg debugupgraderepo --optimize re-delta-all --run
> >
> > (this will take a while)
> Wow, what an impressive difference the optimisation operation has made
> to the repo! I've included the same command outputs from my first email
> and a "du -h" for the optimised repo at the end of this email for
> comparison.
>
> To provide a bit more detail and context...
>
> The original full conversion from Subversion was done circa Feb 2014
> with whatever version of Mercurial was current at that time (I could
> probably figure out what I used if it's of interest). Since that time
> I've been running an incremental convert script every 15 mins to sync
> new changesets from svn.
>
> The FreeBSD development model frequently interleaves (in time) commits
> to the "head" (default/master) branch and stable branches as changes are
> merged back after some soak time, which given the size of the repo and
> changesets going in makes it a pretty brutal use case.
>
> Prior to re-cloning, the manifest had ballooned to:
>
> % ll -Sh freebsd_base_pre20190529_conversion/.hg/store/00manifest.d
>
> -rw-r--r-- 1 mercurial mercurial 16G May 29 02:10
> freebsd_base_pre20190529_conversion/.hg/store/00manifest.d
>
> which is what prompted me to investigate advancements in the on-disk
> format and initiate the Mercurial v4.9 re-clone which I accomplished
> locally on the server side with:
>
> % hg clone -U --pull freebsd_base freebsd_base_newpass1
>
> i.e. the 8.4GiB repo referred to in my first email was already a 2x
> improvement (presumably from generaldelta + sparserevlog?) from the old
> format pre-cloned repo (which I also still have if having access to it
> would be useful).
>
> > A bit more details:
> >
> > Looking at The statistic you pasted, I see 12222 delta against
> "others",
> > while you only have 249 snapshots. So I suspect your in-place pull
> > inherited bad delta that the previous format generated. The command
> > above should fix that by forcing these deltas to be recomputed
> toward a
> > better base.
>
> Yes that would appear to be the case. It is both unfortunate and
> surprising that both the in situ re-clone on the server side and a
> remote full clone using a modern 4.9 client on an end system via HTTP
> continue to be stuck with the inefficiencies of outdated on-disk
> formats/deltas.
>
> All the documentation I read suggested to me that the wire protocol
> should abstract disk format details sufficiently to allow a from-scratch
> clone to create a local repo which benefits from all the advancements
> available in a new client, but I guess not in this particular case, as
> my fresh clone still had an 8.4GiB manifest file.
>
>
> The wire protocol - with the exception of `hg clone --stream`, which is
> basically `tar | nc` - does attempt to abstract storage differences. So
> you can e.g. have zstd compression on the server and a non-zstd client
> can push/pull just fine. Or vice-versa.
>
> The storage on the server does play a role though. Since servers are
> often a CPU bottleneck in the grand scheme of things, servers tend to be
> rather aggressive about minimizing the work they do. For example, the
> data that servers send out over the wire tends to be the deltas that are
> in local storage. That way they simply have to read a delta from storage
> and that's it. No expensive computing a new delta, etc. Similarly, the
> client logic for receiving the deltas performs minimal work to ensure
> the incoming data (usually a delta) is "reasonable." And the threshold
> for "reasonable" is by default pretty low. (The various optimizations
> that can be enabled in `hg debugupgraderepo` change this logic to look
> for more options.)
>
> There is definitely room to add configuration options to tweak behavior
> on both the client and the server. This has been brought up a few times.
> We're not opposed to adding the config options.
The behavior of the "receiving" side can be controlled by two config
options:
To prevent mercurial to blindly reused the delta it receive, one can set:
[storage]
revlog.reuse-external-delta-parent = no
That behavior can be tweaked further, force all delta to be recomputed,
even if the received one is valid and match the first delta parent
candidate:
[storage]
revlog.reuse-external-delta=no
> But changing defaults -
> especially in a direction that would make clones slower or increase CPU
> - would be a difficult argument to win. I would particularly like a
> "server mode" that performs aggressive searching on push operations
> (leading to optimal deltas in storage) but does the least amount of work
> on pulls (leading to minimal CPU usage).
Setting the above option(s) server side will lead to that behavior.
> Not sure if there's some way to programmatically detect inefficient repo
> metadata and perhaps warn the user that there may be improvements to be
> had. I was completely unaware of the "hg debugupgraderepo --optimize
> re-delta-all --run" command's existence. Admittedly, it's probably a
> rare situation to have such an old enormous repo, so perhaps it's not
> worth investing any development cycles into...
The way this was spotted in the freebsd case was from the high amount of
delta using "prev" as the delta base (with prev ≠ p1 or p2). Maybe we
could try to generalize this test.
> Various functionality in storage / `hg debugupgraderepo` is still
> relatively new. I believe it was the 4.9 release that added support for
> a brand new way of choosing multiple levels of snapshots for delta bases
> ("sparse revlogs").
The option is available since 4.7 (but got further improved in 4.8 and 4.9)
> That change in particular was responsible for
> drastically reducing the sizes of some repos. I'm willing to bet it
> shaved >10% from the FreeBSD repo over what was possible before. And
> years before that, Mercurial wasn't the best at choosing optimal deltas,
> leading to very large manifests. Or we didn't have the more aggressive
> delta computation enabled by default because it was too slow. And years
> before that, the delta had to be the previous revision in storage.
> Depending on when the FreeBSD repo conversion was initially performed,
> it likely inherited these legacy sub-optimal-to-today's-standards
> settings, bloating the manifest size.
Yes, "bad" but still "valid" delta were simply inherited from the
previous storage (to save CPU on clone)
> I think having a way to programmatically detect inefficient repo storage
> could be useful. But, so much burden is on server operators to have the
> correct settings. Or at least server operators running optimal repos is
> the easiest way to ensure optimal outcomes everywhere. I'm not sure how
> we'd warn server operators about things other than say "it is a best
> practice to run `hg debugupgraderepo` periodically." (A client-side
> message about a sub-optimal server may not be optimal.) I believe we've
> talked about a "server mode" config setting in the past. And a dedicated
> help page for server operators. There's definitely room to improve
> things here...
>
>
> >> Am I missing anything obvious that would improve the situation?
> >
> > I am getting a local clone of your repo to have a look at it. I will
> > keep you posted.
>
> I will proceed with an optimisation pass on the ports repo too and
> report back afterwards.
>
> I also welcome any further suggestions on things I can/should do to
> further improve things.
>
> Cheers,
> Lawrence
>
> % du -h -d 3 freebsd_base
> 633K freebsd_base/.hg/store/dh
> 1.2G freebsd_base/.hg/store/data
> 1.5G freebsd_base/.hg/store
> 512B freebsd_base/.hg/wcache
> 26M freebsd_base/.hg/cache
> 1.6G freebsd_base/.hg
> 1.6G freebsd_base
>
>
> % ll -Sh freebsd_base/.hg/store/ | head -5
> total 275508
> -rw-r--r-- 1 mercurial mercurial 123M May 30 01:58 00manifest.d
> -rw-r--r-- 1 mercurial mercurial 94M May 30 02:00 00changelog.d
> -rw-r--r-- 1 mercurial mercurial 19M May 30 02:00 00changelog.i
> -rw-r--r-- 1 mercurial mercurial 19M May 30 01:58 00manifest.i
>
>
> % ( cd freebsd_base && hg debugrevlog -m )
> format : 1
> flags : generaldelta
>
> revisions : 314061
> merges : 0 ( 0.00%)
> normal : 314061 (100.00%)
> revisions : 314061
> empty : 34 ( 0.01%)
> text : 0 ( 0.00%)
> delta : 34 (100.00%)
> snapshot : 317 ( 0.10%)
> lvl-0 : 10 ( 0.00%)
> lvl-1 : 32 ( 0.01%)
> lvl-2 : 77 ( 0.02%)
> lvl-3 : 98 ( 0.03%)
> lvl-4 : 75 ( 0.02%)
> lvl-5 : 21 ( 0.01%)
> lvl-6 : 3 ( 0.00%)
> lvl-7 : 1 ( 0.00%)
> deltas : 313710 (99.89%)
> revision size : 129170655
> snapshot : 64011354 (49.56%)
> lvl-0 : 8255561 ( 6.39%)
> lvl-1 : 10602667 ( 8.21%)
> lvl-2 : 19453203 (15.06%)
> lvl-3 : 15488620 (11.99%)
> lvl-4 : 8245249 ( 6.38%)
> lvl-5 : 1759706 ( 1.36%)
> lvl-6 : 165476 ( 0.13%)
> lvl-7 : 40872 ( 0.03%)
> deltas : 65159301 (50.44%)
>
> chunks : 314061
> 0x00 : 185204 (58.97%)
> empty : 34 ( 0.01%)
> 0x78 (x) : 128823 (41.02%)
> chunks size : 129170655
> 0x00 : 14274035 (11.05%)
> empty : 0 ( 0.00%)
> 0x78 (x) : 114896620 (88.95%)
>
> avg chain length : 501
> max chain length : 1000
> max chain reach : 71198271
> compression ratio : 7458
>
> uncompressed data size (min/max/avg) : 11478 / 6316502 / 3067780
> full revision size (min/max/avg) : 4705 / 2294460 / 825556
> inter-snapshot size (min/max/avg) : 5786 / 1161113 / 181614
> level-1 (min/max/avg) : 12595 / 1161113 / 331333
> level-2 (min/max/avg) : 29707 / 1151391 / 252639
> level-3 (min/max/avg) : 5786 / 684039 / 158047
> level-4 (min/max/avg) : 7772 / 254821 / 109936
> level-5 (min/max/avg) : 49153 / 165992 / 83795
> level-6 (min/max/avg) : 24740 / 75899 / 55158
> level-7 (min/max/avg) : 40872 / 40872 / 40872
> delta size (min/max/avg) : 12 / 235186 / 207
>
> deltas against prev : 292653 (93.29%)
> where prev = p1 : 292653 (100.00%)
> where prev = p2 : 0 ( 0.00%)
> other : 0 ( 0.00%)
> deltas against p1 : 21025 ( 6.70%)
> deltas against p2 : 0 ( 0.00%)
> deltas against other : 32 ( 0.01%)
> _______________________________________________
> Mercurial mailing list
> Mercurial at mercurial-scm.org <mailto:Mercurial at mercurial-scm.org>
> https://www.mercurial-scm.org/mailman/listinfo/mercurial
>
--
Pierre-Yves David
More information about the Mercurial
mailing list