Tricks for taming large repo manifests and/or improving large repo clone times?

Thu May 30 08:17:02 UTC 2019

Hi Pierre-Yves,

Thank you for your email, analysis and suggestion. Further comments
inline...

On 30/5/19 12:06 am, Pierre-Yves David wrote:
> 
> 
> On 5/29/19 12:28 PM, Lawrence Stewart wrote:
>> Greetings all,
>>
>> I maintain unofficial Mercurial seed repos [1] for the FreeBSD project's
>> Subversion base system [2] and ports (non-base-system software) [3]
>> repositories. They have sizable code bases and 20+ year development
>> histories with quite a few active branches, and I'm curious if there are
>> any tricks to make first-time cloning operations non-glacial?
>>
>> The server-side repos have been re-cloned in place to bring them up to
>> the latest on-disk format as of mercurial 4.9 so they've got
>> generaldelta and sparserevlog enabled, yet the manifest remains a beast
>> (see end of email for some stats). A git clone from Github [4] is on the
>> order of 1.5GiB and substantially faster to clone.
> 
> Manifest size issues I have met in the past are usually tackled through
> the use of sparse-revlog. However, you might need to force a delta
> recomputation to get the full benefit. You can do it using the following
> command:
> 
>   $ hg debugupgraderepo --optimize re-delta-all --run
> 
> (this will take a while)
Wow, what an impressive difference the optimisation operation has made
to the repo! I've included the same command outputs from my first email
and a "du -h" for the optimised repo at the end of this email for
comparison.

To provide a bit more detail and context...

The original full conversion from Subversion was done circa Feb 2014
with whatever version of Mercurial was current at that time (I could
probably figure out what I used if it's of interest). Since that time
I've been running an incremental convert script every 15 mins to sync
new changesets from svn.

The FreeBSD development model frequently interleaves (in time) commits
to the "head" (default/master) branch and stable branches as changes are
merged back after some soak time, which given the size of the repo and
changesets going in makes it a pretty brutal use case.

Prior to re-cloning, the manifest had ballooned to:

% ll -Sh freebsd_base_pre20190529_conversion/.hg/store/00manifest.d

-rw-r--r--  1 mercurial  mercurial    16G May 29 02:10
freebsd_base_pre20190529_conversion/.hg/store/00manifest.d

which is what prompted me to investigate advancements in the on-disk
format and initiate the Mercurial v4.9 re-clone which I accomplished
locally on the server side with:

% hg clone -U --pull freebsd_base freebsd_base_newpass1

i.e. the 8.4GiB repo referred to in my first email was already a 2x
improvement (presumably from generaldelta + sparserevlog?) from the old
format pre-cloned repo (which I also still have if having access to it
would be useful).

> A bit more details:
> 
> Looking at The statistic you pasted, I see 12222 delta against "others",
> while you only have 249 snapshots. So I suspect your in-place pull
> inherited bad delta that the previous format generated. The command
> above should fix that by forcing these deltas to be recomputed toward a
> better base.

Yes that would appear to be the case. It is both unfortunate and
surprising that both the in situ re-clone on the server side and a
remote full clone using a modern 4.9 client on an end system via HTTP
continue to be stuck with the inefficiencies of outdated on-disk
formats/deltas.

All the documentation I read suggested to me that the wire protocol
should abstract disk format details sufficiently to allow a from-scratch
clone to create a local repo which benefits from all the advancements
available in a new client, but I guess not in this particular case, as
my fresh clone still had an 8.4GiB manifest file.

Not sure if there's some way to programmatically detect inefficient repo
metadata and perhaps warn the user that there may be improvements to be
had. I was completely unaware of the "hg debugupgraderepo --optimize
re-delta-all --run" command's existence. Admittedly, it's probably a
rare situation to have such an old enormous repo, so perhaps it's not
worth investing any development cycles into...

>> Am I missing anything obvious that would improve the situation?
> 
> I am getting a local clone of your repo to have a look at it. I will
> keep you posted.

I will proceed with an optimisation pass on the ports repo too and
report back afterwards.

I also welcome any further suggestions on things I can/should do to
further improve things.

Cheers,
Lawrence

% du -h -d 3 freebsd_base
633K    freebsd_base/.hg/store/dh
1.2G    freebsd_base/.hg/store/data
1.5G    freebsd_base/.hg/store
512B    freebsd_base/.hg/wcache
 26M    freebsd_base/.hg/cache
1.6G    freebsd_base/.hg
1.6G    freebsd_base

% ll -Sh freebsd_base/.hg/store/ | head -5
total 275508
-rw-r--r--   1 mercurial  mercurial   123M May 30 01:58 00manifest.d
-rw-r--r--   1 mercurial  mercurial    94M May 30 02:00 00changelog.d
-rw-r--r--   1 mercurial  mercurial    19M May 30 02:00 00changelog.i
-rw-r--r--   1 mercurial  mercurial    19M May 30 01:58 00manifest.i

% ( cd freebsd_base && hg debugrevlog -m )
format : 1
flags  : generaldelta

revisions     :    314061
    merges    :         0 ( 0.00%)
    normal    :    314061 (100.00%)
revisions     :    314061
    empty     :        34 ( 0.01%)
                   text  :         0 ( 0.00%)
                   delta :        34 (100.00%)
    snapshot  :       317 ( 0.10%)
      lvl-0   :              10 ( 0.00%)
      lvl-1   :              32 ( 0.01%)
      lvl-2   :              77 ( 0.02%)
      lvl-3   :              98 ( 0.03%)
      lvl-4   :              75 ( 0.02%)
      lvl-5   :              21 ( 0.01%)
      lvl-6   :               3 ( 0.00%)
      lvl-7   :               1 ( 0.00%)
    deltas    :    313710 (99.89%)
revision size : 129170655
    snapshot  :  64011354 (49.56%)
      lvl-0   :         8255561 ( 6.39%)
      lvl-1   :        10602667 ( 8.21%)
      lvl-2   :        19453203 (15.06%)
      lvl-3   :        15488620 (11.99%)
      lvl-4   :         8245249 ( 6.38%)
      lvl-5   :         1759706 ( 1.36%)
      lvl-6   :          165476 ( 0.13%)
      lvl-7   :           40872 ( 0.03%)
    deltas    :  65159301 (50.44%)

chunks        :    314061
    0x00      :    185204 (58.97%)
    empty     :        34 ( 0.01%)
    0x78 (x)  :    128823 (41.02%)
chunks size   : 129170655
    0x00      :  14274035 (11.05%)
    empty     :         0 ( 0.00%)
    0x78 (x)  : 114896620 (88.95%)

avg chain length  :      501
max chain length  :     1000
max chain reach   : 71198271
compression ratio :     7458

uncompressed data size (min/max/avg) : 11478 / 6316502 / 3067780
full revision size (min/max/avg)     : 4705 / 2294460 / 825556
inter-snapshot size (min/max/avg)    : 5786 / 1161113 / 181614
    level-1   (min/max/avg)          : 12595 / 1161113 / 331333
    level-2   (min/max/avg)          : 29707 / 1151391 / 252639
    level-3   (min/max/avg)          : 5786 / 684039 / 158047
    level-4   (min/max/avg)          : 7772 / 254821 / 109936
    level-5   (min/max/avg)          : 49153 / 165992 / 83795
    level-6   (min/max/avg)          : 24740 / 75899 / 55158
    level-7   (min/max/avg)          : 40872 / 40872 / 40872
delta size (min/max/avg)             : 12 / 235186 / 207

deltas against prev  : 292653 (93.29%)
    where prev = p1  : 292653     (100.00%)
    where prev = p2  :      0     ( 0.00%)
    other            :      0     ( 0.00%)
deltas against p1    :  21025 ( 6.70%)
deltas against p2    :      0 ( 0.00%)
deltas against other :     32 ( 0.01%)