Tricks for taming large repo manifests and/or improving large repo clone times?

[Augie Fackler]

> On May 29, 2019, at 10:06, Pierre-Yves David <pierre-yves.david at ens-lyon.org> wrote:
> 
> 
> 
> On 5/29/19 12:28 PM, Lawrence Stewart wrote:
>> Greetings all,
>> I maintain unofficial Mercurial seed repos [1] for the FreeBSD project's
>> Subversion base system [2] and ports (non-base-system software) [3]
>> repositories. They have sizable code bases and 20+ year development
>> histories with quite a few active branches, and I'm curious if there are
>> any tricks to make first-time cloning operations non-glacial?
>> The server-side repos have been re-cloned in place to bring them up to
>> the latest on-disk format as of mercurial 4.9 so they've got
>> generaldelta and sparserevlog enabled, yet the manifest remains a beast
>> (see end of email for some stats). A git clone from Github [4] is on the
>> order of 1.5GiB and substantially faster to clone.
> 
> Manifest size issues I have met in the past are usually tackled through the use of sparse-revlog. However, you might need to force a delta recomputation to get the full benefit. You can do it using the following command:
> 
>  $ hg debugupgraderepo --optimize re-delta-all --run

I ran this, and it saved almost all of the manifest size. It's now 124 megabytes instead of 8.4 gigabytes. Total .hg/store size is now 1.7G.

I'm happy to send people a copy of the cleaned up repo, magic-wormhole[0] is probably easiest.

0: https://magic-wormhole.readthedocs.io/en/latest/

> (this will take a while)
> 
> A bit more details:
> 
> Looking at The statistic you pasted, I see 12222 delta against "others", while you only have 249 snapshots. So I suspect your in-place pull inherited bad delta that the previous format generated. The command above should fix that by forcing these deltas to be recomputed toward a better base.

Good catch!

> 
>> Am I missing anything obvious that would improve the situation?
> 
> I am getting a local clone of your repo to have a look at it. I will keep you posted.

Let me know if you still want a copy, either of the original big one or of the compressed one. I have tars of both and can easily wormhole them to you.

> 
> Cheers,
> 
> -- 
> Pierre-Yves David
> _______________________________________________
> Mercurial mailing list
> Mercurial at mercurial-scm.org
> https://www.mercurial-scm.org/mailman/listinfo/mercurial