Tricks for taming large repo manifests and/or improving large repo clone times?

[Pulkit Goyal]

On Wed, May 29, 2019 at 9:48 PM Augie Fackler <raf at durin42.com> wrote:

>
>
> > On May 29, 2019, at 10:06, Pierre-Yves David <
> pierre-yves.david at ens-lyon.org> wrote:
> >
> >
> >
> > On 5/29/19 12:28 PM, Lawrence Stewart wrote:
> >> Greetings all,
> >> I maintain unofficial Mercurial seed repos [1] for the FreeBSD project's
> >> Subversion base system [2] and ports (non-base-system software) [3]
> >> repositories. They have sizable code bases and 20+ year development
> >> histories with quite a few active branches, and I'm curious if there are
> >> any tricks to make first-time cloning operations non-glacial?
> >> The server-side repos have been re-cloned in place to bring them up to
> >> the latest on-disk format as of mercurial 4.9 so they've got
> >> generaldelta and sparserevlog enabled, yet the manifest remains a beast
> >> (see end of email for some stats). A git clone from Github [4] is on the
> >> order of 1.5GiB and substantially faster to clone.
> >
> > Manifest size issues I have met in the past are usually tackled through
> the use of sparse-revlog. However, you might need to force a delta
> recomputation to get the full benefit. You can do it using the following
> command:
> >
> >  $ hg debugupgraderepo --optimize re-delta-all --run
>
> I ran this, and it saved almost all of the manifest size. It's now 124
> megabytes instead of 8.4 gigabytes. Total .hg/store size is now 1.7G.
>

I got this smaller repo from Augie and updated that to use zstd compression
for revlogs. It saves another 45.4 MB.

(Upgrading to zstd can be done running hg debugupgraderepo with `--config
format.revlog-compression=zstd`. Make sure you have latest hg which is 5.0)

zstd will lead to faster read and write times too. Look
https://www.mercurial-scm.org/repo/hg-committed/rev/bb271ec2fbfbd456d1281c5dad21f92db6d30f7b
for some numbers.

>
> I'm happy to send people a copy of the cleaned up repo, magic-wormhole[0]
> is probably easiest.
>

I took me 1 minute to get the tar-ed repo using my fast internet connection.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mercurial-scm.org/pipermail/mercurial/attachments/20190529/d885cac8/attachment-0002.html>