Tricks for taming large repo manifests and/or improving large repo clone times?

[Lawrence Stewart]

Hi Pulkit,

On 30/5/19 5:08 am, Pulkit Goyal wrote:
> On Wed, May 29, 2019 at 9:48 PM Augie Fackler <raf at durin42.com
> <mailto:raf at durin42.com>> wrote:
>     > On May 29, 2019, at 10:06, Pierre-Yves David
>     <pierre-yves.david at ens-lyon.org
>     <mailto:pierre-yves.david at ens-lyon.org>> wrote:
>     >
>     >
>     >
>     > On 5/29/19 12:28 PM, Lawrence Stewart wrote:
>     >> Greetings all,
>     >> I maintain unofficial Mercurial seed repos [1] for the FreeBSD
>     project's
>     >> Subversion base system [2] and ports (non-base-system software) [3]
>     >> repositories. They have sizable code bases and 20+ year development
>     >> histories with quite a few active branches, and I'm curious if
>     there are
>     >> any tricks to make first-time cloning operations non-glacial?
>     >> The server-side repos have been re-cloned in place to bring them
>     up to
>     >> the latest on-disk format as of mercurial 4.9 so they've got
>     >> generaldelta and sparserevlog enabled, yet the manifest remains a
>     beast
>     >> (see end of email for some stats). A git clone from Github [4] is
>     on the
>     >> order of 1.5GiB and substantially faster to clone.
>     >
>     > Manifest size issues I have met in the past are usually tackled
>     through the use of sparse-revlog. However, you might need to force a
>     delta recomputation to get the full benefit. You can do it using the
>     following command:
>     >
>     >  $ hg debugupgraderepo --optimize re-delta-all --run
> 
>     I ran this, and it saved almost all of the manifest size. It's now
>     124 megabytes instead of 8.4 gigabytes. Total .hg/store size is now
>     1.7G.
> 
> 
> I got this smaller repo from Augie and updated that to use zstd
> compression for revlogs. It saves another 45.4 MB.
> 
> (Upgrading to zstd can be done running hg debugupgraderepo with
> `--config format.revlog-compression=zstd`. Make sure you have latest hg
> which is 5.0)
> 
> zstd will lead to faster read and write times too. Look
> https://www.mercurial-scm.org/repo/hg-committed/rev/bb271ec2fbfbd456d1281c5dad21f92db6d30f7b
> for some numbers.

Nice! Thanks for the suggestion and data point on the extra space
savings. Mercurial 5.0 hasn't hit the FreeBSD ports tree yet, but as
soon as it does I'll look into trying this tweak.

I'm guessing this feature is backwards compatible, in that pre-5.0
clients will happily talk to a 5.0 zstd compressed repo, but end up
without the compression benefits in their local clone?

Cheers,
Lawrence