Tricks for taming large repo manifests and/or improving large repo clone times?

[Gregory Szorc]

On Thu, May 30, 2019 at 1:26 AM Lawrence Stewart <lstewart at room52.net>
wrote:

> Hi Augie,
>
> On 29/5/19 11:16 pm, Augie Fackler wrote:
> >
> >
> >> On May 29, 2019, at 06:28, Lawrence Stewart <lstewart at room52.net>
> wrote:
> >>
> >> Greetings all,
> >>
> >> I maintain unofficial Mercurial seed repos [1] for the FreeBSD project's
> >> Subversion base system [2] and ports (non-base-system software) [3]
> >> repositories. They have sizable code bases and 20+ year development
> >> histories with quite a few active branches, and I'm curious if there are
> >> any tricks to make first-time cloning operations non-glacial?
> >>
> >> The server-side repos have been re-cloned in place to bring them up to
> >> the latest on-disk format as of mercurial 4.9 so they've got
> >> generaldelta and sparserevlog enabled, yet the manifest remains a beast
> >> (see end of email for some stats). A git clone from Github [4] is on the
> >> order of 1.5GiB and substantially faster to clone.
> >>
> >> Am I missing anything obvious that would improve the situation?
> >
> > Are you using clonebundles?
> https://www.mercurial-scm.org/wiki/ClonebundlesExtension talks about it.
> The extension is used on the server, and clients will handle it
> automatically if enabled. The basic idea is you pre-compute a bundle of
> your repo and stick it someplace fast like a CDN and then clients don't
> have to pull everything from the main server, only what's changed since the
> bundle was generated.
>
> No, but thanks for the pointer. I was not familiar with the clonebundles
> extension.
>

I'd like to also throw out a link to
https://gregoryszorc.com/blog/2018/07/27/benefits-of-clone-offload-on-version-control-hosting/
(which I just added to the wiki page). tl;dr clonebundles is offloading
~97% of bytes served from hg.mozilla.org. It's about as close to a magic
bullet for server scaling as you can get. Plus if you host your bundles in
a CDN, clients can download ~99% of the repository data from a closer
endpoint instead of potentially from a server on a different continent.
Mozilla's European and Asian based contributors definitely noticed when
their clones didn't have to transfer large amounts of data under an ocean
from California!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mercurial-scm.org/pipermail/mercurial/attachments/20190530/e3ab409e/attachment-0002.html>