hg way to get just a bundle of a remote repository without applying it

[Gregory Szorc]

On Thu, Feb 1, 2018 at 3:35 PM, Avi Kelman <patcherton.fixesthings at gmail.com
> wrote:

> On Wed, 2018-01-31 at 10:07 -0500, Avi Kelman wrote:
> > On Wed, 2018-01-31 at 09:58 -0500, Avi Kelman wrote:
> > > On Tue, 2018-01-30 at 11:54 -0600, Kevin Bullock wrote:
> > > > > On Jan 25, 2018, at 18:35, Avi Kelman <
> patcherton.fixesthings at gmail.com
> > > > > >
> > > > > wrote:
> > > > >
> > > > > Hi all. I'm looking for how to do basically the first part of hg
> clone,
> > > > > where a
> > > > > bundle file is retrieved over the wire from a remote server, but
> then
> > > > > just
> > > > > saving that bundle file
> > > >
> > > > You can use 'hg incoming --bundle FILE' for this. You'll need to run
> it
> > > > inside of an empty local hg repo though.
> > >
> > > Ok, this sort of works, but...
> > >
> > > After downloading, the process for `hg incoming --bundle bundlefile
> --quiet
> > > htt
> > > ps://hg.mozilla.org/mozilla-unified/`
> <http://hg.mozilla.org/mozilla-unified/> fills up a whopping 12GB of RAM
> which
> > > is
> > > catastrophic to the system.
> > >
> > > The whole bundle is only ~5GB uncompressed, and mozilla specifically
> > > premakes
> > > compressed bundles for clone to fetch quickly.
> > >
> > > I'm really looking to just save what clone is downloading and then stop
> > > there,
> > > even if it means a python script. :\
> >
>
> Ugh. After logging out and back in the RAM problem has stopped happening. I
> guess something in my system interfered with the update to 4.5. Sorry for
> the
> spam I guess. Thanks for the help. Does anything control the output bundle
> format of hg incoming?
>

When running `hg pull <server>` or `hg unbundle`, the bundle application
code essentially streams the bundle into the repo. It's pretty efficient.

When running `hg pull <bundle>` or `hg incoming --bundle <FILE>`, things go
through a separate code path where the bundle is "indexed." This code is
not very efficient and memory use can explode for large bundles. I think
there's a reported bug on it in the bug tracker.

In theory, it is possible to get a Mercurial server to emit a bundle that
can be saved to a file instead of applied. Calling the "getbundle" wire
protocol command with the right set of arguments is effectively the same as
remotely running `hg bundle` on the server. Unfortunately, I don't think we
have a good interface for manually calling wire protocol commands. But if
you read `hg help internals.wireprotocol`, you can construct an HTTP
request. e.g.

curl -H 'x-hgarg-1:
bookmarks=1&bundlecaps=HG20%2Cbundle2%3DHG20%250Abookmarks%250Achangegroup%253D01%252C02%250Adigests%253Dmd5%252Csha1%252Csha512%250Aerror%253Dabort%252Cunsupportedcontent%252Cpushraced%252Cpushkey%250A'hgtagsfnodes%250A'listkeys%250Aphases%253Dheads%250Apushkey%250Aremote-changegroup%253Dhttp%252Chttps%250Astream%253Dv2&cg=0&common=0000000000000000000000000000000000000000&heads=5fed3813f7f5e1824344fdc9cf8f63bb662c292d&listkeys=bookmarks&phases=1'
-H 'x-hgproto-1: 0.1 0.2 comp=zstd' https://some/repo?cmd=getbundle

Just plug in the SHA1 of the DAG head(s) you want to fetch and it should
work.

Come to think of it, I bet we could teach `hg bundle` to communicate with a
remote server and save the bundle file locally. The missing piece here is
the ability to convert the requested arguments into a "getbundle" wire
protocol command. This feels within the realm of possibility. I've added
the feature to https://www.mercurial-scm.org/wiki/WeShouldDoThat :)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mercurial-scm.org/pipermail/mercurial/attachments/20180220/bde52182/attachment-0002.html>