[External] remotefilelog: Large Manifest

[Pulkit Goyal]

On Wed, Mar 4, 2020 at 3:25 PM Son Luong Ngoc <son.luong at booking.com> wrote:
>
> Hey Pulkit,
>
> > On Mar 4, 2020, at 10:15, Pulkit Goyal <7895pulkit at gmail.com> wrote:
> > Ah, I see. Since you have merges, you should try narrow without
> > ellipses first i.e. not use the --depth flag. Can you try:
> > `hg clone --narrow my-repo-treemanifest --include README.md  --stream`
> > and see how long does that take? It should be faster if you have quite
> > good internet connection because it's a kind of rsync.
> Oh damn adding --stream definitely helped a ton, Clone speed down from 10min to 2 minutes
>
> But widening is still take ~20 mins
>
> > time hg tracked --import-rules some-simple-narrow.spec
> searching for changes
> adding changesets
> adding manifests
> adding file changes
> files [         <=>                                                                                 ] 9078 -34s
> files [                                                                <=>                        ] 81549 -616s
>
> And for some reason the estimation count in the UI is negative :D

Yeah, I encountered it too but never got around fixing it.
>
> > About merges and ellipses, there is an algorithm in narrow which is
> > not yet optimized when there are merges. That needs to be
> > improved to make `--depth` usable with merges.
> Thanks a lot for confirming this.
>
> > Yep, I also faced this issue around a year and half ago and hacked
> > some caching thing.
> Could you please share how this was done?

IIRC I wrote all the required map in cache file in simple `<key>
<value>\n` format and read it from there. I was trying to optimize
work done in this callback:
https://www.mercurial-scm.org/repo/hg-committed/file/02c47b74366c/mercurial/changegroup.py#l1155.
I guess caching the end result of `tmfclnodes` but not sure.
> Perhaps a patch?

Unfortunately I never managed to work on that more than just perf
testing locally, so never pushed that publicaly. And looks like I lost
that patch somewhere while switching machines in last year.
>
> >
> >>
> >> The user experience with narrow spec-file format is also not very friendly as you are restricted to full path and no glob support.
> > IIRC, you can definitely do that. The narrow spec-file format supports
> > mercurial file patterns mentioned in `hg help patterns`
> This is from tests/test-narrow-patterns.t
>
> > Illegal patterns are rejected
> >
> >   $ hg tracked --addinclude glob:**
> >   abort: invalid prefix on narrow pattern: glob:**
> >   (narrow patterns must begin with one of the following: path:, rootfilesin:)
> >   [255]
>
> Cheers,
> Son Luong.