[External] remotefilelog: Large Manifest

[Augie Fackler]

> On Mar 4, 2020, at 04:55, Son Luong Ngoc <son.luong at booking.com> wrote:
> 
> Hey Pulkit,
> 
>> On Mar 4, 2020, at 10:15, Pulkit Goyal <7895pulkit at gmail.com> wrote:
>> Ah, I see. Since you have merges, you should try narrow without
>> ellipses first i.e. not use the --depth flag. Can you try:
>> `hg clone --narrow my-repo-treemanifest --include README.md  --stream`
>> and see how long does that take? It should be faster if you have quite
>> good internet connection because it's a kind of rsync.
> Oh damn adding --stream definitely helped a ton, Clone speed down from 10min to 2 minutes
> 
> But widening is still take ~20 mins
> 
>> time hg tracked --import-rules some-simple-narrow.spec
> searching for changes
> adding changesets
> adding manifests
> adding file changes
> files [         <=>                                                                                 ] 9078 -34s
> files [                                                                <=>                        ] 81549 -616s
> 
> And for some reason the estimation count in the UI is negative :D
> 
>> About merges and ellipses, there is an algorithm in narrow which is
>> not yet optimized when there are merges. That needs to be
>> improved to make `--depth` usable with merges.
> Thanks a lot for confirming this.
> 
>> Yep, I also faced this issue around a year and half ago and hacked
>> some caching thing.
> Could you please share how this was done?
> Perhaps a patch?
> 
>> 
>>> 
>>> The user experience with narrow spec-file format is also not very friendly as you are restricted to full path and no glob support.
>> IIRC, you can definitely do that. The narrow spec-file format supports
>> mercurial file patterns mentioned in `hg help patterns`
> This is from tests/test-narrow-patterns.t
> 
>> Illegal patterns are rejected
>> 
>>  $ hg tracked --addinclude glob:**
>>  abort: invalid prefix on narrow pattern: glob:**
>>  (narrow patterns must begin with one of the following: path:, rootfilesin:)
>>  [255]

Yes, it's intentional that the set of matchers you can use in a narrow spec is limited, because otherwise it could be Very expensive for a server. Can you help us understand your use case better so we can think through a solution? Globbing is probably a non-starter, as on sufficiently large repositories it'd be too expensive for the server.

> 
> Cheers,
> Son Luong.