[External] remotefilelog: Large Manifest

[Pulkit Goyal]

On Mon, Mar 2, 2020 at 12:12 AM Son Luong Ngoc <son.luong at booking.com> wrote:
>
> (cc facebook folks)
>
> Hi Pulkit,
>
> I have tested narrow but its has a heavy load on I/O that I am not sure how to scale it up further :(

Last time when I was testing narrow a year ago, I did encountered
memory issues on checkout, merge etc. commands because of
treemanifest. What kind of I/O issues are you facing and on what
commands?

>
> Realistically remotefilelog's approach with memcached + infinitepush is a lot more appealing for the scaling requirement.
> I found https://github.com/facebookexperimental/eden/blob/master/eden/scm/edenscm/hgext/treemanifest/__init__.py which seems to support a server-client treemanifest transfer similar to remotefilelog but I am not sure how compatible is it.
>
I looked at code, it's definitely not compatible with current core
mercurial but I am not sure how much work will be required to make it
compatible.

> Gona dig further, appreciate if anybody can share if they have explored facebook's treemanifest extension (outside of facebook context ofc)
>
> Cheers,
> Son Luong.
>
> On Mar 1, 2020, at 12:26, Pulkit Goyal <7895pulkit at gmail.com> wrote:
>
> Hi,
>
> On Tue, Feb 25, 2020 at 5:09 PM Son Luong Ngoc via Mercurial
> <mercurial at mercurial-scm.org> wrote:
>
>
> Just to add some more info:
> The repo was convert from git -> hg using hg-git latest version(Cloned from Heptapod) and HG 5.2.2 (or 5.2.1)
>
> The following is the debug info:
>
> hg debugformat
>
> format-variant    repo
> fncache:           yes
> dotencode:         yes
> generaldelta:      yes
> sparserevlog:      yes
> sidedata:           no
> copies-sdc:         no
> plain-cl-delta:    yes
> compression:       zlib
>
>
> You should definitely try out the zstd compression. It won't make a
> large difference but it will make things better.
>
> compression-level: default
>
> My current hypothesis is that because this repo was converted from git with hg-git:
> - too many HEADs/Branches
> - a non-linear history because of "merge commit"
>
> I would appreciate if somebody could give me a hint on how to deal with this case :(
>
> Cheers,
> Son Luong.
>
> On Feb 21, 2020, at 18:13, Son Luong Ngoc <son.luong at booking.com> wrote:
>
> Hey folks,
>
> Another question tinkering with RemoteFileLog extension is how to deal with large number of changelog and manifest?
> In particular, this is my current store after shallow cloning with RemoteFileLog (on a large repo, total clone time is ~20 mins)
>
> ~/test/repo-hg/.hg/store> du -sh ./*
> 304M    ./00changelog.d
> 81M    ./00changelog.i
> 1.8G    ./00manifest.d
>
>
> The manifest is quite big. A win will be to partially clone manifests
> too. With narrow extension, that can be achieved using treemanifest.
> Not sure if that works with remotefilelog too.
>
> 80M    ./00manifest.i
> 0B    ./data
> 0B    ./phaseroots
> 4.0K    ./undo
> 4.0K    ./undo.backupfiles
> 0B    ./undo.phaseroots
>
> Client side config
> ~> cat ~/.hgrc
> [extensions]
> remotefilelog =
> fsmonitor =
> sparse =
>
> [remotefilelog]
> cachepath = /Users/sluongngoc/test/hgcache
> cachelimit = 10
>
> Serverside config
>
> cat .hg/hgrc
>
> [paths]
> # HG-Git stuff
> default = git+ssh://git@gitserver/path/repo.git
>
> [remotefilelog]
> server = True
> serverexpiration = 10
>
> Thanks,
> Son Luong.
>
>
> _______________________________________________
> Mercurial mailing list
> Mercurial at mercurial-scm.org
> https://urldefense.com/v3/__https://www.mercurial-scm.org/mailman/listinfo/mercurial__;!!FzMMvhmfRQ!8YYfL1cU1fLBFj3tCOMeg5gsOGMX6nLVb76GLxlvU4oAeupr1GXWxk5kTPtx2PTD$
>
>