[External] remotefilelog: Large Manifest

[Pierre-Yves David]

On 3/2/20 10:39 AM, Son Luong Ngoc wrote:
> Hi Yves,

(My first name is actually "Pierre-Yves" as a whole.)

> 
>> On Mar 1, 2020, at 20:51, Pierre-Yves David 
>> <pierre-yves.david at ens-lyon.org 
>> <mailto:pierre-yves.david at ens-lyon.org>> wrote:
>> +1 using zstd don't have a huge impact on size, but speed up 
>> read/write up to about 50%
>>
>> You can see some number here 
>> https://urldefense.com/v3/__https://www.mercurial-scm.org/repo/hg/rev/bb271ec2fbfb__;!!FzMMvhmfRQ!6rrRJJuYzgXSEUsccdjBdOayyRReVBhJmffLSSH9LKaeW20V_3FzB0FhFMjgUoRq$
> I can try this but not sure whats the best way to do a format conversion?
> A re-convert (from using hg-git) is expensive on large repo.
> Trying "hg --config format.revlog-compression=zstd clone local1 local2" 
> now, hopefully I can just copy the hg-git files over after.

You can set format.revlog-compression=zstd in your hgrc (repo or user) 
and run `hg debugupgraderepo --run`.

> 
>> Can you share some information to help with diagnostic ?
> I will not be able to give the exact number but I will try to give a 
> rough estimations
> 
>> - How many changeset do you have ?
> 150k
> 
>> - How many files do you have in your working copy ?
> Should not be relevant since we really try to have a sparse-checkout 
> working, most of my tests are with --noupdates
> But I would say around 200k

`hg manifest --rev tip | wc -l` would give you a number for the later head.

>> - How many different file did you had in the repository ever ? `hg 
>> manifest --all | wc -l`
> 450k
> 
>> - How many merge do you have.
> Using "git rev-list --merges --count HEAD" in the default branch of our 
> git repo: 100k

Do you mean you have 150k changeset, 100k of them being merge ? (so ⅔) 
If so, this is a pretty high merge ratio.

In all cases, having a 1.8GB manifest for 150k changeset is definitely 
very high I have seems repository with 10× more changeset with manifest 
2× smaller. So I would be surprised if we cannot get something better.

-- 
Pierre-Yves David