Tricks for taming large repo manifests and/or improving large repo clone times?

[Arne Babenhauserheide]

Pierre-Yves David <pierre-yves.david at ens-lyon.org> writes:

> On 5/29/19 12:28 PM, Lawrence Stewart wrote:
> Manifest size issues I have met in the past are usually tackled through 
> the use of sparse-revlog. However, you might need to force a delta 
> recomputation to get the full benefit. You can do it using the following 
> command:
>
>    $ hg debugupgraderepo --optimize re-delta-all --run
>
> (this will take a while)

The effect I see from that is:


$ time hg debugupgraderepo --optimize re-delta-all --run
upgrade will perform the following actions:

requirements
   preserved: dotencode, fncache, generaldelta, revlog-compression-zstd, revlogv1, sparserevlog, store

re-delta-all
   deltas within internal storage will be fully recomputed; this will likely drastically slow down execution time

beginning upgrade...
repository locked and read-only
creating temporary repository to stage migrated data: /tmp/freebsd_base/.hg/upgrade.wMMBte
(it is safe to interrupt this process any time before data migration completes)
migrating 1808767 total revisions (1179022 in filelogs, 314044 in manifests, 315701 in changelog)
migrating 9.35 GB in store; 927 GB tracked data
… 
replaced files will be backed up at /tmp/freebsd_base/.hg/upgradebackup.0DFqmi
… 
finalizing requirements file and making repository readable again
removing temporary repository /tmp/freebsd_base/.hg/upgrade.wMMBte
copy of old repository backed up at /tmp/freebsd_base/.hg/upgradebackup.0DFqmi
the old repository will not be deleted; remove it to free up disk space once the upgraded repository is verified

real	51m53,468s
user	49m51,621s
sys	1m30,654s

… 

$ du -hs .hg/store/* | sort -h
0	.hg/store/phaseroots
0	.hg/store/undo.phaseroots
4,0K	.hg/store/undo.backupfiles
688K	.hg/store/dh
6,2M	.hg/store/fncache
6,5M	.hg/store/undo
20M	.hg/store/00changelog.i
20M	.hg/store/00manifest.i
98M	.hg/store/00changelog.d
125M	.hg/store/00manifest.d
1,5G	.hg/store/data

Note also that most data is now in .hg/store/data/, so it’s no longer
bound to a single huge file but rather optimized for efficient
streaming retrieval from filesystem.

Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein
ohne es zu merken
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 1076 bytes
Desc: not available
URL: <http://lists.mercurial-scm.org/pipermail/mercurial/attachments/20190529/6839db33/attachment.asc>