D9283: bundle: optional multithreaded compression, ATM zstd-only

[joerg.sonnenberger (Joerg Sonnenberger)]

joerg.sonnenberger created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  Compression type can be a huge chunk of "hg bundle", especially when
  using the higher compression levels. With level=22 and threads=7, the
  NetBSD test repository took 28:39 wall time and 157:47 user time.
  Before, level=22 would take 129:20 wall time and 129:07 user time.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D9283

AFFECTED FILES
  mercurial/commands.py
  mercurial/configitems.py
  mercurial/utils/compression.py
  relnotes/next
  tests/test-bundle-type.t

CHANGE DETAILS

diff --git a/tests/test-bundle-type.t b/tests/test-bundle-type.t
--- a/tests/test-bundle-type.t
+++ b/tests/test-bundle-type.t
@@ -201,6 +201,15 @@
   (see 'hg help bundlespec' for supported values for --type)
   [255]
 
+zstd supports threading
+
+  $ hg init test-compthreads
+  $ cd test-compthreads
+  $ hg debugbuilddag +3
+  $ hg --config experimental.bundlecompthreads=1 bundle -a -t zstd-v2 zstd-v2-threaded.hg
+  3 changesets found
+  $ cd ..
+
 #else
 
 zstd is a valid engine but isn't available
diff --git a/relnotes/next b/relnotes/next
--- a/relnotes/next
+++ b/relnotes/next
@@ -1,6 +1,11 @@
 == New Features ==
 
 
+ * The new options `experimental.bundlecompthreads` and
+   `experimental.bundlecompthreads.<engine>` can be used to instruct
+   the compression engines for bundle operations to use multiple threads
+   for compression. The default is single threaded operation. Currently
+   only supported for zstd.
 
 == New Experimental Features ==
 
diff --git a/mercurial/utils/compression.py b/mercurial/utils/compression.py
--- a/mercurial/utils/compression.py
+++ b/mercurial/utils/compression.py
@@ -682,9 +682,11 @@
         # while providing no worse compression. It strikes a good balance
         # between speed and compression.
         level = opts.get(b'level', 3)
+        # default to single-threaded compression
+        threads = opts.get(b'threads', 0)
 
         zstd = self._module
-        z = zstd.ZstdCompressor(level=level).compressobj()
+        z = zstd.ZstdCompressor(level=level, threads=threads).compressobj()
         for chunk in it:
             data = z.compress(chunk)
             if data:
diff --git a/mercurial/configitems.py b/mercurial/configitems.py
--- a/mercurial/configitems.py
+++ b/mercurial/configitems.py
@@ -546,6 +546,21 @@
     b'experimental', b'bundlecomplevel.zstd', default=None,
 )
 coreconfigitem(
+    b'experimental', b'bundlecompthreads', default=None,
+)
+coreconfigitem(
+    b'experimental', b'bundlecompthreads.bzip2', default=None,
+)
+coreconfigitem(
+    b'experimental', b'bundlecompthreads.gzip', default=None,
+)
+coreconfigitem(
+    b'experimental', b'bundlecompthreads.none', default=None,
+)
+coreconfigitem(
+    b'experimental', b'bundlecompthreads.zstd', default=None,
+)
+coreconfigitem(
     b'experimental', b'changegroup3', default=False,
 )
 coreconfigitem(
diff --git a/mercurial/commands.py b/mercurial/commands.py
--- a/mercurial/commands.py
+++ b/mercurial/commands.py
@@ -1629,6 +1629,14 @@
     if complevel is not None:
         compopts[b'level'] = complevel
 
+    compthreads = ui.configint(
+        b'experimental', b'bundlecompthreads.' + bundlespec.compression
+    )
+    if compthreads is None:
+        compthreads = ui.configint(b'experimental', b'bundlecompthreads')
+    if compthreads is not None:
+        compopts[b'threads'] = compthreads
+
     # Allow overriding the bundling of obsmarker in phases through
     # configuration while we don't have a bundle version that include them
     if repo.ui.configbool(b'experimental', b'evolution.bundle-obsmarker'):



To: joerg.sonnenberger, #hg-reviewers
Cc: mercurial-patches, mercurial-devel