Speeding up Mercurial on NFS

[Matt Mackall]

On Mon, 2010-12-06 at 08:23 -0800, mg at lazybytes.net wrote:
> Hi everybody,
> 
> My client is migrating from ClearCase to Mercurial. Their repositories are
> too big for the individual developers machines, so they want to host
> everything (repository and working copy) on NFS.
> 
> The result is that operations like 'hg status' run much slower than the
> equivalent ClearCase command -- for one repository with 24k files, 'hg
> status' takes 8 seconds. They have some other repositories with 100k
> files.
> 
> Two years ago, someone asked if Git could be made to run faster over NFS.
> The result was a patch that made Git use threads to preload data in
> parallel. This gave a 5 time speedup:
> 
>   http://kerneltrap.org/mailarchive/git/2008/11/14/4089834/thread
> 
> 
> I have tried to replicate these results with some simple test programs:
> 
>   http://bitbucket.org/mg/parallelwalk
> 
> The 'walker' program is a single-threaded C-based program that will walk a
> directory tree as fast as possible, 'pywalker' is a Python-based version
> of it, but with support for multiple threads.

Hmm, to the extent that pywalker spends its time waiting in syscalls, it
might avoid the GIL.

> I have taken the OpenOffice repository and balanced the directories so
> that I have 8 directories of approximately the same size. There are 63k
> files in total. I then ran through these trees using 1-8 pywalker
> processes started in parallel:
> 
>   % timeit -i 1 'pywalker -q dir-1 & ... & pywalker -q dir-8 & wait'
> 
> I also timed the Python threads:
> 
>   % timeit -i 1 'pywalker -t 8 -q dir-[12345678] & wait'
> 
> The results of these tests:
> 
>       processes  threads
>   1:  24 sec     24 sec
>   2:  24 sec     19 sec
>   4:  26 sec     20 sec
>   6:  24 sec     19 sec
>   8:  23 sec     19 sec

What do your C results look like?

> A ~20-25% speedup by using at least two threads in parallel.
> 
> This is on a NFS setup where I export a directory and mount it back on
> localhost. I add an artificial network delay of 0.2 ms (0.1 ms in each
> direction) with
> 
>   % sudo tc qdisc change dev lo root netem delay 0.1ms
> 
> and I empty the disk cache before each run:
> 
>   % sync; sudo sh -c "echo 3 > /proc/sys/vm/drop_caches"

Is this the cache on the client or server?

> Compared to the speedups reported by Git users in the thread I linked
> above, a 25% speedup is very small. Does anybody have any idea for how
> this can be improved?
> 
> While investigating this, I discovered that the NFS protocol has two
> commands for listing filenames in a directory: READDIR and READDIRPLUS.
> The latter will also return stat information for each file found -- this
> saves a lot of round-trips. Unfortunately, I saw the kernel use
> READDIRPLUS only once in a while -- this is a shame since the speed
> advantage of READDIRPLUS is huge. Does anybody know if it is possible for
> a user-space program like Mercurial to influence how the kernel chooses
> each operation?

A good question. You should try tracing ls(1) or find(1) and see if it's
doing the right thing for starters. If not, you're probably screwed.

-- 
Mathematics is the supreme nostalgia of our time.