How can isdir give unstable results?

[Johannes]

Dear all

I've a strange effect of clones failing because os.path.isdir giving
wrong results. I'm looking for pointers how to find root cause of this
failure.

TLDR: cloning on Windows from a network share only succeeds when
using argument --pull. Otherwise it will fail with unexpected
network error.

Detailed description (sorry, this is somewhat long)

This all started when switching server from Windows 7 to Samba. You
may think: go away, this ML is about hg, it's not about Samba and
network problems. But please bear with me and read on.

It does not happen on all repositories, only some are effected by
this.

First I looked into this by firing up Process Monitor to see what went
wrong during cloning.

Cloning process looked OK up to one point where hg was requesting a
directory listing for an *.i file somewhere down the path. Of course
this cannot work.

Then I fired up Wireshark to look at smb2 packets. For sure Samba was
giving correct information about given *.i file to be a regular file
and not a directory.

OK, now I tried to find out how all this looks like on mercurial side
of things.

For this I downloaded "python-3.8.7-embed-win32.zip", extracted it to
some directory and copied some pip files into Lib/site-packages. This
way I could

    python.exe -m pip install mercurial

I got mercurial-5.6.1 installed in Lib/site-packages and started to
look around. To produce the failure I did

    Scripts\hg.exe clone Z:/some/repository

and for sure at some point it failed with a network error.

Adding some print statements finally guided me to copyfiles in
util.py. The failure happens in

    util.py:1956    if os.path.isdir(src):

This call is /sometimes/ giving a wrong result during recursion. What
makes me wonder is this: if I call os.path.isdir in a separate .py
script I've never observed a wrong a result. Calling

    os.path.isdir(b'/server/some/repository/.hg/store/data/some/file.i')

in a loop 1000 times always reports False. The file path is copied from
my debug output print from util.py for given file that failed during clone.

Interestingly also 'kind' as resulting from listdir(src) always reports
correct value.

BTW hardlink is constantly None in all these calls so I don't think
it has anything to do with hardlink problems on network shares where
--pull used to be a workaround for several years ago.

The ugly workaround:

diff -u mercurial/util.py.orig mercurial/util.py
--- a/mercurial/util.py	2021-01-31 14:41:07.493332832 +0100
+++ b/mercurial/util.py	2021-02-07 15:43:59.307098253 +0100
@@ -1964,6 +1964,11 @@
         for name, kind in listdir(src):
             srcname = os.path.join(src, name)
             dstname = os.path.join(dst, name)
+            dirresult = True
+            if kind != 16384:
+                dirresult = False
+            while os.path.isdir(srcname) != dirresult:
+                time.sleep(0.1)
             hardlink, n = copyfiles(srcname, dstname, hardlink, progress)
             num += n
     else:

This will wait for os.path.isdir to report correct result as
determined by listdir before going into recursive call to
copyfiles. Doing so makes hg clone succeed.

I've tried different amount of delays in time.sleep() together with
counting how often time.sleep was called. If os.path.isdir is giving a
wrong result the correct result will show up after 10 seconds of total
delay.

Summary

I seem to observe some strange probably timing related effect, most
probably bound to my specific network setup. I can't observe the
failure in network packets (Wireshark) and also not outside mercurial
(os.path.isdir gives correct results in separate Python script).

Any pointers on how to narrow down this strong behavior are highly
welcome.


Johannes











-- 
SHA1 Fingerprint=84:EF:6B:44:23:DC:62:11:15:F4:C6:32:CD:E1:EA:41:EF:0A:EF:8B
http://www.room40.de/ca/policy.html