Fwd: Mercurial/TortoiseHg py3 TODO list

Thu Nov 5 17:05:27 UTC 2020

This is a list of things to cleanup that I sent Gregory offline a few
weeks ago.  I think we covered most of this in the sprint this
morning, but it's a good idea to have them written down.  Since then,
I've seen that mercurial_keyring has some py3 issues[6], and I'm not
sure if SCM Manager [7] supports py3 Mercurial yet.  The latter isn't
directly our problem, but we may care due to the general lack of
hosting solutions.

- Pip installed Mercurial puts the setuptools generated `hg` on PATH.
I think that invokes python.exe to run it, but it's useless because it
doesn't set the various python options that hg.exe does to make it
work, like legacy stdio.  I tried getting setup.py to build hg.exe
without that, hoping it would end up on PATH, but failed (and don't
remember why_ I'm sure I have patches somewhere.)

- We currently ship hg.exe with the Windows manifest to enable long
file paths.  There's a feature request for this in PyOxidizer[1], and
we probably shouldn't regress this (though IDK how many people need it
in practice).

- There are currently a handful of py3 specific failures in CI, mostly
around the lock file.  I assume this is trivial since most commands
don't fail (so locking must work), but I haven't looked at it.

- Not a user facing issue, but when I build py3 stuff with `make
local`, I routinely hit small hassles like a dialog that says it can't
find python38.dll, or the PyOxidizer files can't be cleaned up with
`make clean` because the paths are too long.  The former is because I
kept python off of PATH and use the py.exe launcher.  Maybe there's
something we can do in setup.py to help that.  And maybe a command for
cleaning in PyOxidizer takes care of the latter (the former is a much
bigger problem).

That might be it for core Mercurial issues, though I'm always
concerned about what test coverage we lack.  As far as TortoiseHg
issues:

- The build system is too complicated.  There's a thgbuild
repo that clones in a bunch of extensions, hg, and thg, and then a
winbuild repo that has its own setup.py that in turn uses some *.wxs
files in the original thg repo.  It is *extremely* brittle, with all
of the DLLs that need to be manually included/excluded.  I'd love to
burn that all down and put something in thg's contrib/ like you've got
for building hg installers.  I ripped some of that off to script
building a venv, qt5, etc with py2.  The good news is I think we can
just grab PyQt5 and friends off of pypi to create the virtualenv,
instead of building that from source.

- We need to jettison py2exe and use PyOxidizer on Windows

- We're currently using Qt5.9 on Windows and Mac.  I was going to try
to upgrade to Qt5.15 for the November release, but I probably won't
bother if we're looking to switch to py3 in the next cycle or so.  The
point being, IDK if there are any regressions in 5.15.  We've
currently got a hack for launching the terminal on Windows because of
a bug in 5.9, for example.

- I'm pretty sure the non-workbench commands are mostly broken in py3.
(You can `thg commit` from the command line to bring up the commit
dialog, for example.)  This isn't a priority for me, because I never
use them, but it's something to acknowledge.

- There's a bunch of Qt4 code that needs to be dropped, both in the
codebase and setup.py.  I've been holding off, thinking that it might
help with highlighting what areas need to be conditionalized for Qt6,
but maybe that's not a good idea.

- The default fonts have changed, and it's no longer a fixed width in
the commit widget.  (This might be a Windows only issue.)  That throws
off the 80 character red line, and makes it unusable for commiting.  I
submitted a patch[2] that Yuya rejected, saying it must be a bug with
the python binding[3].  Not sure where to go with it.

- I'm guessing PyOxidizer doesn't handle macOS apps, so I hacked
enough with py2app to build the .app for macOS, but it doesn't run[4].

- There's a series of scripts to build the macOS app, and that will
need to be updated accordingly.

- I've seen issues when switching between py2 and py3 where the repo
list gets corrupted, tabs fail to load, and/or duplicate tabs appear
(the difference seems to only be uppercase vs lower case drive
lettering).  While maybe not a problem when switching over completely
to py3, it makes me wonder if something isn't being encoded to the
config files properly.  (It's %APPDATA%\TortoiseHg\* files, not
mercurial.ini and friends)

- There is a seemingly endless stream of bytes vs str issues in the
code.  There aren't a whole lot of tests, so it's mostly a case of
needing to run the code and hit them.  I'm slowly adding type hints,
and I can see that there are issues in the custom tools code, as well
as with subrepo handling in reporegistry.  I can see that there are
similar issues in things I don't use like tortoisehg.util.gpg.  It
would likely help a lot if we got typing in Mercurial in some critical
areas like ui, config (I've got code for that), encoding, and
pycompat.  There are a handful of known issues on the bug tracker[5],
though I haven't gone in and added explicit py3 tags to them.

I'm sure there will be other issues, but they basically all boil down
to 1) building it, and 2) making it production quality.  If we can
resolve the commit widget text and the config file getting corrupted,
I'd be much more likely to routinely use it from a virtualenv to try
it out and find more issues.  But I need the py2 escape hatch so I'm
not debugging problems when I'm supposed to be working.

What really concerns me is I use a limited subset of it, and we don't
have a lot of tests.  I'd be hesitant to roll it out at work for a
while, for example.  So maybe we can figure out something where we
make the py3 builds more prominent to get more people to test it, but
still have py2 builds around for a bit as a lifeboat?

If/when I think of more issues, I'll try to write them down and send
them in a batch.  Any help you can provide will be much appreciated
because even with 2-3 weeks of vacation at the end of the year, I'm
certainly not going to clear all of this myself by January.

[1] https://github.com/indygreg/PyOxidizer/issues/197
[2] https://groups.google.com/d/msg/thg-dev/Jx43APKp4cE/UZS7SueTAQAJ
[3] https://groups.google.com/d/msg/thg-dev/Jx43APKp4cE/_oIFcvrCAQAJ
[4] https://groups.google.com/forum/?oldui=1#!topic/thg-dev/kL-2SHpY1v0
[5] https://foss.heptapod.net/mercurial/tortoisehg/thg/-/issues
[6] https://foss.heptapod.net/mercurial/mercurial_keyring/-/issues/74
[7] https://github.com/scm-manager