About Mercurial and TortoiseHg native Windows applications

[Matt Harbison]

> On Jan 10, 2025, at 9:02 AM, PIERRE AUGIER <pierre.augier at univ-grenoble-alpes.fr> wrote:
> 
> Hello Mercurial developers,
> 
> I invested a bit of time on Mercurial in the hope of improving its situation for the next years.
> 
> Good news: Mercurial on the default branch now (after https://foss.heptapod.net/mercurial/mercurial-devel/-/merge_requests/1131) supports the basic of modern Python packaging (PEP 517), which involves "PEP 517 frontends" (pip, pipx, UV, PDM, ...), "PEP 517 backends" (setuptools in the case of Mercurial) and "isolated builds". There are still things to be fixed in terms of packaging (in particular the Windows wheels), but it seems doable and actually not so difficult.

Thanks for doing this.  It’s a hassle, which is probably why nobody has bothered with it.

> I recently worked a bit on the situation on Windows and learned why native Windows applications are so important and how they can be built.
> 
> I asked few questions regarding this subject on https://discuss.python.org (in particular https://discuss.python.org/t/how-to-create-a-window-command-line-application-and-its-installer-from-wheels/76136/4) and what I learned could be interesting for Mercurial.

It’s interesting that our exewrapper.c is almost nothing like the C code they came up with.  The embedded python in a subdirectory might be the modern equivalent to the “hackable” mercurial references you might come across, but I never used it and don’t know the details of it.

> I'd like with the email to open a discussion on this subject, share few useful links and ask few questions.
> 
> ## Why native applications are useful
> 
> - Some Windows users like installers (download by clicking, execute, everything is installed). Other are more happy with the Microsoft Store. Both solutions require a native Windows application.
> 
> - The terminal commands on Windows have to be implemented with .exe files (unlike on Unix where Python scripts can be used). .bat files are not a good solution. The .exe files used by solutions like pipx, UV and pixi are not signed (and cannot be signed).

Why not?  Is it just that they are stuffed directly into the wheel? (So in theory it could be expanded, signed, and then re-zipped)

> - These .exe files have to be signed to please the antivirus software (otherwise there can be big startup delays of the other of hundreds of milliseconds). Note that this is not particularly an issue related to Python. chg/rhg would also suffer from this potential issue.
> 
> Therefore native applications are useful on Windows... And much less on Unix.
> 
> ## About creation and distribution of native Windows applications
> 
> Maintainers have to do few things:
> 
> 1. Create the application (a binary executable .exe or a directory containing at least one executable file .exe)
> 
> 2. Sign the binary (not absolutely necessary though, but for Mercurial the point of an app is to distribute a signed executable)

It would definitely be nice to sign the executable and installer. I’m not sure of the details on this though, or how it would work for an open source project. I know at work, the signing works with a single physical dongle.  But we sign drivers, so maybe it’s just a matter of we don’t need a software only signing solution because the dongle can do both drivers and apps.  I’m pretty much the only Windows dev, but the project needs to account for the bus factor.

It would be nice to sign the Mac installer (and thg app), but that’s requires a yearly developer subscription.  No need (or ability) to sign hg on Mac, unless we bundle a private python like we did with py2.

> 3. Create an installer (from nothing - manual download of the app - to .msi or MSIX installers)
> 
> 4. Distribute the installer (from just put it on the web to register to the Microsoft Store)
> 
> Many different tools can be used to help for these steps.
> 
> ## What is the current situation for Mercurial
> 
> - TortoiseHg applications are based on py2exe, an old project which creates native Windows applications through a distutils extension. Distutils does not exist anymore in Python >=3.12 so it is actually a setuptools extension.
> 
> The advantage of this approach is that (if I understand correctly) py2exe tries to automatically remove from the application parts of Python (and other libraries) which are not used.
> 
> The disadvantage of py2exe is that it is by design (being a distutils/setuptools extension) not up-to-date with the improvements of Python packaging (more on this in the next section). Therefore, Mercurial's setup.py currently contains py2exe related hacks whereas it should be used only to create a source distribution (sdist) and wheels. Note that even from the point of view of py2exe, using setup.py is deprecated (https://github.com/py2exe/py2exe?tab=readme-ov-file#usage).

I saw the reference to deprecating setup.py in newer releases, so I haven’t messed with them.  Not sure how well it (or py2app on Mac) will work with your recent changes.  I wonder if we can sidestep the py2exe special code by installing hg into a venv, and somehow pointing py2exe at that.

The other thing to be aware of with the thg installer is that it also bundles a shell extension to indicate file status and provide a context menu in Explorer.  It also installs a pile of help/documentation that is based on the old Windows 98(?) era help.  We can get rid of that if needed, but the shell extension is nice to have.

One other random thing I noticed is that the bundled hgrc files aren’t visible in a venv install (or at least an editable one with thg- I only noticed because thg has more args on some of its merge-tool configs, and not picking that up resulted in different behavior for the visual diff command.  Not sure how to replicate that situation in pure hg).

> - PyOxidizer has been used to build Mercurial Windows native applications. I don't exactly know why but it does not work anymore with recent Mercurial versions.

The problem there was it was using the —no-pep517 option internally.  I was able to get it working again thanks to your landed setup.py work.

> PyOxidizer produces from a Python project a Rust binary containing the Python interpreter.
> 
> The advantage of the approach is that you get a one-file application (hg.exe), which cannot be modified after build. The one-file nature is actually a quite weak advantage. All Windows installers know how to handle a .zip file by unzip it in the right place and adding the right path in the PATH. Users don't see that the resulting app is a directory (and most Windows apps are actually installed as directories).

The one file thing was definitely interesting, but in practice we don’t use it.  Both the windows and Mac installers had hg.exe with an embedded python, but the data files (and I think the *.py files) live in the file system.  I think I saw a reference recently that the all-in-one-exe mode is considered deprecated.

> The disadvantage of PyOxidizer is that few things do not work like in standard Python on Windows. Moreover, PyOxidizer is complicated and seems unmaintained since its creator declared that he stopped his implication in the project.

Yes, this and py39 seems to be the latest embedded python for it.

> Moreover, this is not the method proposed by Microsoft engineers working on Python and Python packaging.
> 
> ## Alternative methods and tools
> 
> ### What are native Windows Python apps and how to create them?
> 
> An important point to realize is that native Windows Python apps are conceptually very simple. It is "just" a directory with
> 
> 1. a launcher (for Mercurial hg.exe) written in a native language (usually C, or Rust if one wants but it is so simple that it does not seem so useful for this application)
> 2. a directory with a Python interpreter (one should download the official "Windows embeddable package" from https://www.python.org/downloads/windows/)
> 3. another directory with the Python packages (an important point is that they can be installed here from wheels with just a `pip install`)
> 
> So it's quite simple to write by hands few scripts to create with few commands a native app. There are also modern tools to help.
> 
> Few links about this possibility:
> 
> - https://discuss.python.org/t/how-to-create-a-window-command-line-application-and-its-installer-from-wheels/76136
> - https://stevedower.id.au/blog/build-a-python-app
> - https://github.com/pypa/packaging.python.org/pull/1778 (doc PR, preview result at https://python-packaging-user-guide--1778.org.readthedocs.build/en/1778/)
> 
> Three modern tools to help:
> 
> - https://github.com/zooba/pymsbuild
> - https://beeware.org/project/projects/tools/briefcase/
> - https://pynsist.readthedocs.io
> 
> ### pyapp: another alternative, very simple and potentially usable for Mercurial
> 
> [pyapp](https://github.com/ofek/pyapp) is an alternative solution to create (with Rust) another type of native application to use a Python CLI (like hg).
> 
> The principle is quite different because it creates a one-file app (hg.exe in our case) which is both a Mercurial installer (using UV in the background) and a Mercurial caller.
> 
> This is attractive for a simple Mercurial installer because it is VERY simple and easy to create (once Mercurial works with a console-script Python entry point and is installable with UV on Windows). I think we could have the app and the installer after literally few minutes of work.
> 
> ### Windows apps and installers based on conda-forge
> 
> Spyder uses this other approach for its "Standalone installers" (https://docs.spyder-ide.org/current/installation.html#standalone-installers) and it works very well. See https://github.com/spyder-ide/spyder/tree/master/installers-conda.
> 
> ## Questions for Mercurial/TortoiseHg devs
> 
> - Are the hg executables in current Windows apps signed?

Unfortunately, no.  The build scripts have args to do the signing, but we lack a certificate.

> - Does the Mercurial project have what is needed to sign executable?
> 
> - Did I misunderstand something?
> 
> ## What's next
> 
> Windows apps and installers for Mercurial and TortoiseHg have to be produced for recent versions.

It would be interesting to see if the Inno installer script in the repo still works with the recent setup.py changes. I might try that if I get time over the weekend.  The pyoxidizer exe at least builds, but I think the installer script needs to be run to make the MSI.

> Since one needs to work on the process to create them, it seems to me that modern methods could be considered, namely an app based on the Windows embeddable Python package for TortoiseHg and maybe an app based on pyapp for Mercurial. However, if the work is done to produce an app from the Windows embeddable Python package for TortoiseHg, it should be very easy to adapt it for Mercurial.

It might be easier to do the other way around- do hg first, then thg, as long as the hg changes don’t outright break thg much.  The setup.py situation there is way worse (e.g. we have a separate setup.py for building the Mac app, because the main one is such a mess), and we have to figure out how to account for the shell extension.

> For TortoiseHg, a conda-forge based installer similar to the one of Spyder could also be an interesting solution. However, there is the question of how to sign `hg.exe` in this case.

It would be great to have wheels and whatever for thg, especially on Linux, because the current experience there is to clone the repo, and then run setup.py to build the _ui.py files (even though the documentation says not to), so it’s a real pain point.  The Mac app is a very special/complicated thing, and I don’t expect other tools to be able to replicate it, or users there to install a wheel or launch from the command line there.

> An aspect to consider is the ability to use pip, UV or conda (from a new Mercurial command) to manage (install/uninstall/update) Mercurial extensions. Some methods could allow that and others not (in particular I don't think it would be possible with PyOxidizer).

There was talk about having a command to install and manage extensions several years ago.  I think it’s a good idea.  I tinkered with in a bit on windows, ran into some issues invoking python on Windows, and set it aside.  I forget what the problems were now, but I can probably find the code if/when it gets to the point of implementing this.

> Pierre
> 
> --
> Pierre Augier - CR CNRS                 http://www.legi.grenoble-inp.fr
> LEGI (UMR 5519) Laboratoire des Ecoulements Geophysiques et Industriels
> BP53, 38041 Grenoble Cedex, France                tel:+33.4.56.52.86.16
> _______________________________________________
> Mercurial-devel mailing list
> Mercurial-devel at lists.mercurial-scm.org
> https://lists.mercurial-scm.org/mailman/listinfo/mercurial-devel