About Mercurial and TortoiseHg native Windows applications

Tue Jan 21 11:01:22 UTC 2025

----- Mail original -----
> De: "Matt Harbison" <matt_harbison at yahoo.com>
> À: "PIERRE AUGIER" <pierre.augier at univ-grenoble-alpes.fr>
> Cc: "mercurial-devel" <mercurial-devel at mercurial-scm.org>
> Envoyé: Dimanche 19 Janvier 2025 07:05:03
> Objet: Re: About Mercurial and TortoiseHg native Windows applications

>> On Jan 10, 2025, at 9:02 AM, PIERRE AUGIER
>> <pierre.augier at univ-grenoble-alpes.fr> wrote:

>> ## Why native applications are useful
>> 
>> - The terminal commands on Windows have to be implemented with .exe files
>> (unlike on Unix where Python scripts can be used). .bat files are not a good
>> solution. The .exe files used by solutions like pipx, UV and pixi are not
>> signed (and cannot be signed).
> 
> Why not?  Is it just that they are stuffed directly into the wheel? (So in
> theory it could be expanded, signed, and then re-zipped)

The issue is about the main executable for a program (black.exe for example, placed in a directory in the PATH) which, for such solutions, are not an executable provided in the wheel but a .exe created by these tools by modifying a generic .exe with the absolute path of the Python environment used for this program.

>> ## About creation and distribution of native Windows applications
>> 
>> Maintainers have to do few things:
>> 
>> 1. Create the application (a binary executable .exe or a directory containing at
>> least one executable file .exe)
>> 
>> 2. Sign the binary (not absolutely necessary though, but for Mercurial the point
>> of an app is to distribute a signed executable)
> 
> It would definitely be nice to sign the executable and installer. I’m not sure
> of the details on this though, or how it would work for an open source project.
> I know at work, the signing works with a single physical dongle.  But we sign
> drivers, so maybe it’s just a matter of we don’t need a software only signing
> solution because the dongle can do both drivers and apps.  I’m pretty much the
> only Windows dev, but the project needs to account for the bus factor.
> 
> It would be nice to sign the Mac installer (and thg app), but that’s requires a
> yearly developer subscription.  No need (or ability) to sign hg on Mac, unless
> we bundle a private python like we did with py2.

It seems to me that at least for hg solutions like pipx are fine on macOS since on Unix pipx (and similar tools) can just use a .py file and a sheban.

>> ## What is the current situation for Mercurial

> The other thing to be aware of with the thg installer is that it also bundles a
> shell extension to indicate file status and provide a context menu in Explorer.
> It also installs a pile of help/documentation that is based on the old Windows
> 98(?) era help.  We can get rid of that if needed, but the shell extension is
> nice to have.

What is this shell extension is practice?

> One other random thing I noticed is that the bundled hgrc files aren’t visible
> in a venv install (or at least an editable one with thg- I only noticed because
> thg has more args on some of its merge-tool configs, and not picking that up
> resulted in different behavior for the visual diff command.  Not sure how to
> replicate that situation in pure hg).

Where are these files? Are they stored in wheels? If we want hg to be sensible to a .hgrc file in a venv, wouldn't it be possible to just adapt hg ?

>> - PyOxidizer has been used to build Mercurial Windows native applications. I
>> don't exactly know why but it does not work anymore with recent Mercurial
>> versions.
> 
> The problem there was it was using the —no-pep517 option internally.  I was able
> to get it working again thanks to your landed setup.py work.
> 
>> The disadvantage of PyOxidizer is that few things do not work like in standard
>> Python on Windows. Moreover, PyOxidizer is complicated and seems unmaintained
>> since its creator declared that he stopped his implication in the project.
> 
> Yes, this and py39 seems to be the latest embedded python for it.

Interesting. This is a strong argument in favor of leaving PyOxidizer. Python 3.9 will be end of life this year (https://devguide.python.org/versions/).

>> ## Questions for Mercurial/TortoiseHg devs
>> 
>> - Are the hg executables in current Windows apps signed?
> 
> Unfortunately, no.  The build scripts have args to do the signing, but we lack a
> certificate.

It seems to me that for Mercurial (in contrast to thg) the main advantage of a native application is to be able to sign the main hg.exe executable, so it would be very good to get a certificate for Mercurial. However, I don't know anything about Windows certificate.

>> ## What's next
>> 
>> Windows apps and installers for Mercurial and TortoiseHg have to be produced for
>> recent versions.
> 
> It would be interesting to see if the Inno installer script in the repo still
> works with the recent setup.py changes. I might try that if I get time over the
> weekend.  The pyoxidizer exe at least builds, but I think the installer script
> needs to be run to make the MSI.

>> Since one needs to work on the process to create them, it seems to me that
>> modern methods could be considered, namely an app based on the Windows
>> embeddable Python package for TortoiseHg and maybe an app based on pyapp for
>> Mercurial. However, if the work is done to produce an app from the Windows
>> embeddable Python package for TortoiseHg, it should be very easy to adapt it
>> for Mercurial.
> 
> It might be easier to do the other way around- do hg first, then thg, as long as
> the hg changes don’t outright break thg much.  The setup.py situation there is
> way worse (e.g. we have a separate setup.py for building the Mac app, because
> the main one is such a mess), and we have to figure out how to account for the
> shell extension.

Yes, it seems reasonable to start by producing a new Windows app for Mercurial with a modern method, i.e. from the wheels, as described in:

- https://stevedower.id.au/blog/build-a-python-app
- https://github.com/zooba/pymanager
- https://github.com/pypa/packaging.python.org/pull/1778

I can contribute if it is useful. For the installer, Inno seems fine. 

IMHO, we first need to finalize other tasks related to packaging. I have two MR which are somehow ready

- https://foss.heptapod.net/mercurial/mercurial-devel/-/merge_requests/1210
- setuptools-scm https://foss.heptapod.net/mercurial/mercurial-devel/-/merge_requests/1207

and these ones would be very useful:

- PYTHONLEGACYWINDOWSSTDIO https://foss.heptapod.net/mercurial/mercurial-devel/-/merge_requests/1193
- hg as a standard console-script Python entry point https://foss.heptapod.net/mercurial/mercurial-devel/-/merge_requests/1206

An online Windows runner would be useful to go forwards.

Static MANIFEST.in would also be satisfactory. As using setuptools-rust.

>> For TortoiseHg, a conda-forge based installer similar to the one of Spyder could
>> also be an interesting solution. However, there is the question of how to sign
>> `hg.exe` in this case.
> 
> It would be great to have wheels and whatever for thg, especially on Linux,
> because the current experience there is to clone the repo, and then run
> setup.py to build the _ui.py files (even though the documentation says not to),
> so it’s a real pain point.  The Mac app is a very special/complicated thing,
> and I don’t expect other tools to be able to replicate it, or users there to
> install a wheel or launch from the command line there.
> 
>> An aspect to consider is the ability to use pip, UV or conda (from a new
>> Mercurial command) to manage (install/uninstall/update) Mercurial extensions.
>> Some methods could allow that and others not (in particular I don't think it
>> would be possible with PyOxidizer).
> 
> There was talk about having a command to install and manage extensions several
> years ago.  I think it’s a good idea.  I tinkered with in a bit on windows, ran
> into some issues invoking python on Windows, and set it aside.  I forget what
> the problems were now, but I can probably find the code if/when it gets to the
> point of implementing this.

This question of a Mercurial command to install/update/uninstall Mercurial extension depends a bit where Mercurial wants to go and the importance of extensions in future.