Producing Mercurial wheels from Github Actions + modernize build process

[PIERRE AUGIER]

I realize that this email is actually about three different and decoupled issues

- wheels on PyPI
- support Python 3.13 (no distutils)
- how to be able to distribute with PyPI Mercurial with C extensions and Mercurial with Rust extensions.

----- Mail original -----
> De: "Pierre-Yves David" <pierre-yves.david at ens-lyon.org>
> À: "PIERRE AUGIER" <pierre.augier at univ-grenoble-alpes.fr>, "mercurial" <mercurial at lists.mercurial-scm.org>
> Envoyé: Vendredi 27 Septembre 2024 22:41:53
> Objet: Re: Producing Mercurial wheels from Github Actions + modernize build process

> On 9/27/24 09:14, PIERRE AUGIER wrote:
>> Hello,
>>
>> I tried to see if Mercurial wheels could be produced on Github Actions.
>>
>> https://github.com/paugier/mercurial-devel/blob/refs/heads/wheels-with-github-actions/.github/workflows/wheels.yml
>>
>> Nothing works because simple standard commands do not work for Mercurial.
> 
> 
> I am not surprised. As you noticed our setup.py is really ancient and
> has really not caught up with evolution of python packaging of the last
> 15 years.
> 
>> https://github.com/paugier/mercurial-devel/actions/runs/11064702442
>>
>> Even producing the sdist fails:

My understanding of why it failed was wrong. The problem was actually that Mercurial setup.py assumes that the source is either a mercurial repository or an archive, and not a git mirror!

I somehow fixed the version detection in this case, and I'm able to build a lot of wheels on Github Actions.

So I created a MR https://foss.heptapod.net/mercurial/mercurial-devel/-/merge_requests/974 to better discuss about this possibility.

Currently, there are still issues but I think it is promising since a lot of wheels appear to be correctly built (to be tested though): Windows, macos (x86_64, arm64), linux (x86_64, arm64) for different implementations (CPython and PyPy) and different versions (3.9 to 3.12) of Python.

There are few issues (like Windows and PyPy) but overall it works.

See https://github.com/paugier/mercurial-devel/actions

I ask few questions in this MR :

- Is such strategy to get wheels uploaded on PyPI for each new release could be acceptable to Mercurial core developers? Is there a reasonable alternative to automatically build wheels for Windows and the different flavors of macos?

- @gracinet do you think heptapod/heptapod#355 could be implemented quite soon so that we can get a good mirror of https://foss.heptapod.net/mercurial/mercurial-devel on Github?

>> I had a look at the Makefile and setup.py. It seems that they contain a lot of
>> legacy code and that a simplification and modernization would be welcome. For
>> example, setup.py should never be called directly (and it is even better if we
>> can avoid it).
>>
>> I guess I don't understand why setup.py has to be so complex and I suspect that
>> it would much simpler with another build system, for example Meson (used by
>> Numpy, Scipy and many other projectshttps://mesonbuild.com/Users.html).
>>
>> I have a bit of experience with using Meson so I can help if one wants to give
>> it a try for Mercurial build.
> 
> 
> I don't think we need to look at a new build system yet. Mercurial is
> basically still using 2025's distutils. Modern python packaging is
> likely to be able to do what we need.
> 
> However this means the setup.py need to be understand and migrated to
> something more modern. If you are willing to give it a try, I would be
> happy to do a couple of call to help you understand what's in there.
> 
> (note: I am not against using something else, but I suspect we don't
> need to)

I agree that you don't need to. What you need is to stop using distutils, which was deprecated 4 years ago (https://peps.python.org/pep-0632/).

The simplest way to do that would be to just use setuptools.

This should be done along with a big clean up of setup.py.

In particular :

- All the code about version computation should be moved in another Python script. (Also supporting installing Mercurial from a mercurial repo without hg installed might not be very useful).

- The code in setup.py related to testing and doc should just be removed.

- The static data given to the setup function should be moved to pyproject.toml.

However, it seems to me that from the point of view of "Modern python packaging", it would even be better to completely avoid setup.py and use a more modern solution than setuptools.
I think Meson is one of the best build system for Python packages with extensions.

Adopting Meson would make the build faster and it would help for supporting different cases by doing the same as popular projects like Numpy/Scipy/scikit-learn/scikit-image, etc. For example there is an issue with setuptools with building Mercurial on Windows with PyPy. 

>> Regarding Rust, if I understand correctly, it is now a build option leading to
>> two different Mercurial wheels. Meson-python can also have build options, but
>> this is not good practice for the PyPA. Did you think about having the Rust
>> extensions (and binaries) in another PyPI package (like mercurial-rust)?
> 
> We currently have 3 "flavor": "py", "c" and "c+rust". Pure does not
> compile anything, "c" use some compiled CPython extension, "c+rust" also
> have some Rust extensions.
> 
> I remember using build option to install different version of dulwich
> from PyPA, so  I guess we could still do. However, if we start to build
> whell, this open the way to shipping rust extension more broadly.

>From what I see (https://pypi.org/project/dulwich/#files), the different wheels are built on Github Actions with cibuildwheel (as for my MR https://foss.heptapod.net/mercurial/mercurial-devel/-/merge_requests/974).

It's not possible to have different flavors for wheels on PyPI.

The right way (from the point of view of the PyPA) to distribute with PyPI "Mercurial with C extensions" and "Mercurial with Rust extensions" is to have a package mercurial (with C extensions) and another package hg-rust, which could be an optional dependency of mercurial so that things like

pipx install mercurial[rust]

works. One would need in mercurial's pyproject.toml

[project.optional-dependencies]
rust = ["hg-rust"]

> While looking around I discovered `|cibuildwheel` that is compatible
> with Gitlab so it probably works fine with Heptapod and we could be
> building wheel from the main repository directly. … but we need to fix
> the setup.py first…|

Unfortunately, 95% of my students are on Windows or MacOS... So we still would need to find a solution (compatible with Mercurial) for automatic builds of Windows and MacOS wheels.

> Thanks for looking into this !
> 
> --
> Pierre-Yves David