Fwd: Producing Mercurial wheels from Github Actions + modernize build process

Wed Oct 9 07:42:06 UTC 2024

I tried to send this email to mercurial-devel but apparently it was not accepted and published. I send it here (sorry for the noise).

I also complete with some good news about my experiments with Github Actions and Mercurial!

----- Mail transféré -----
> De: "PIERRE AUGIER" <pierre.augier at univ-grenoble-alpes.fr>
> À: "mercurial-devel" <mercurial-devel at mercurial-scm.org>
> Envoyé: Lundi 7 Octobre 2024 15:05:09
> Objet: Re: Producing Mercurial wheels from Github Actions + modernize build process

> Hi mercurial-devel,
> 
> We have now a good demo that it would be possible and easy to produce and upload
> wheels to PyPI with Github Actions.
> 
> I first thought that this could be useful for regular CI with a fully automatic
> setup for releases (hence
> https://foss.heptapod.net/mercurial/mercurial-devel/-/merge_requests/974).
> 
> However, after some feedback from George, I see that there could be an
> intermediate solution based on manually triggered workflows on Github Actions
> in a tiny repo hosted on Github (like
> https://github.com/paugier/mercurial-ga-ci).
> 
> We could have 2 manual workflows, test.yml and release.yml.
> 
> - Before the release, the release manager can trigger the test.yml workflow,
> that downloads an archive created from the stable branch, tries to create
> wheels and tests them.
> 
> - When everything is fine on all CI, a new tag is pushed on the main repo. An
> sdist is created on Heptapod CI and stored somewhere (but not on PyPI).
> 
> - The release manager can then trigger the release.yml workflow which download
> the right sdist, create the wheels, test them and upload the wheels (first) and
> the sdist (second) on PyPI.
> 
> There is a bit of human actions, but very limited so not really time consuming
> and hard to introduce human errors.
> 
> An advantage is that there is no need to link Heptapod and Github
> programmatically, so it's very simple.
> 
> Since it is so simple, I think it could be setup in few days.
> 
> However, I now see that there could also be a nicer solution to use Github
> Actions for regular CI (advantage Windows and macos for free) without relying
> on hg-git.
> 
> We could cache the whole mercurial repo on Github Actions and just use hg to get
> the source of the reference that has to be tested. I can't see why it would not
> work.

It actually works! See 

- https://github.com/paugier/mercurial-ga-ci-cache-repo/blob/main/.github/workflows/ci.yml
- https://github.com/paugier/mercurial-ga-ci-cache-repo/actions/runs/11248552560

With cache, the Mercurial mercurial-devel repo is obtained and updated in only 12s! The sdist and all wheels are produced directly from the repo, without hg-git conversion.

A question for Mercurial developers: how should we test the wheels? cibuildwheel has few options related to testing https://cibuildwheel.pypa.io/en/stable/options/#testing

> Of course, it leaves open the problem of how to trigger the workflow and to send
> the information on the reference. And the problem of how one could take into
> account the result of a Github Actions workflow from Heptapod.
> 
> I would be interested to know what mercurial developers think about these
> possibilities.
> 
> 
> I would also like to know what is your opinion on the Rust extensions. Is it
> still necessary to upload wheels without Rust or can we only upload wheels with
> Rust (so that all users using Mercurial installed from PyPI would have by
> default the Rust extensions)? If it is still necessary for some cases to use
> Mercurial without Rust, I really think Rust extensions should be moved in a
> separate PyPI project.
> 
> Regards,
> Pierre A.
> 
> ----- Mail original -----
>> De: "Pierre-Yves David" <pierre-yves.david at ens-lyon.org>
>> À: "PIERRE AUGIER" <pierre.augier at univ-grenoble-alpes.fr>, "mercurial"
>> <mercurial at lists.mercurial-scm.org>
>> Cc: "mercurial-devel" <mercurial-devel at mercurial-scm.org>
>> Envoyé: Lundi 7 Octobre 2024 11:09:18
>> Objet: Re: Producing Mercurial wheels from Github Actions + modernize build
>> process
> 
>> I just realized that this discussion is taking place on the user mailing
>> list. We should move it on the developer mailing list.
>> 
>> On 9/27/24 09:14, PIERRE AUGIER wrote:
>>> Hello,
>>>
>>> I tried to see if Mercurial wheels could be produced on Github Actions.
>>>
>>> https://github.com/paugier/mercurial-devel/blob/refs/heads/wheels-with-github-actions/.github/workflows/wheels.yml
>>>
>>> Nothing works because simple standard commands do not work for Mercurial.
>>>
>>> https://github.com/paugier/mercurial-devel/actions/runs/11064702442
>>>
>>> Even producing the sdist fails:
>>>
>>> Run python -m build --sdist
>>> * Creating isolated environment: venv+pip...
>>> * Installing packages in isolated environment:
>>>    - setuptools
>>>    - wheel
>>> * Getting build dependencies for sdist...
>>> /!\
>>> /!\ Could not determine the Mercurial version
>>> /!\ You need to build a local version first
>>> /!\ Run `make local` and try again
>>> /!\
>>> Run `make local` first to get a working local version
>>>
>>> And same error for wheels:
>>>
>>> python -m pip wheel /project --wheel-dir=/tmp/cibuildwheel/built_wheel --no-deps
>>>    Processing /project
>>>      Installing build dependencies: started
>>>      Installing build dependencies: finished with status 'done'
>>>      Getting requirements to build wheel: started
>>>      Getting requirements to build wheel: finished with status 'error'
>>>      error: subprocess-exited-with-error
>>>      
>>>      × Getting requirements to build wheel did not run successfully.
>>>      │ exit code: 1
>>>      ╰─> [6 lines of output]
>>>          /!\
>>>          /!\ Could not determine the Mercurial version
>>>          /!\ You need to build a local version first
>>>          /!\ Run `make local` and try again
>>>          /!\
>>>          Run `make local` first to get a working local version
>>>          [end of output]
>>>
>>>
>>> I had a look at the Makefile and setup.py. It seems that they contain a lot of
>>> legacy code and that a simplification and modernization would be welcome. For
>>> example, setup.py should never be called directly (and it is even better if we
>>> can avoid it).
>>>
>>> I guess I don't understand why setup.py has to be so complex and I suspect that
>>> it would much simpler with another build system, for example Meson (used by
>>> Numpy, Scipy and many other projectshttps://mesonbuild.com/Users.html).
>>>
>>> I have a bit of experience with using Meson so I can help if one wants to give
>>> it a try for Mercurial build.
>>>
>>> Regarding Rust, if I understand correctly, it is now a build option leading to
>>> two different Mercurial wheels. Meson-python can also have build options, but
>>> this is not good practice for the PyPA. Did you think about having the Rust
>>> extensions (and binaries) in another PyPI package (like mercurial-rust)?
>>>
>>> Pierre
>>>
>>> --
>>> Pierre Augier - CR CNRShttp://www.legi.grenoble-inp.fr
>>> LEGI (UMR 5519) Laboratoire des Ecoulements Geophysiques et Industriels
>>> BP53, 38041 Grenoble Cedex, Francetel:+33.4.56.52.86.16
>>> _______________________________________________
>>> Mercurial mailing list
>>> Mercurial at lists.mercurial-scm.org
>>> https://lists.mercurial-scm.org/mailman/listinfo/mercurial
>> 
>> --
> > Pierre-Yves David