Producing Mercurial wheels from Github Actions + modernize build process

Fri Oct 4 20:25:05 UTC 2024

Hi George,

I agree that for a big repo like mercurial-devel, it makes sense to avoid the full conversion. However, for small repo, the Git mirroring is still useful, in particular for visibility.

For Mercurial, we "just" need:

- to have a job a Heptapod CI job that produces the sdist and pushes it somewhere where it can be downloaded. It is better because the sdist is produced from the hg repo.
- a tiny repo on Githut (basically only a README, license and workflow files) with a workflow triggered by foss.heptapod.net which downloads the sdist, builds the wheels and tests them.

We need to figure out how cibuildwheel can build wheels from a sdist and not from a repo but that should be quite easy. I can check that with a sdist from PyPI.

We need to figure out how a job on foss.heptapod.net can trigger a job on Github, and if it is possible to send a bit of data (where the sdist has to be downloaded and if it is a simple commit or a release).

For the release we can even download the sdist from PyPI. And we will even be able to use this workflow on a Mercurial version already released, to upload wheels for 6.8.1.

For the rust part, Mercurial core dev has to choose. Is it still necessary to upload wheels without Rust or can we only upload wheels with Rust (so that all users using Mercurial installed from PyPI would have by default the Rust extensions)? If it is still necessary for some cases to use Mercurial without Rust, I really think Rust extensions should be moved in a separate PyPI project.

----- Mail original -----
> De: "Georges Racinet" <georges.racinet at cloudcrane.io>
> À: "mercurial" <mercurial at lists.mercurial-scm.org>
> Envoyé: Vendredi 4 Octobre 2024 16:13:04
> Objet: Re: Producing Mercurial wheels from Github Actions + modernize build process

> Hi Pierre !
> 
> On 10/4/24 2:17 PM, PIERRE AUGIER wrote:
> 
> 
> 
> - @gracinet do you think heptapod/heptapod#355 could be implemented quite soon
> so that we can get a good mirror of [
> https://foss.heptapod.net/mercurial/mercurial-devel |
> https://foss.heptapod.net/mercurial/mercurial-devel ] on Github?
> I may have a better idea, but let me answer the direct question first:
> 
> 
> 
> 
> On one hand, I do not have time for much these days. On the other hand, nice
> wheels would certainly save me (if with Rust) and my coworkers quite a bunch of
> time, so I can consider it (probably not for the upcoming 17.5). Notably in
> maintenance of CI images, such wheels would make much easier to change version
> from the one provided in distro packages. I would probably go the virtualenv
> way everywhere and be happy with it.
> 
> I wouldn't make a UI though, as it is too time consuming and not necessary to
> accommodate Mercurial. I suppose that the maintainers would not have any
> problem setting it up with the API :-)
> 
> A simpler solution, perhaps more robust and less time-consuming / risky for me :
> 
> 
> 
> 
> Why not create a Git project on GitHub dedicated to building Mercurial only (not
> a mirror). Let's call it `hg-bin-build` for the time being.
> 
> 
> ts CI/CD actions could then just grab the relevant tarball from
> foss.heptapod.net (or pull from mercurial-scm.org) and perform the binary
> builds. I don't mean moving the actual build recipes in there, just to execute
> them. It would thus have only the GitHub action definitions a `VERSION` file.
> 
> 
> There are many advantages: no need to rely on hg-git etc to do that (and I do
> not have to dodge heptapod#1764). One could even imagine a CI job on
> foss.heptapod.net that triggers it in tag pipelines by committing and pushing
> (with Git).
> 
> There can be variations on the same idea. Here's a more self-hosted variant:
> have the `hg-bin-build` Git repo on foss.h.n (somewhat easier to use the proper
> tokens to push on it from a `mercurial-devel` CI job) and activate Git push
> mirroring to GitHub.
> 
> If noone has objections with such a plan, I would prefer it. Switching back to
> using Mercurial-to-Git push mirroring (heptapod#355) would still be possible in
> the future if people find it more satisfying.
> 
> In terms of PR, one could say that Mercurial is depending on Git, and that is to
> be avoided. My take is that this use case is well bounded and actually should
> be considered on the positive side as a demo that using Mercurial in a
> Git-centric world is perfectly possible.
> 
> 
> 
> 
> 
> It's not possible to have different flavors for wheels on PyPI.
> 
> The right way (from the point of view of the PyPA) to distribute with PyPI
> "Mercurial with C extensions" and "Mercurial with Rust extensions" is to have a
> package mercurial (with C extensions) and another package hg-rust, which could
> be an optional dependency of mercurial so that things like
> 
> pipx install mercurial[rust]
> 
> works. One would need in mercurial's pyproject.toml
> 
> [project.optional-dependencies]
> rust = ["hg-rust"]
> This would be fine by me, and indeed a huge progress. A practical example: in
> the Heptapod Development Kit, managing of dependencies is a real pain (the
> whole setup is been done by a collection of Makefiles, not my choice) and it
> happens that packages depending on Mercurial get installed hence also Mercurial
> (without Rust) before the target that is supposed to install Mercurial (with
> Rust), and there is no way to replace that with the proper version other than
> uninstalling the wrong one. Of course I tend not to notice immediately, only
> when some tests start failing because the repos have the nodemap. Worse, there
> is sometimes a cached wheel (without Rust). I would be really happy to just
> depend on `hg-rust` or whatever the end name happens to be.
> 
> Thank you for your efforts on this subject, it seems we are not so far from a
> solution.
> 
> 
> Side note: well yes I would be happy if we could have Windows and MacOS runners
> on foss.heptapod.net. Our past experience with the former is that they are
> awfully time-consuming. Last time I checked, Heptapod Runner worked on Darwin,
> but I have no reasonable place to host it.
> 
> 
> Best,
> --
> Georges Racinet [ https://orbeet.io/a-propos/cloudcrane |
> https://orbeet.io/a-propos/cloudcrane ] , [ https://heptapod.net/ |
> https://heptapod.net ] GPG: 09719FFE0B476DC26923F8EEB8EB20361976F291
> 
> _______________________________________________
> Mercurial mailing list
> Mercurial at lists.mercurial-scm.org
> https://lists.mercurial-scm.org/mailman/listinfo/mercurial