Producing Mercurial wheels from Github Actions + modernize build process

[Raphaël Gomès]

Hi all,

Pierre-Yves, Pierre and jumped into a VC yesterday to discuss this 
progress and how to best fit the needs of the project while keeping this 
new momentum going.

Here are the main takeaways:
     - The Github free tier wouldn't stretch more than a couple runs of 
CI of MacOS/Windows, so it's out of the question even outside of other 
issues we may have with relying on Github for anything Mercurial
     - We shouldn't be distributing wheels to users without testing 
them, so building a Windows wheel also implies getting a working CI at 
least once every release
     - Wheels would be equally easy to create in a Heptapod runner 
provided we have the right hardware for each platform
     - We are currently making progress in fixing the tests for Windows 
and MacOS and setting up runners at least for Windows, thanks in large 
part to Matt Harbison and the Heptapod community
     - Octobus has two Apple Silicon machines that should be setup as 
runners within the next few weeks
     - We should create our wheels with everything inside (Python, C and 
Rust on tested platforms, so Linux for now) and set the modulepolicy to 
be "allow" (C with a fallback to Python) by default. This way we have a 
single wheel, and users that want to opt-in to Rust will be able to do 
it without affecting the rest of the users. Maybe we'll turn on Rust 
whenever available at some point, but that implies a little more work in 
core to make it more clever about when to use Rust.
     - Wheels can be computed outside of a given release timing, so we 
don't have to "catch the next release window" per se.

Here is the plan we've agreed to:
     - Pierre Augier will add a manual job in our heptapod CI to create 
a wheel for Linux only. This is an easy step and will already make us 
move forward a little bit
     - He also agreed to try (but not promise) to take a look at 
improving our setup.py so it's less... horrible.
     - We will keep making progress with the Windows and MacOS CI as 
well as setting up their runners
     - We will probably set up a post-landing CI for Windows and MacOS 
because our workers will be too overloaded/slow to be in the normal CI. 
This is not ideal, but until we have more/faster hardware, this is 
already much better than the current situation.
     - Once that is all set up, I - being the release manager - will 
have to figure out how to best handle the publications of the wheels for 
each release

I hope I haven't forgotten anything, but in any case, I'm glad that this 
is moving along. :)

Thank you again to everyone involved,
Raphaël

On 10/7/24 11:09 AM, Pierre-Yves David wrote:
>
> I just realized that this discussion is taking place on the user 
> mailing list. We should move it on the developer mailing list.
>
> On 9/27/24 09:14, PIERRE AUGIER wrote:
>> Hello,
>>
>> I tried to see if Mercurial wheels could be produced on Github Actions.
>>
>> https://github.com/paugier/mercurial-devel/blob/refs/heads/wheels-with-github-actions/.github/workflows/wheels.yml
>>
>> Nothing works because simple standard commands do not work for Mercurial.
>>
>> https://github.com/paugier/mercurial-devel/actions/runs/11064702442
>>
>> Even producing the sdist fails:
>>
>> Run python -m build --sdist
>> * Creating isolated environment: venv+pip...
>> * Installing packages in isolated environment:
>>    - setuptools
>>    - wheel
>> * Getting build dependencies for sdist...
>> /!\
>> /!\ Could not determine the Mercurial version
>> /!\ You need to build a local version first
>> /!\ Run `make local` and try again
>> /!\
>> Run `make local` first to get a working local version
>>
>> And same error for wheels:
>>
>> python -m pip wheel /project --wheel-dir=/tmp/cibuildwheel/built_wheel --no-deps
>>    Processing /project
>>      Installing build dependencies: started
>>      Installing build dependencies: finished with status 'done'
>>      Getting requirements to build wheel: started
>>      Getting requirements to build wheel: finished with status 'error'
>>      error: subprocess-exited-with-error
>>      
>>      × Getting requirements to build wheel did not run successfully.
>>      │ exit code: 1
>>      ╰─> [6 lines of output]
>>          /!\
>>          /!\ Could not determine the Mercurial version
>>          /!\ You need to build a local version first
>>          /!\ Run `make local` and try again
>>          /!\
>>          Run `make local` first to get a working local version
>>          [end of output]
>>
>>
>> I had a look at the Makefile and setup.py. It seems that they contain a lot of legacy code and that a simplification and modernization would be welcome. For example, setup.py should never be called directly (and it is even better if we can avoid it).
>>
>> I guess I don't understand why setup.py has to be so complex and I suspect that it would much simpler with another build system, for example Meson (used by Numpy, Scipy and many other projectshttps://mesonbuild.com/Users.html).
>>
>> I have a bit of experience with using Meson so I can help if one wants to give it a try for Mercurial build.
>>
>> Regarding Rust, if I understand correctly, it is now a build option leading to two different Mercurial wheels. Meson-python can also have build options, but this is not good practice for the PyPA. Did you think about having the Rust extensions (and binaries) in another PyPI package (like mercurial-rust)?
>>
>> Pierre
>>
>> --
>> Pierre Augier - CR CNRShttp://www.legi.grenoble-inp.fr
>> LEGI (UMR 5519) Laboratoire des Ecoulements Geophysiques et Industriels
>> BP53, 38041 Grenoble Cedex, Francetel:+33.4.56.52.86.16
>> _______________________________________________
>> Mercurial mailing list
>> Mercurial at lists.mercurial-scm.org
>> https://lists.mercurial-scm.org/mailman/listinfo/mercurial
> -- 
> Pierre-Yves David
>
> _______________________________________________
> Mercurial-devel mailing list
> Mercurial-devel at lists.mercurial-scm.org
> https://lists.mercurial-scm.org/mailman/listinfo/mercurial-devel