Using Rust in Mercurial

Wed Dec 6 19:22:17 UTC 2017

On Wed, Dec 06, 2017 at 12:33:14AM +0100, Juan Francisco Cantero Hurtado wrote:
> On Tue, Dec 05, 2017 at 11:00:41AM -0800, Gregory Szorc wrote:
> > Mercurial Packagers,
> > 
> > We would like to start using Rust in Mercurial. A brain dump is at
> > https://www.mercurial-scm.org/wiki/OxidationPlan. The anticipated sequence
> > of events is likely something like:
> > 
> > 1) Replace exewrapper.c/hg.exe with a Rust binary on Windows
> > 2) Rewrite existing C Python extensions in Rust
> > 3) Make `hg` a Rust binary (instead of a Python script) on all platforms.
> > This requires the Rust binary to link against libpython and embed a Python
> > interpreter.
> > 
> > There is no timeline for these milestones. But I'd like to do #1 in the 4.5
> > release if possible and #2 and #3 as soon as possible after that.
> > 
> > The Mercurial Project itself is interested in shipping a
> > self-contained/standalone Mercurial distribution. This would include its
> > own bundled version of Python, including the standard library.
> > 
> > When we start requiring Rust to use Mercurial, I anticipate that downstream
> > packagers will want to package an "unbundled" Mercurial - one that links
> > against the system libpython2.7 and uses the system Python's standard
> > library. This is totally fine and shouldn't be difficult to support.
> > 
> > My question to packagers is: what is your current level of support for a)
> > Rust b) embedding CPython? e.g. what is the minimum version of Rust we can
> > target? Do you support a shared libpython2.7.so or only a static
> > libpython2.7.a?
> 
> Hi, I'm the OpenBSD maintainer.
> 
> a) We have rust support on amd64. The support on i386 has some problems
> with big programs like Firefox. We don't have support for rustup.
> 
> OpenBSD has a 6 months cycle for the -stable release and a kind of rolling
> release -current. I try to keep the package updated on -stable when
> there are security bugs but I don't update to bugfix releases. -current
> is always updated.
> 
> So, the rust features used by mercurial should have at least 12 months.
> 6 months + the time needed by our maintainer to update rust + the
> mercurial cycle.
> 
> Security updates are a pain with mercurial because you only provide a
> patched new version, even without pointing to the commits with the
> patches. Patching python code is way easier than rust. Maybe the project
> could have someone in charge of create the backports for the security
> bugs while the bugs are under embargo instead of the actual situation.
> The migration to rust will not help here.
> 
> Also, changing to rust, without a C or python fallback, we are going to
> lose the package on a lot of platforms. The same for NetBSD, FreeBSD,
> Debian or Gentoo. Here is our list of platforms with packages:
> https://fastly.cdn.openbsd.org/pub/OpenBSD/snapshots/packages/
> 
> b) We have libpython2.7.so.0.0 (the major and minor numbers could change
> in the future). We don't have the static lib, but I could add the lib if
> you need it.
> 
> > 
> > I'll likely capture replies on the aforementioned OxidationPlan wiki page.
> > If you want to just add relevant info there (anywhere is fine - I'll see
> > the wiki updates and refactor if needed), feel free to do that.
> 
> Eventualy, you are going to lose every current extension because the
> migration will break more and more APIs. So, why not create a DSL
> written in rust for small extensions and a rust library for the big
> ones?. Bundling python looks like a pain which will give you a lot of
> problems.
> 
> 
> > I want to emphasize that nothing is yet set in stone. But adding Rust to
> > Mercurial is something we would very much like to do and we would like to
> > have as many packagers on board as possible.
> 
> I'm not against the Rust migration but just be aware that this kind of
> change are not always easy from a maintainer perspective. We are not
> windows, we don't have isolated programs with their own libraries.
> Everything is coordinated in a big tree of packages.
> 
> As user, my biggest problem with mercurial is the speed with big repos
> (100000-200000 commits), so I'm happy with this change (also with your
> zstd work).
> 

I sent the thread to our Rust maintainer (Sebastien Marie) and he has
some additional comments:

- current level of support for rust

amd64 and i386 only. there is work for arm64. and i386 could become
broken soon (the problem isn't firefox only, it is ENOMEM when building
the compiler itself).

- minimum version of Rust we can target

from rust point of vue, any stable version.

but please don't start depending on version X just 1 day after the
release (as firefox did initially - now there leave to us 1 week, and it
could be still short).

but problems could arrive with security issues management: we couldn't
just import a new version on -release if the new version depends on
newer rust version.

the ideal solution would be backporting security fix (fix commits only)
and that any security fix to don't use recent features (that would be
not available in 6 month old rustc version).

-- 
Juan Francisco Cantero Hurtado http://juanfra.info