Improving the CI experience

Raphaël Gomès raphael.gomes at octobus.net
Mon Sep 28 14:43:52 UTC 2020


Hi all,

As you may know, we (Octobus) and other Mercurial contributors have been 
using
the Heptapod [1] CI to vet our patches pre-submission against 7 variants of
Mercurial:
- Python 2 pure
- Python 3 pure
- Python 2 python+c
- Python 3 python+c
- Python 2 rust+c
- Python 3 rust+c
- Python 2 chg
- (also code checks and Rust unit tests, of course)

All variants above are run on x86_64 Debian machines. Our CI does not 
(yet) have
the MacOS, Windows, or FreeBSDoptionsthat the Mercurial Buildbot has, but
does test more variants on Linux.

Nevertheless, it catches a lot of the mistakes that we all make sometimes,
before we even send the patches upstream: regressions in Python 3, API 
breakage
with Rust, bad formatting, and other bugs of various nature. It does so way
before it touches the `committed` repo, which means that if I screw up my
patch, it only has to bother me.

The value of such a system is highly dependent on it being trustworthy: 
flaky
tests do exist, but they should be addressed and fixed in a timely 
manner and
my CI being red should mean that *I* missed something. Yet, I have heard the
phrase "I don't even look at the pipelines anymore, they're always 
broken for
something I haven't done".

Most of the time it's something minor: I've even been guilty of breaking 
the CI
myself in my last patch series (oops) due to bad formatting! However, these
issues should be few and far between: the developer experience suffers 
greatly
and we grow to accept that the tests are always broken, don't bother to run
them at all, etc.

# Proposal

I thought a start to fixing this situation would be for me to setup
an automation to send a small email to this mailing-list every time 
either the
`stable` or `default` branches (that track `hg-committed`) are broken, 
with a link
to the pipeline results.
This will not catch everything (only a subset of the possible targets is 
tested), but
I think this would be an improvement for all contributors, even those 
that don't
use Heptapod.

What do you all think?
Raphaël

[1] https://foss.heptapod.net/octobus/mercurial-devel/-/pipelines
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.mercurial-scm.org/pipermail/mercurial-devel/attachments/20200928/14197fe7/attachment-0001.html>


More information about the Mercurial-devel mailing list