Automated Staged Repos with Continuous Integration

[Tom Anderson]

On 10 December 2011 20:57, Scott Palmer <swpalmer at gmail.com> wrote:

> The idea is that any developer can push to the "integration" repository, but they would always pull from the "validated" repository, so the sharing of broken bits is minimized.  It seems like it should be fairly easy to set up, but I'm wondering about how to set up the "integration" repository.   The issue is that multiple developers will be pushing to it and since the developers will be basing there code on the good code from the "validated" repo I see a situation with multiple heads arising as each developer tries to get their changes to the integration server.
>
> Am I just dreaming or is it possible to make something like this work?

I don't think there's any way of doing this really well. There's a
fundamental tension between not sharing broken code, and not delaying
merges.

Options i've come across:

1. Say that as long as there is a change in the integration repo that
has not been promoted to the validated repo, nobody can push up. Avoid
the problem.

2. Have developers pull from the validated repo, *except* when they're
about to push up, when they should merge from the integration repo. At
least that constrains the spread of broken code to just before a push,
at which point local tests will catch it, and it can be backed out.

3. Allow multiple heads in the integration repo. Have Jenkins test all
of them. On getting a green bar against any head which is a descendant
of the current tip of the validated repository, promote it into the
validated repository. In other words, allow multiple heads, but pick
one as a winner. Notify developers that they need to merge their head
with the validation repo and push again. Note that Jenkins can test
every tip, even if it knows it can't promote it; that's useful
feedback for developers.

4. Allow multiple heads, and promote one at a time to the validation
repository, but have Jenkins do an automatic merge of each head. It
would need to take a head, build and test it, merge it, build and test
the merge (because the merge could easily break things), then promote
it. You still need to bounce failed merges back to the developers.

It might also be worth looking at splitting the validation into fast
and slow parts. Fast validation can check that the code compiles
(using an incremental compiler, if there is one for your language) and
passes unit tests; that should take a few minutes at most. You could
run that on a pretxnchangegroup hook on the server, so really bad code
can't even be pushed. You would then the slow parts, building the
system and running functional tests, after checkin. Then, even if
people are pulling code from the integration repo, at least they know
it has some level of non-brokenness.

Approaches 3 and 4 turn the integration repo into a sort of queue of
finished work items. They enable a workflow where a developer takes a
task, does some work, makes some commits, pushes up, and is then free
to update back to his starting point (wherever the current validated
repo tip is at the time) and start a new task on a fresh branch. At
some point, Jenkins will finish building and testing his stuff, and
promote it, updating the validated tip, and he can merge his new
branch into the previous branch. Or, if the previous branch is broken,
update over to it, fix it, push it up again, then update back to his
in-progress branch to carry on with that while Jenkins tests his
fixes.

tom

-- 
Tom Anderson         |                e2x Ltd, 1 Norton Folgate, London E1 6DB
(e) tom at e2x.co.uk    |    (m) +44 (7960) 989794    |    (f) +44 (20) 7100 3749