Things we ought to do to improve our packaging
Angel Ezquerra
angel.ezquerra at gmail.com
Thu Aug 8 06:30:37 UTC 2013
On Thu, Aug 8, 2013 at 8:13 AM, Steve Borho <steve at borho.org> wrote:
>
>
>
> On Thu, Aug 8, 2013 at 12:43 AM, Brendan Cully <brendan at kublai.com> wrote:
>>
>> On Wednesday, 07 August 2013 at 23:37, Steve Borho wrote:
>> > On Tue, Aug 6, 2013 at 5:53 PM, Matt Mackall <mpm at selenic.com> wrote:
>> >
>> > > On Tue, 2013-08-06 at 15:04 -0500, Steve Borho wrote:
>> > > >
>> > > >
>> > > >
>> > > > On Tue, Aug 6, 2013 at 2:47 PM, Matt Mackall <mpm at selenic.com>
>> > > > wrote:
>> > > > On Tue, 2013-08-06 at 14:15 -0500, Steve Borho wrote:
>> > > > >
>> > > > >
>> > > > >
>> > > > > On Tue, Aug 6, 2013 at 2:01 PM, Augie Fackler
>> > > > <raf at durin42.com> wrote:
>> > > > > On Tue, Aug 06, 2013 at 01:53:34PM -0500, Matt
>> > > > Mackall wrote:
>> > > > > > [this discussion needs to be cross-posted]
>> > > > > >
>> > > > > > We've got a few long-standing problems with our
>> > > > current
>> > > > > packaging:
>> > > > > >
>> > > > > > - lag official release
>> > > > > > - not automated
>> > > > > > - not reproduceable by third parties
>> > > > > > - not signed
>> > > > > > - lack nightly builds
>> > > > > > - notable missing platforms (RHEL)
>> > > > > > - not uniform in configuration
>> > > > > >
>> > > > > > Our attitude to date has been "well, we're not
>> > > > really in
>> > > > > charge of that,
>> > > > > > that's done unofficially by volunteers". I think
>> > > > we've
>> > > > > reached a point
>> > > > > > where this needs to start changing.
>> > > > > >
>> > > > > > In an ideal world, a release would work like
>> > > > this:
>> > > > > >
>> > > > > > - I tag the release
>> > > > > > - I push it
>> > > > > > - Automated builders kick off on a bunch of VMs
>> > > > or
>> > > > hosts
>> > > > > > - Builders run stock makefile targets or scripts
>> > > > in tarball
>> > > > > > - Packages get signed
>> > > > > > - Packages get uploaded to one or more
>> > > > locations,
>> > > > including
>> > > > > > mercurial.selenic.com
>> > > > > > - Download links get updated
>> > > > > >
>> > > > > > Importantly, this same process would also be
>> > > > used
>> > > > to produce
>> > > > > automated
>> > > > > > nightly builds for testing and bug reporting and
>> > > > discovering
>> > > > > packaging
>> > > > > > problems before release dates.
>> > > > > >
>> > > > > > The job of packagers would thus shift from
>> > > > "(manually?)
>> > > > > building and
>> > > > > > uploading binaries" to "making sure the
>> > > > committed
>> > > > build
>> > > > > rules are
>> > > > > > correct and up-to-date".
>> > > > > >
>> > > > > > I think this also needs to be inclusive of
>> > > > builds
>> > > > for
>> > > > > packaging systems
>> > > > > > (Debian, Ubuntu, RHEL, Centos, Solaris, Pypi,
>> > > > etc.). These
>> > > > > actually
>> > > > > > suffer from more or less the same problems even
>> > > > though the
>> > > > > distribution
>> > > > > > model is different. So we should be building
>> > > > nightlies and
>> > > > > release
>> > > > > > builds for these systems and be more directly
>> > > > responsible
>> > > > > for quality
>> > > > > > and timeliness here as well.
>> > > > >
>> > > > >
>> > > > > This honestly sounds like a great improvement,
>> > > > given
>> > > > how stale
>> > > > > the
>> > > > > packages for Ubuntu tend to be, and how some
>> > > > packagers took it
>> > > > > upon
>> > > > > themselves in the past to enable all the
>> > > > extensions.
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > > On the Windows side, everything is automated except for
>> > > > two
>> > > > steps:
>> > > > >
>> > > > > 1 - signing packages requires a passphrase to be entered
>> > > > > 2 - uploading is currently manually done through BB's
>> > > > clunky
>> > > > web
>> > > > > interface
>> > > > >
>> > > > >
>> > > > > If we had an SCP or FTP URL to upload to, and a sane
>> > > > policy
>> > > > for
>> > > > > deleting old nightlies after a short while, it could be
>> > > > automated.
>> > > >
>> > > >
>> > > > Well, you already have SCP access, no?
>> > > >
>> > > >
>> > > > Yep, it's just a question of whether you want nightlies in the same
>> > > > location as releases, or in a different folder that was
>> > > > size-managed.
>> > > >
>> > > > In addition to having a fully automated process that builds
>> > > > releases and
>> > > > nightlies, I think its also important to get to the build
>> > > > script into
>> > > > the tree so that people who need to build their own copies
>> > > > for
>> > > > various
>> > > > reasons (need bugfix, need custom deployment, St. Louis
>> > > > trampled by
>> > > > kaiju) can do so.
>> > > >
>> > > >
>> > > > So.. building windows frozen packages is fairly convoluted
>> > > > (especially
>> > > > for thg but the simple hg MSI is no cakewalk) and has a boatload of
>> > > > prereqs to be installed.
>> > >
>> > > Yep, I'm well aware. And this is actually MORE reason to insist on
>> > > full
>> > > automation. Let me back up and explain some of my motivation here:
>> > >
>> > > While I think you do a great job with packaging, it's lately become
>> > > clear that package maintainers are a significant single point of
>> > > failure. And we've also not coincidentally discovered that we don't
>> > > actually know how some of our packages are actually built (like OS X)
>> > > because the instructions don't work any more. So when the packager
>> > > disappears, we've got several problems: find someone with the
>> > > hardware,
>> > > install the OS(es), install the deps, reverse-engineer the build
>> > > process, hit unknown number of gotchas..
>> > >
>> > > The OS X situation is not as complex as Windows, but it's still bad
>> > > enough that the current approach is clearly not good enough.
>> > >
>> > > And that's not the only problem in this area. For instance, we also
>> > > have
>> > > real uses for continuous builds that are not being met by monthly
>> > > releases. I really want the BTS robot to point people at builds and
>> > > say
>> > > "try this and report back, please", but can't.
>> > >
>> > > (And it's also a perennial confusion that we announce releases before
>> > > there are any packages built.)
>> > >
>> > > Fortunately, we know how to fix these sorts of things:
>> > >
>> > > - manifest the instructions and dependencies as build scripts
>> > > - version control the build scripts in the canonical repo
>> > > - use the build scripts, continuously and non-interactively
>> > > - get people to use those builds and discover problems before releases
>> > > - fix the problems found via check-in
>> > >
>> > >
>> > > With my 'hackable' Windows builds, I've gone as far as scripting the
>> > > download and install of the right versions of Python and gcc and all
>> > > the
>> > > other deps if not present to make sure its reproduceable, so I know
>> > > this
>> > > sort of thing can be done, even on Windows. I've obviously not yet
>> > > gone
>> > > as far as putting this knowledge in the repo, but I ought to.
>> > >
>> > >
>> > > All that said, let's focus on phase 1: publishing nightly builds,
>> > > somehow, somewhere.
>> >
>> >
>> > Angel has offered to build nightlies before (using an automated
>> > framework
>> > like Jenkins), he just needed a place to upload them. We just need to
>> > circle around and get that closed. The auto-builds might be unsigned at
>> > first.
>> >
>> > As far as getting this into the tree and scripted.. I agree this would
>> > be
>> > astonishingly useful for the long run (it mostly frees me up as well)
>> > but
>> > my hg hacking time is extremely limited these days and I don't know when
>> > I'll get a chance to work on it.
>> >
>> > One suggestion I have, if we try to automate this process in the
>> > hg/contrib
>> > folder, is that we make an attempt to drop the requirement on
>> > Microsoft's C
>> > compilers and try to use recent MinGW GCCs to compile the Explorer
>> > extension and the Mercurial (and other hg extension) Python extensions.
>> > We
>> > had problems with GCC some 5-6 years ago but perhaps those are all
>> > resolved
>> > now (that said, the Explorer extension hasn't changed in 19 months, we
>> > could probably check in pre-compiled versions in its repo).
>>
>> We do have an existing buildbot. I'm not sure what advantage there
>> would be in a separate Jenkins to do the builds? The buildbot even has
>> a windows slave.
>
>
> The only advantage is that Angel already has the prereqs in place to build
> packages.
>
> --
> Steve Borho
Unfortunately the server I used to do the nightlies has been down due
to a hardware problem for a few weeks. I am still waiting for it to be
repaired but it might take a while before is up again.
Cheers,
Angel
More information about the Mercurial-packaging
mailing list