Cannot pull/push to https server with self-signed certificate

[Matt Mackall]

On Fri, 2011-01-07 at 20:14 +0100, Mads Kiilerich wrote:
> [moved from mercurial to -devel]
> 
> On 01/07/2011 03:31 AM, Adrian Buehlmann wrote:
> > On 2011-01-07 03:18, Mads Kiilerich wrote:
> >> Adrian Buehlmann wrote, On 01/07/2011 03:15 AM:
> >>> On 2011-01-07 02:53, Mads Kiilerich wrote:
> >>>> Brian Sullivan wrote, On 01/06/2011 07:31 PM:
> >>>>> This discussion actually started as a bug reported about TortoiseHG
> >>>>> here:
> >>>>> https://bitbucket.org/tortoisehg/thg/issue/63/cannot-pull-push-to-https-server-with-self
> >>>>>
> >>>>>
> >>>>> I installed the latest version of TortoiseHg (1.1.8) on a new Windows
> >>>>> machine with no previous TortoiseHg or Mercurial installation.  We're
> >>>>> running our shared Mercurial server on Windows Server 2008 R2 under
> >>>>> IIS 7.5 with SSL using a self-signed certificate.  Things have been
> >>>>> running just fine for other users at our company on previous versions
> >>>>> of TortoiseHg.
> >>>>>
> >>>>> When I try to push or pull from this new THg 1.1.8 machine, I get the
> >>>>> following error:
> >>>>> abort: error: _ssl.c:490: error:14090086:SSL
> >>>>> routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
> >>>> Yes. The windows installers started shipping with a cacerts file
> >>>> configured. That could be considered a convenient security improvement
> >>>> for some users, but it is a regression for those with self-signed
> >>>> certificates.
> >>> I'm far from being an expert in that area, but I do find this a very
> >>> strange view.
> >>>
> >>> Given that Mercurial as installed by these installers now finally checks
> >>> the certificates *at all* and *by default* -- which benefits the vast
> >>> majority of the users -- is probably the lesser evil than simply
> >>> throwing a warning at everyone downloading and installing this stuff.
> >>
> >> Perhaps. But it is a regression for those with self-signed certificates.
> >
> > I wouldn't call something that didn't work at all (or rather: only
> > worked with a gaping security whole like that) a regression.
> 
> That is in my opinion not a correct description.
> 
> The role of web.cacerts was made (quite) clear when SSL server 
> certificate verification was introduced one year ago (
> http://selenic.com/repo/hg/rev/4c94a3df4b10 ). I made the role of 
> web.cacerts more clear in 1.7.3 with the warning introduced in 
> http://selenic.com/repo/hg/rev/2fa2e6444645 (where "will" should have 
> been "would"). That is what caused all this fuzz.
> 
> The most serious issue here is that nobody noticed how they should use 
> Mercurial with https connection. Few knew and nobody cared, so improved 
> documentation wouldn't help. That is why I introduced a warning to make 
> users aware there was something they should configure and some 
> documentation they should read. I think it was so important that a new 
> warning in stable was appropriate, but not so serious that more than 
> that should be done in a bugfix release.
> 
> I think the next step should be to try to improve the situation with 1.8 
> with better system integration, but I am not sure I would recommend 
> packagers to do anything that would disrupt any use of Mercurial. It 
> seems like I misjudged how users would react to the warning and how much 
> fear and demand for action it would cause.
> 
> 
> Then there was a bug in the cacerts check that made it almost worthless. 
> That could be considered a gaping security hole. It was fixed in 
> f2937d6492c5 before 1.7.
> 
> 
> The whole Python/Mercurial SSL situation - and PKI in general - is a 
> mess. There is no good solutions, so we have to consider the trade-offs.
> 
> When something worked as intended and documented in 1.7.2 and suddenly 
> stops working in a minor update to 1.7.3 then it is a regression in my book.
> 
> It is possible that a regression can be justified if it is necessary to 
> fix a security issue, but then we should do that deliberately and 
> explain that to the users.

Let's make a table.

              old    new             new
                     without certs   with certs
normal        I        I W             S
self-signed   I        I W             F

I = works, insecure (vulnerable to MITM)
W = warning
S = works, secure
F = fail

The only problem point with the new behavior is the F in the lower
right. We don't have a good story for what to do with this fairly common
situation (more common because we've made self-signed HTTPS the easy
route in the past!). Thus, we're going to have lots of users in need of
a work-around.

Both wget and curl have command-line switches to bypass this headache
(curl uses --insecure). We should probably have one too.

> Adrian, it is fine that disagree with me, but I don't think it is OK 
> that you just removed the half of my reasoning on the wiki that you 
> don't like - 
> http://mercurial.selenic.com/wiki/CACertificates?action=diff&rev1=14&rev2=15

I think this is ok because the advice is now stale and no longer
germaine to the wiki: certs did get introduced in a stable release a
week before this edit (and two days before the page itself existed).

> Anyway, I can see that Matt just restored the balance by removing the 
> part you agree with.

I moved 'em to Packaging, as it's not really 'user-facing' advice.

-- 
Mathematics is the supreme nostalgia of our time.