Testing Mercurial with Hypothesis

[Pierre-Yves David]

On 02/22/2016 03:38 PM, David MacIver wrote:
> On 22 February 2016 at 14:24, Pierre-Yves David
> <pierre-yves.david at ens-lyon.org <mailto:pierre-yves.david at ens-lyon.org>>
> wrote:
>
>
>
>         1. How would you like this integrated, if indeed it is a thing
>         people
>         are interested in seeing in core mercurial at all? Is the
>         prospect of
>         running it as part of run-tests interesting, or would you prefer a
>         different approach? In particular, what should be done with the
>         generated failures?
>
>
>     I think that running them often is valuable.
>
>
> I agree, but it might be worth thinking about what the ideal workflow of
> "running them often" should look like in a Mercurial context.

I think we'll have them on in the buildbot setup. That will run them on 
any new push.

>     We have a "long running" test concept, disabled by default. If we
>     have them there and buildbot running them would be nice.
>
>
> One thing I was considering was having them check whether long-running
> tests are permitted and adjust their timeout appropriately, as the tests
> can be run for a configurable amount of time.

Can we get anything useful in less than a minutes? Because our longest 
test is 80 seconds and we don't want to delay test-running of everybody.

>     Having some easy way to start 100 exploration in parallel for a
>     couple of hours could probably be valuable too.
>
> Yeah. At the moment the stateful testing is not /very/ well set up for
> this usage pattern. In particular it will stop when it finds a single
> failure. I'm going to have a think about what the best way to set this
> up is.

That would be nice. Apparently Simon found some bug after a long time so 
parrallel running would help a lot here.

>     Finally I'm sure we want to make sure any bug it find stay covered.
>     I'm not sure if we should write a dedicated manual tests case for
>     each of them or rely on hypothesis to save these case and run them
>     during any normal run-tests.py.
>
>
> In general I discourage using the Hypothesis example database for long
> term storage. Its intended usage pattern is more that it's a local cache
> / state file for development - it ensures that if you find a bug it
> doesn't go away until you fix it.
>
> The design of the tests I wrote for Mercurial are that they write a .t
> file capturing the behaviour they observed. Turning those into normal
> Mercurial tests may be a better way forward.

Okay, it seems reasonable to request human intervention to immortalise 
any test finding a bug.

-- 
Pierre-Yves David