[PATCH 0 of 5] Add 'unified' tests to the test framework

[Matt Mackall]

On Tue, 2010-08-03 at 08:56 +0200, Dirkjan Ochtman wrote:
> On Tue, Aug 3, 2010 at 07:15, Matt Mackall <mpm at selenic.com> wrote:
> > This is an experiment in unifying test commands and results as
> > discussed at the 1.5 sprint. See the last three patches for what tests
> > look like in this model. It's quite an improvement. And when something
> > breaks, you get to see the changed output in the context of the
> > commands that caused it. Also, there's some basic support for
> > filtering of output with regexes, like this:
> >
> > Strip 1: expose an old head:
> >
> >  $ hg --config extensions.mq= strip 5
> >  saved backup bundle to .*
> >
> > ..which means a lot of the hoops tests jump through right now with sed
> > and grep are no longer needed.
> 
> It looks very nice! Definitely something I'd use in other projects as well.
> 
> One thing that I think could be better is the need for indentation
> before $ hg command. I understand that we need something to
> disambiguate comments from command output, but I feel that doing it
> the other way around would be better (prefix the comments, not the
> output). Primarily because I think it's easier to cut-and-paste
> command output that way, and we probably have more command output than
> comments anyway.

There's not much need to cut and paste command output though. You simply
write the test without output, then run it, and you end up with the
unified result in .err. And you either copy that over, or use -i to let
run-tests do it for you. In other words, just like you'd write a normal
test.

>  We could prefix comments with something like # or %.
> I guess that introduces a chance of ambiguity for output starting with
> # or %, but we could mitigate that by having a longer prefix, perhaps?
> (e.g. %%, although I think % is sufficiently rare in our test output
> and/or should be easy to prevent in most cases).

I think that would reduce the readability substantially. Consider:


First we create a repo:

  $ hg init a
  $ cd a

Then we create a changeset:

  $ echo a > a
  $ hg ci -Am0
  adding a


vs


%% First we create a repo:

$ hg init a
$ cd a

%% Then we create a changeset:

$ echo a > a
$ hg ci -Am0
adding a


The first reads exactly how you'd write instructions in an email or
tutorial.

-- 
Mathematics is the supreme nostalgia of our time.