If you saw my FOSDEM talk about Building a Universe in Perl, I show some examples of the code we use to model behaviors in our universe. I start talking about out test suite at one point and that's probably the most exciting part. You see, we've been fixing our leaderboard system in part because I can do this:

That's a truncated version of our test failure report for tests failing on master. Leaderboard tests show up twice in there.

Ever have an failing test and trying to remember if it failed before? Is it fragile code? Is it fragile tests? Has it ever failed on your master branch? What percentage of your tests fail? Well, now you can find out.

Test::Class::Moose::History works like this.

my $runner = Test::Class::Moose::Runner->new( %opts );
$runner->runtests;

# get current branch and latest commit id
# example assumes `git`, but you can use any info you need here
chomp( my $branch = `git rev-parse --abbrev-ref HEAD` );
chomp( my $commit = `git rev-parse HEAD` );

my $history = Test::Class::Moose::History->new(
    runner => $runner,
    branch => $branch,
    commit => $commit,
);
$history->save;
my $report = $history->report;

my %report_methods = (
    last_test_status => [qw/Class Method Runtime Passed/],
    last_failures    => [qw/Class Method/],
    top_failures     => [qw/Class Method First Last Errors/],
);

my $builder = Test::Builder->new;
foreach my $method ( sort keys %report_methods ) {
    my $results = $report->$method;
    my @rows = ( $report_methods{$method}, @$results );
    $builder->diag("\nReport for $method");
    $builder->diag( generate_table( rows => \@rows, header_row => 1 ) );
}

Alternatively, if you've already saved report data, you can do this:

my $report = Test::Class::Moose::History::Report->new;
my $failures = $report->last_failures;
...

So on the Tau Station test suite, I can do thing like:

prove t/tcm.t :: --failures

And that will rerun the last set of failing tests rather than the entire test suite.

Why Test::Class::Moose?

If you wanted to build a single page web site and you just printed the HTML in a large heredoc, no one's going to care. If you build a large-scale application like that, people are going to care. In fact, as projects get larger, many people reach for frameworks because many of the bits and pieces you need are already in place.

Similarly for test suites: if you're just writing a small test suite for a module, that's fine. If you're building a large test suite for a huge application, you should be more careful with your test suite. I may be biased as I wrote it, but I recommend Test::Class::Moose (now maintained by Dave Rolsky). More and more companies are using because it makes test suites much easier to organize and maintain. Set up your test runner, your test base class, and start writing great tests.

It also has great reporting, but until now, that information hasn't been saved anywhere. Now with Test::Class::Moose::History, that's changed. It's not on the CPAN yet as I'd like to iron out the kinks, but if you use Test::Class::Moose, I think you'll love saving your test history.

Note: If you have trouble getting up and running with Test::Class::Moose, the Test::Class::Moose::History test suite has a small test suite that's a great example to start from. Also, you can download that module and run prove t/tcm.t :: --report to see some sample reports showing my test suite development for the code.

While the code is a proof of concept, we've been using it with the Tau Station team for over half a year. It's also running on our Jenkins box so we can see what tests fail there, too.

Hire Us!

As always, if you want top-notch software development or training, or help building your test suite, drop me a line at ovid at allaroundtheworld.fr. Most of us are Perl devs, but we also do front-end development, Golang, C++, Lua, and Python. Due to our extensive links in multiple software communities, if you need an expert in another language, let me know.