Chad 'Exodist' Granum

Lately there has been a buzz of activity in the Test-More project (Test::More, Test::Simple, Test::Builder). In fact, it is safe to say that very little code has gone untouched in this process. We are currently on the 34th alpha release of the new Test::Builder, and I wanted to take a moment to discuss the things that are happening!

This is the first of several blog posts that will cover the changes to Test::Builder. This one covers the "why".

Why are things changing? The simple answer of course is "Testing". A more helpful answer is this: Test::Builder is the de facto standard when you go to write a test. Just about any testing tool you use will rely on Test::Builder under the hood. The problem though, is that testing your testing tools is still somewhat archaic.

Currently, if you want to test a test library, you have two choices:
Test::Builder::Tester
Test::Tester

Test::Builder::Tester captures the TAP output, and compares it, as a string, to a string you author that predicts the expected output. This system is really hard to use. You have problems with whitespace, and with minor differences. The really bad thing is that if any changes are made to TAP output within Test::Builder your tests will likely break. Recently in fact we changed the indentation of a diagnostic message, and it broke a lot of things. We should be able to change messages intended for human consumption without breaking things for machines.

Test::Tester works differently, it monkey-wrenches the guts of Test::Builder and intercepts the events before that can be turned into TAP. Did you spot the problem here? By relying on hacks to the Test::Builder internals it has backed itself into a very fragile corner. The other problem is that it only looks at 'ok', it ignores diagnostics and notes.

Another problem, neither of these testing tools understands subtests. That's right, subtests, being just a hack anyway, are completely ignored. For those that do not know, any output that is not understood by a TAP parser is ignored by default. Subtests work by indenting the TAP so that the parser will completely ignore it. Ultimately the only thing that matters about a subtest is the ok/not ok line that follows. If part of your soul just shattered, you are not alone, a friend of mine fall into a depressed fog for several minutes after finding this out.

So what is being done about this?! Test::Tester2 (And a lot of supporting changes under the hood of Test::Builder).

• Test::Tester has been integrated into the Test-Simple dist, and will be deprecated over time. (but likely will never completely removed)
• Test::Builder::Tester has been integrated for a long time, but will also soon be deprecated.
• Test::Tester2, is part of the Test-Simple package, and is a shiny new interface that understands the tests at every level.

At its heart there are two components to Test::Tester2, intercept(), and events_are(). the intercept function will capture any and all output produced by Test::Builder. The difference here is that it captures event OBJECTS, and it does so long before they can be transformed into TAP. There are event objects for any event Test::Builder produces, including ‘ok’, ‘diag’, ‘note’, ‘plan’, etc. These events are even captured from subtests! intercept simply returns an arrayref with all of the produced events, in order.

events_are() is an interface intended to help you validate the events returned from intercept(). The first argument should be the arrayref of events. The additional arguments are a DSL describing what you expect to see. This DSL also provides ways to skip over events you do not care about, such as diagnostic messages that may or may not be present. The interface here is very full-featured, and I hope you like it! Here is some example usage:

my $events = intercept {
    ok(1, "pass");
    ok(0, "fail");
    diag("xxx");
};
 
events_are(
    $events,
    name => 'Name of the test',                       # Name this overall test
    ok   => { id => 'a', bool => 1, name => 'pass' }, # check an 'ok' with ID 'a'
    ok   => { id => 'b', bool => 0, name => 'fail' }, # check an 'ok' with ID 'b'
    diag => { message => qr/Failed test 'fail'/ },    # check a 'diag' no ID
    diag => { message => qr/xxx/ },                   # check a 'diag' no ID
    'end'                                             # directive 'end'
);

There is a lot more to be found here: Test::Tester2

My next blog post will be about some of the internal changes that went into allowing this interface.

EDIT (8/17/2014) - 'Results' were renamed 'Events' in alpha 35 which was just released, this post was updated to reflect that.