How to write a test description

If you were to read the TAP grammar, you would see the following line:

test ::= status positiveInteger? description? directive?

What that means is that a test line of TAP (if you read the rest of the grammar) must have an "ok" or "not ok" bit, followed by an optional test number (in practice, it's almost always there) and a test description (the directive refers to "skip" or "todo" tests). Sadly, many people don't pay attention to the powerful benefits of the description. A bad description may as well be left off; a good description is the difference between a pile of confusing code and documentation.

This sounds strange if you're not used to writing good test descriptions, but here's what's going on. Let's first look atTest::More's is() function:

sub is ($$;$) {
    my $tb = Test::More->builder;
    return $tb->is_eq(@_);
}

The ($$;$) prototype means "pass two things in scalar context, along with an optional third thing in scalar context". So you might see a test like this:

is sreverse(17), 71;

That's because the test description is optional. More often, though, you see a test like this:

is sreverse(17), 71, 'sreverse ok';

And that might print out:

ok 1 - sreverse ok

And many programmers then pat themselves on the back thinking "job well done". But what the hell is sreverse()? I can see that the test passed and having sreverse ok as a test name is redundant. You may as well leave the test name off! However, what if the test name looked like this?

is sreverse(17), 71, 'sreverse($scalar) should act like "scalar reverse $scalar"';

Ah hah! Now we're getting somewhere. Now the description is describing the behavior, not the boolean value of the test. We get output like this:

ok 1 - sreverse($scalar) should act like "scalar reverse $scalar"

Now I can read my test output and have an idea of what the code is supposed to do and not just whether or not the tests pass. In other words, the tests are approaching documentation, not just helping to prevent changes in behavior.

Case in point: recently I encountered some test output like this:

ok 1 - redirect
ok 2 - transparent gif
ok 3 - cookies ok
ok 4 - log data

Can you tell what that means? Me neither. In fact, this code was a bit confusing and in the process of understanding it, I wound up rewriting the test messages similar to this:

ok 1 - If no cookies are found, we should redirect the user
ok 2 - ... to a transparent gif
ok 3 - ... and the cookie should have a value of 0
ok 4 - ... and the log should have default values and be marked as not valid

Which of those would you rather read as a maintenance programmer? The second version doesn't completely document what's going on, but it's a heck of a lot better than the first version. In fact, if you rewrite that as a subtest, you get this:

    ok 1 - If no cookies are found, we should redirect the user
    ok 2 - to a transparent gif
    ok 3 - and the cookie should have a value of 0
    ok 4 - and the log should have default values and be marked as not valid
    1..4
ok 1 - click detected but no cookies present

In other words, you can skip the grouping by ellipses (...) that I used to always do and group related tests in a subtest with a descriptive name of the test case. Don't neglect those test descriptions. Even if you can perfectly understand what's going on, a well-written test description is going to be a huge benefit to any programmer coming behind you and working on the system.

Side note: if the prototype "pass things in scalar context" comments sound strange, watch the following test pass and the strange test description:

use Test::More;

my @a = (1,2,3);
my @b = (4,5,6);
is @a, @b, @b;       

done_testing;

7 Comments

I've been in the habit of writing my descriptions to note what should have happened, just as you have done.

This is very well written and hits an important peeve i've encountered often in the past and which i've done wrong often as well. (Until i realized what was annoying me and started fixing it in myself.)

Thank you. :)

On the subject of better test descriptions, 'note' is very useful for separating out blocks of tests when reading test output.

Here's one case I've found where no test description is better than having one.

With Test::WWW::Mechanize, get_ok() has a default test message that prints the URL that is being fetched.

These can often be constructed using query strings which were dynamically generated by the test.

Often times, descriptions for these tests ending being something like "get the page".

Here's an example:

not ok 1 - Get a page

Failed test 'Get a page' at /home/mark/tmp/t.pl line 9. 404 Not Found

1..1

If the test fails and I need to debug it, often the first thing I want to do is to manually visit the URL in the browser.

In this case, the default error message is better than a custom error message, because it provides a URL I can copy and paste.

Perhaps the best solution here is patch Test::WWW::Mechanize to include the URL in the diagnostic output, so you can have a descriptive test label without repeating the valuable URL in the description.

Yup... Further, I would add that an optimistic description that implies success makes the test output hard to read:

not ok 2 - things are working perfectly

It's easy to miss the "not" when there's a long chunk of text telling you everything is fine.

I use the somewhat verbose practice of beginning every test description with a phrase like "Testing that...". That forces me to write a description that makes sense if the test fails or succeeds.

Don't complain: your project at least has test descriptions!

I'm dreaming of a Perl::Critic policy (or a super test) that would make test descriptions mandatory.

Loving this test style.

We're trying to follow a strict two-line format in the perl code, which makes the tests really easy to skim through.

ok my $thing = Classname->new( %args ),
  "Can create a Classname";

is $thing->accessor, $expected_value,
  "... and attribute was set ok";

ok my $result = $thing->method_call,
  "... and can method_call";

is $result, $expected_value,
  "... and method_call returned thing_we_wanted";

/me is enjoying not creating tmpvars on their own lines all the time and having oodles of space to write test descriptions :-)

Leave a comment

About Ovid

user-pic Have Perl; Will Travel. Freelance Perl/Testing/Agile consultant. Photo by http://www.circle23.com/. Warning: that site is not safe for work. The photographer is a good friend of mine, though, and it's appropriate to credit his work.