Test::Class: why have tests in startup?

If you use Test::Class frequently, sooner or later you'll see the following:

sub startup : Tests(startup => 1) {
    my $test = shift;
    my $engines = $test->start_engines;
    ok $engines->are_running, 'Engines should start OK.';
}

At first blush that looks sensible, but I don't think it is. I'll explain my view of the problem and hope that others can explain why I might be wrong.

There are, in my opinion, two reasons not to use that construct. First, why is that a test? If that test fails, the rest of the tests in that class won't be run, but in this case, why not just die? Test control methods like startup, shutdown, setup and teardown shouldn't be an extra place to stuff tests, they're places to control the actual tests. If something has gone horribly wrong in them, just die and the rest of the tests in that class will be skipped.

Now that doesn't sound to everyone like a compelling argument, but what happens when you subclass?

sub startup : Tests(startup => 1) {
    my $test = shift;
    $test->next::method;
    ok $test->create_grue, 'Grue created';
}

Oops. Now you have a problem. That "next::method" has run a test in your superclass and you've run two tests instead of one (a patch to Test::Class could fix this, but you would likely break existing tests which workaround this issue). This means that you now have to count the tests in your superclass(es) startup method(s):

sub startup : Tests(startup => 2) {
    my $test = shift;
    $test->next::method;
    ok $test->create_grue, 'Grue created';
}

That looks wrong because you only see one test. Then when someone adds a test in a superclass startup, they've broken subclasses. Since there's no real reason (that I can see) to have tests in these test control methods and since they can be very fragile in the face of inheritance, why have them? I recommend that people don't use them and I've eliminated them in our production code at the BBC with no drawback. If something fails, just die and let Test::Class do the right thing.

Update: I completely forgot that you can use the following syntax to avoid counting superclass tests:

sub startup : Tests(startup => +1) {
    my $test = shift;
    $test->next::method;
    ok $test->create_grue, 'Grue created';
}

Note that you're specifying the additional number of tests, not the absolute number of tests. This renders much of my argument moot.

6 Comments

Now that doesn’t sound to everyone like a compelling argument

Yeah, but why? What is their counterargument? Because I can’t think of any reasonable one – not if the test is checking something that may cause all subsequent tests to fail. Is there any point to running your tests once you know they’ll just all fail? I can’t imagine any… can these people?

(My 00.load.t tests will BAIL_OUT if my modules fail to load. And my regular tests just use modules, and they open files, set up DB connections and what-have-you using dies et al – as would any other program.)

This reminded me that I had left I want better testing practices as a draft post. I'd like to see us all put away our computers at the next QA workshop and just talk about what a test file should be, what it should do, and how it should do it.

As far as use_ok and require_ok, they have their place. Remember, though, that you can't rely on a load.t if you are merely running a single test file.

That’s fine. If I’m merely manually running a single test program, then crashing upon module load failure is exactly what I want. The sole point of having a 00.load.t at all, as far as I’m concerned, is to reduce noise from repetitive errors. (There’s no point to getting the same error over and over, sprinkled through a mass of dozens to thousands of test failures utterly devoid of any significance.)

Ovid: I had hoped you would tell me what arguments these people make! My own imagination does not stretch far enough to come up with any – but since they’re arguing for it, they have to have some reason that I just can’t think of, and since you’ve talked to them you must have heard one or several… no?

This is pretty common, and lazy. But programming doesnt happen in a vacuum. In many software environments, ppl don't even write tests.. especially in Wall Street. Getting the product out the day is more crucial, and sometimes u gotta skip a test or 2. Give me a raise, and a bonus, and i'll write better tests, deal? =)

Leave a comment

About Ovid

user-pic Have Perl; Will Travel. Freelance Perl/Testing/Agile consultant. Photo by http://www.circle23.com/. Warning: that site is not safe for work. The photographer is a good friend of mine, though, and it's appropriate to credit his work.