Ovid

We're still hacking away on the Veure MMORPG and things are moving forward nicely, but I thought some folks would like to hear more about our development process. This post is about our test suite. I'd love to hear how it compares to yours.

Here's the full output:

$ prove -l t
t/001sanity.t ... ok   
t/perlcritic.t .. ok     
t/sqitch.t ...... ok     
t/tcm.t ......... ok       
All tests successful.
Files=4, Tests=740, 654 wallclock secs ( 1.57 usr  0.20 sys + 742.40 cusr 15.79 csys = 759.96 CPU)
Result: PASS

Let's break that down so you can see what we've set up. You'll note that what we've built strongly follows my Zen of Application Test Suites recommendations.

First, the 001sanity.t bails out of the test suite if any critical problems are found:

• A security check (yeah, this is vague)
• sqitch database changes not fully deployed
• Redis is not running
• Output is not UTF-8

Any of the above issues is considered "stop everything and fix." If the failures occur, the developer will usually also get a useful diagnostic message telling them the likely cause of the failure and how to resolve it.

The perlcritic tests ensures we pass severity 5 (gentle) checks. We're close to passing on 4, but we have a strong enough test suite that I'm not terribly worried about it.

The t/sqitch.t test lets us test our sqitch changes. I get very tired of broken database migrations, but we're finding more and more issues with a standard database migration strategy (particularly the tension between data and its structure, but more on that in a later post).

Finally, we have the t/tcm.t tests. The core of our test suite is built on top of Test::Class::Moose. Once you get used to it, it really speeds up test development time and full test suite run time. However, running an individual test class can be a bit slower because it has to load the full app. It's a trade-off, but one I'm happy with.

You'll note that the summary line only claims 740 tests, but that's not true. For example, here is a regression test for a bug from the Veure Archives functionality. A particular URL should be available to players, but if you're not logged in, it should redirect you to the root URL of the archives rather than generating a 500:

sub test_travel_doesnt_500_when_logged_out {
    my $test = shift;
    my $mech = $test->test_mech;
    $mech->login;
    $mech->get_ok(
        '/archive/travel/sol/fake-station-name-for-a-blog-post',
        'We should be able to get travel pages for stations we have visited'
    );
    $mech->logout;
    $mech->get_ok(
        '/archive/travel/sol/fake-station-name-for-a-blog-post',
        '... but if we are logged out, it should not be a 500'
    );
    is $mech->path, '/archive', '... and we should instead be redirected to the /archive';
}

You probably count three tests there, but in true xUnit philosophy, those three tests are for one feature, so they're counted as one test. If you count individual tests, we have about 6,000 tests for our system and we're at 85% test coverage. What's nice is that these are real tests, not just "I happened to run this code" tests. When we break something, we find out pretty quickly.

The test suite also has no warnings, something we've been hyper-vigilant about. We do have some front-end exposure so we're getting Selenium tests in place and that will help with a major concern we've had.

Our fixture handling is also pretty easy to work with: Here are the fixture declarations for our combat tests:

with qw(
  Test::Role::Fixture::Characters
  Test::Role::Fixture::StationArea
  Test::Role::Fixture::Skills
  Test::Role::Fixture::Ships
);

As a bonus, those fixtures are loaded on demand. Consuming the roles merely declares an intent to use those fixtures.

The end result of all of this? We have multiple developers working on Veure and we find out fast if something has gone off the rails. Devs feel a heck of a lot more confident that their code does what it's supposed to do and merely having this level of quality and coverage for tests puts tremendous social pressure on devs to not drop the ball here. If they're not clear how to test something, they could hop on Slack and ask, but in reality, they check the docs/developer/testing.pod document and that tells them the vast majority of what they need to know.

Once we launch, I'm curious to see if we can maintain this level of quality.

Feel free to ask any questions and I'll answer what I can.