David Cantrell

At work I recently set up a Jenkins instance to run all our tests whenever someone pushes changes to the master branch of our git repo. And, after changing some stuff in our code that made the perl debugger cry*, I added Devel::Cover to the mix. So now, Jenkins not only notifies us by email whenever someone is naughty and makes the tests fail, it also generates a lovely report of how much of our code is actually tested.

Naturally, the first thing I looked at was how well-covered our tests were. This seems like a bit of a weird thing to do, but it has exposed bugs in the tests themselves. Let me explain just one example.

The application under test includes an online shopping thing. Users can add things to their "shopping basket" both when they are logged in, but also when they are not logged in. In the latter case, they will be prompted to either log in or sign up for an account when they try to pay us. All fairly normal.

Now, we have a test "add a thing to the shopping basket". What should happen is 95% the same regardless of whether the user is logged in or not. So, of course, someone had the sensible idea of dealing with both cases in one piece of code with a little condition in there to check one field or another depending on whether the user is logged in or not. Devel::Cover tells me that we're only taking one of the branches, and not the other, thus exposing a hole in the tests: we're only running the test "add a thing to the shopping basket" with the user logged in.

Do you test your tests? Don't you think you should?