brian d foy

"You can get fired for not knowing Perl" I'm still sad that I had to miss the Nordic Perl Workshop in Reykjavík, but I'm going through talks that they've linked on their @nordicperl2010 Twitter feed. I like Tryggvi Björgvinsson's mutation testing talk, especially since he tells how he got a lecturer fired for not knowing Perl. I wish I was there to see if, but I only get to see the slides.

He points out an important but often ignored part of Perl's testing culture: we let the foxes guard the henhouse. That is, the code author and test author are often the same person. Not only that, but CPAN Testers runs the tests that the author provides. Tryggvi asks "Who watches the watchers" (with plenty of appropriate artwork).

How can we suss out problems with tests? One idea is to mutate the source code to see if the tests fail. For instance, we might change a comparison operator. The original code might test for greater than:

if( $foo > 9 ) { ... }

A mutation, or (mutant ninja) turtle in his parlance, changes that > to a <

if( $foo < 9 ) { ... }

Supposedly, a proper complete test suite should now have failures. That complete test suite would have green boxes in all the categories that you'd get from Devel::Cover, including conditional coverage for values on all sides of the boundary.

He shows syntactically valid changes, but random changes might be interesting too. What if none of the files even compile, but the tests still pass? You might think that's odd, but there are a couple of distributions that test pod, pod coverage, and kwalitee, but no real code tests.

Tryggvi doesn't note where he's uploaded Ooze, the software to create and test the turtles, but maybe he'll put it on CPAN if enough people bug him about it.