Tim King

rockyb’s recent post about B::DeparseTree contained several insights on testability and writing good tests. Here are my takeaways.

• Testing a 6,000-line module is as difficult as it sounds. Long methods and large classes are code smells that made Martin Fowler’s Refactoring book. It’s often nigh impossible to unit-test a monolith. It might be time to do some separating of concerns. This will also—no surprise for testing aficionados like me—make it easier to maintain and extend the framework. Rocky notes some specifics under “The need for modularity” in his post, e.g., don’t repeat yourself, separate data from presentation, separate interface from implementation, and separate version-specific behaviors from version-agnostic behaviors.

• Understanding a 500-line test is as difficult as it sounds. Long, complex tests made Gerard Meszaros’s list of test smells, in his book xUnit Test Patterns. And as we’ll see as we go on, we can’t ignore the stink just because we’re Test::More purists: his advice applies equally to class-based and procedural testing.^[1]

• Tests should not depend on each other. Rocky noted that “the slightest error” would generate thousands of lines of follow-on test failures. He didn’t explicitly say so, but I suspect that indicates that the first test failure left the fixture in an invalid state, causing all other tests to fail. This is what Meszaros calls “Data Sensitivity.” Each test needs to start with a clean fixture.

• Each test should test one clear feature or unit of code. This is part of what causes “frail” tests, what Meszaros calls “Fragile Tests.” One big takeaway from Rocky’s post is the technique of round-trip testing. That is, you take the output of Deparse, which should be executable Perl, run it, and see if it generates the same behavior as the original snippet of code. He uses this with self-testing scripts so that the decompiled test code can verify itself. We can use a similar principle to validate HTML or other markup, for example, by asking what output we expect the markup to produce and rendering only those aspects of the generated markup.

• Each test failure should clearly identify which feature or piece of code is broken. This is related to the bullet above but from the perspective of the programmer running the tests. If you can’t tell what went wrong, your tests are not helping you understand or write your code. Meszaros calls this “Obscure Test,” and one consequence is that you can’t use your tests as documentation. It can also result in buggy tests and high maintenance costs. This is why I prefer the four-phase test structure—setup, test, verify, and cleanup—with which you can see at a glance what each test does and why it failed. It’s also important to use complete assertion messages: each test failure should read like a good bug report, indicating what action we took, what we expected to happen, and what occurred instead.

All in all, an insightful read.

Peace, love, and may all your TAP output turn green…

This post originally appeared on The Perl Shop blog as “Testing Insights from B::DeparseTree.”

[1] This distinction is a bit of a myth as well, which I touch on in Testing Strategies for Modern Perl. Test::Class and other xUnit-like frameworks don’t supplant procedural testing practices. Rather, they add new tools by which you can manage your tests. In particular, they can help you manage a collection of small, well-defined test methods. And yes, I can already hear you saying, “I do that with subtest $test_name => sub {}.” Yes, that’s exactly what I mean. I just happen to use sub test_name : Test() {} instead. Plus I can (a) run a named test method in isolation from the command line, (b) set up test fixtures before running each method in a module with a line of code, (c) automatically clean up after each test method (even failing ones) with a line of code, (d) inherit test fixtures across a whole set of test classes, and (e) abort a failed test method with a single line of code without affecting any other test methods.