Your Test Suite is Broken
Here are some indicators that your test suite is broken:
- It has any failing tests.
- It emits anything other than test information.
- It takes too long to run.
People are going to have issues with the second item on that list but they're definitely going to argue with that last one. I don't care. It's time to throw the gauntlet down. Let's look at these three points in detail.
Any failing tests
If a test suite has any failing tests, even "known" failures, developers learn to overlook failing tests. Those who've been testing for any length of time will generally agree with this. It's a bad thing. In fact, it's enough of a bad thing that every developer (well, except one) that I've spoken with agrees that leaving broken tests is bad, so I won't belabour this point.
Any "non-test" output
This one throws a few people off and I confess I'm guilty of ignoring this myself; it's easy to ignore a warning which is hard to suppress and just shrug. However, like the "any failing tests" rule, this should be dealt with quickly. Specifically, if you start getting accustomed to output which isn't test related, sooner or later you'll see non-test output which might indicate a bug but you'll learn ignore it. I still remember cleaning up a test suite warning which had existed for months, only to discover that the warning was indicative of a bug in the test program, making the tests useless.
Long running tests
People will agree that this is annoying, but they won't necessarily agree that their test suite is broken. I argue that it is. I strongly argue that it is. The first problem is the "downtime" problem:
Sure, many argue "but the developer can choose another task to handle at that point", but it's often very hard to find a meaningful task that only takes an hour or two. Plus, many developers are simply going to wander off for a bit. If you expect seven hours of productivity a day and you have seven developers who run their full one-hour test suite once a day, you've lost a day of work every day. It's like having six developers!
No, I'm lying. It's like having less than six developers because they won't come back the second the test suite is done, or they won't be mentally rarin'-to-go the second the test suite is done. A single failure means rerunning that suite, so that's also more time lost. Long-running test suites cause developers to lose productivity. It causes them to lose serious amounts of productivity. (I really should start tracking this at work so I can show management the costs involved. They're staggering!)
So what happens then? Forget for a moment what you think should happen. I think employers should double my paycheck every year. Ain't gonna happen. Instead, what happens is reality. Even if management is not pushing the programmers to be more productive, the developers themselves will want to be more productive and they've found a trick to make this happen. I personally have experienced this trick at five teams across three companies with long-running test suites. I've had many developers in other companies tell me that their teams use the same trick themselves (remember, I'm "that testing guy". People talk to me about this stuff all the time).
Here's the trick: don't always run the full test suite.
Remember: I've had this happen at every company I've worked at with long-running test suites. Many developers have told me their teams do the same thing with their long-running test suites. They don't always run those tests. Forget what "should" be done. This is reality. When you're under pressure to deliver, those deadlines are looming and you just know that X only affects Y and X is well-tested and you can slip this in really quickly and no one will notice, it's easy to commit now and not wait an hour or two. I've seen it eventually happen to every team with a long-running test suite. I'm sure there are exceptions, but I've never seen one first-hand. Sooner or later that failed test is going to creep in.
At my current team, you can't commit to trunk unless you have a stuffed camel on your desk. One of the most common questions on our team is "who has the camel?"
Possibly the next most popular question is "do you mind if I commit this small change to trunk anyway? It doesn't affect anything."
It doesn't affect anything? Really? So why is it being committed? Of course, I've asked this question, too.
Mind you, these aren't lazy developers. These aren't bad or unconscientious developers. These are developers who have deadlines and have to get things done. They have to make individual value calls on whether or not the risk is worth the reward and usually they're right. When they're wrong, the effect cascades across everyone's work. As a result, I've lost a huge amount of time today trying to debug test failures that I didn't realize were in trunk. This is not the first time this has happened to me.
Maybe your team is different. Maybe your developers are so incredibly careful and meticulous that your three-hour test suite is run every single time it should be. Maybe your developers are so conscientious that they immediately find a task which should take two hours and fifty minutes when they run that three hour test suite. And maybe your developers are so anal-retentive that I'd want to hang myself after working with them for more than a week.
For the rest of us, a long-running test suite means a significant loss in productivity and a drop in quality. I've seen this too many times to think that my experiences are anomalous. There's a lot of interesting stuff which needs to be done around testing, but I think more stuff needs to be done around speeding up test suites. For the vast majority of Perl users, this is not a problem. For those of us suffering from this problem, it's pretty damned serious. I've done a lot of work in this area, but more needs to be done. Much more.