Ovid

Back in 2002, the company I worked for had a valuable client with a terrible problem: seems the developer of their POS (point of sale, or "cash register") system sent them a bill for their license: five year's worth. The company disputed the bill and the developer informed them that their POS system would be remotely disabled in 23 days. So our client contacted us and the owner informed a colleague and myself that we had just over three weeks to develop a POS for the client. Never mind that neither of us had any experience with swiped credit cards, or bar code readers, or cash registers, or, or, or ...

This is the story of how I learned to loathe unit tests.

My colleague insisted (correctly) that we write plenty of tests. Previously I had written a few for some open source code, but I wasn't really comfortable with testing, so this was going to be a crash course (for you literary types: the word "crash" is foreshadowing).

Of course, we were going to do things the "right" way, so we made sure to use mock objects and carefully unit test everything to ensure that we could fully exercise all paths through each method in fine-grained detail. It was a lot of work and while I never really felt comfortable mocking up code, I was feeling very comfortable with our test suite. I was watching the tests pass, I was catching bugs that I never would have caught before and bugs were easy to write tests for and fix. Development was so fast that even though we were tired from working overtime and weekends, it looked like we were going to hit our deadline and our client wouldn't go out of business.

But here's the thing with mocked objects: they're not real objects. Think about what happens when you run an application: all the bits and pieces talk to one another. With mocked objects, they don't; they're talking to mocked objects. If the behavior of your mocked objects diverge from the behavior of the code you're mocking, your tests may very well not reveal this. Even if you get all of your mocks perfect, over time your code's behavior will evolve and now you have to not only maintain your classes, but the mocks mirroring them. Hey, higher development costs! Job security!

Sure, for a statically typed language you might mitigate this with proper introspection to build mocked objects, but type systems are generally primitive enough that you can still get plenty of errors. Mocking up a temperature gauge that returns integers intended as Fahrenheit when the actual one returns Celsius is a subtle problem (which suggests other things about how to resolve the problem, but that's a digression).

After writing tons of tests and feeling very comfortable with the code, we decided to do some manual testing on the entire system. Naturally, it failed miserably and wouldn't even launch. We mocked up so many things in such a short period of time and were writing so much new code that it's not surprising that none of our subsystems knew how to talk to one another. We had mocks upon mocks upon mocks and this should really be an anti-pattern. Mocks should be a last resort, not a first.

We worked through the issues and on the 21st day our CTO flew out to California and installed our systems and hallelujah, they worked! By day 23, we fixed some minor bugs and had a couple of feature requests, but those mocked unit tests almost sunk our client.

This, by the way, is one of the reasons I wrote and released Sub::Override. Sometimes you need to mock something up not because you want to skip the behavior of an entire class, but because you have one tiny method that calls out to a server that's not available in your dev environment or returns non-deterministic results. Rather than mocking up everything to avoid that one method, you just override the method in question and you can still talk to everything else.

So this raises an interesting question: why is unit testing so obsessed with mocking everything up to ensure that every method is tested in isolation? Well, as it turns out, that's a relatively recent thing. Unit testing was largely pushed and evangelized by Kent Beck. Recently, David Heinemeier Hansson had a series of interesting Google Hangouts with Martin Fowler and Kent Beck. The series is entitled "Is TDD Dead?" (I have some thoughts on that) and here's the first episode:

In that discussion, both Beck and Fowler make it very clear that there is nothing about unit testing which requires mocking. That's something which sort of came along later as others seized the ideas and ran with them. Beck and Fowler, both of whom are excellent developers, admit that they tend not to use mocks when they write their tests. In fact, Kent Beck said (around the 21 minute mark) "My personal practice is that I mock almost nothing." He then points out that overmocking your code in tests means that your code is now tightly coupled to the implementation and not to the APIs, thus making refactoring much harder.

Martin Fowler later points out that "there is nothing in either TDD or unit testing that says you have to have that kind of isolation (mocking)".

Now before anyone accuses me of pulling an appeal to authority, I want to point out something very important: many people who criticize the appeal to authority actually don't understand the logical fallacy. It's also known as the "fallacious authority" or "questionable authority" fallacy. There is nothing wrong with saying "my oncologist says laetrile is useless for treating cancer" because an oncologist is much more likely to know about that than, say, your woo-woo friend who recommends flower therapy and knows of a friend of a friend who heard that laetrile is effective. The appeal to authority is only a fallacy when the person whose authority you're relying on isn't an authority on the subject in question.

In other words: experts are more likely to know what they're talking about than non-experts. Further, as someone who's been doing extensive work with testing for many, many years, I'm going to put myself forward as an expert in this area. Don't drink the Kool-aid. Don't mock things that don't need to be mocked. Don't buy into dogma.

So I don't really loathe unit tests; I loathe what they've become in the minds of many. I loathe following the dogma put down by zealots who have their idea of the One True Way of testing and you're a damned heathen if you disagree. Unit tests are, well, an ephemeral thing and different people define them differently. That's OK.

Until we develop better systems for identifying issues in code, the main question is: can you sleep at night? Do you feel that your tests really have made your code better, or are you feeling guilty because you didn't follow someone else's definition of the right way to do things?