Sick of being mocked by unit tests

Back in 2002, the company I worked for had a valuable client with a terrible problem: seems the developer of their POS (point of sale, or "cash register") system sent them a bill for their license: five year's worth. The company disputed the bill and the developer informed them that their POS system would be remotely disabled in 23 days. So our client contacted us and the owner informed a colleague and myself that we had just over three weeks to develop a POS for the client. Never mind that neither of us had any experience with swiped credit cards, or bar code readers, or cash registers, or, or, or ...

This is the story of how I learned to loathe unit tests.

My colleague insisted (correctly) that we write plenty of tests. Previously I had written a few for some open source code, but I wasn't really comfortable with testing, so this was going to be a crash course (for you literary types: the word "crash" is foreshadowing).

Of course, we were going to do things the "right" way, so we made sure to use mock objects and carefully unit test everything to ensure that we could fully exercise all paths through each method in fine-grained detail. It was a lot of work and while I never really felt comfortable mocking up code, I was feeling very comfortable with our test suite. I was watching the tests pass, I was catching bugs that I never would have caught before and bugs were easy to write tests for and fix. Development was so fast that even though we were tired from working overtime and weekends, it looked like we were going to hit our deadline and our client wouldn't go out of business.

But here's the thing with mocked objects: they're not real objects. Think about what happens when you run an application: all the bits and pieces talk to one another. With mocked objects, they don't; they're talking to mocked objects. If the behavior of your mocked objects diverge from the behavior of the code you're mocking, your tests may very well not reveal this. Even if you get all of your mocks perfect, over time your code's behavior will evolve and now you have to not only maintain your classes, but the mocks mirroring them. Hey, higher development costs! Job security!

Sure, for a statically typed language you might mitigate this with proper introspection to build mocked objects, but type systems are generally primitive enough that you can still get plenty of errors. Mocking up a temperature gauge that returns integers intended as Fahrenheit when the actual one returns Celsius is a subtle problem (which suggests other things about how to resolve the problem, but that's a digression).

After writing tons of tests and feeling very comfortable with the code, we decided to do some manual testing on the entire system. Naturally, it failed miserably and wouldn't even launch. We mocked up so many things in such a short period of time and were writing so much new code that it's not surprising that none of our subsystems knew how to talk to one another. We had mocks upon mocks upon mocks and this should really be an anti-pattern. Mocks should be a last resort, not a first.

We worked through the issues and on the 21st day our CTO flew out to California and installed our systems and hallelujah, they worked! By day 23, we fixed some minor bugs and had a couple of feature requests, but those mocked unit tests almost sunk our client.

This, by the way, is one of the reasons I wrote and released Sub::Override. Sometimes you need to mock something up not because you want to skip the behavior of an entire class, but because you have one tiny method that calls out to a server that's not available in your dev environment or returns non-deterministic results. Rather than mocking up everything to avoid that one method, you just override the method in question and you can still talk to everything else.

So this raises an interesting question: why is unit testing so obsessed with mocking everything up to ensure that every method is tested in isolation? Well, as it turns out, that's a relatively recent thing. Unit testing was largely pushed and evangelized by Kent Beck. Recently, David Heinemeier Hansson had a series of interesting Google Hangouts with Martin Fowler and Kent Beck. The series is entitled "Is TDD Dead?" (I have some thoughts on that) and here's the first episode:

In that discussion, both Beck and Fowler make it very clear that there is nothing about unit testing which requires mocking. That's something which sort of came along later as others seized the ideas and ran with them. Beck and Fowler, both of whom are excellent developers, admit that they tend not to use mocks when they write their tests. In fact, Kent Beck said (around the 21 minute mark) "My personal practice is that I mock almost nothing." He then points out that overmocking your code in tests means that your code is now tightly coupled to the implementation and not to the APIs, thus making refactoring much harder.

Martin Fowler later points out that "there is nothing in either TDD or unit testing that says you have to have that kind of isolation (mocking)".

Now before anyone accuses me of pulling an appeal to authority, I want to point out something very important: many people who criticize the appeal to authority actually don't understand the logical fallacy. It's also known as the "fallacious authority" or "questionable authority" fallacy. There is nothing wrong with saying "my oncologist says laetrile is useless for treating cancer" because an oncologist is much more likely to know about that than, say, your woo-woo friend who recommends flower therapy and knows of a friend of a friend who heard that laetrile is effective. The appeal to authority is only a fallacy when the person whose authority you're relying on isn't an authority on the subject in question.

In other words: experts are more likely to know what they're talking about than non-experts. Further, as someone who's been doing extensive work with testing for many, many years, I'm going to put myself forward as an expert in this area. Don't drink the Kool-aid. Don't mock things that don't need to be mocked. Don't buy into dogma.

So I don't really loathe unit tests; I loathe what they've become in the minds of many. I loathe following the dogma put down by zealots who have their idea of the One True Way of testing and you're a damned heathen if you disagree. Unit tests are, well, an ephemeral thing and different people define them differently. That's OK.

Until we develop better systems for identifying issues in code, the main question is: can you sleep at night? Do you feel that your tests really have made your code better, or are you feeling guilty because you didn't follow someone else's definition of the right way to do things?

13 Comments

I've always found religious adherence to development patterns ( REST, TDD, Scrum, OOP, MVC, Documenting Everything, Git Workflow patterns ) to be totally counterproductive. Frequently programmers who are obsessed with rules paint themselves into a corner, and spend all day figuring out how to solve a problem that only exists in their head ( this needs to be an HTTP DELETE, but browsers don't support it! Gaaaahh!!! I can't live in a world without all the verbs! ) The focus should be on developing maintainable, stable, fast, and useful software, not on impressing yourself with how many best practices check boxes you ticked

The practice that bothers me is mocking a database. Database's have characteristics of their own (uniqueness constraints, foreign keys, maybe stored procedures and triggers), and you need to match those characteristics precisely, which means you really need something like a test db set-up exactly like the live one, with a known subset of data...

Great article Ovid.

I also recently read this document: http://www.rbcs-us.com/documents/Why-Most-Unit-Testing-is-Waste.pdf by James Coplien referenced in the DHH post: http://david.heinemeierhansson.com/2014/tdd-is-dead-long-live-testing.html

The Coplien article had some very interesting points including the need to delete tests as they become redundant over time.

I think what is needed here is balance - finding the right testing strategy for each context.

When mocking i tend to stick to mocking things purely outside my code. One of the least-mentioned tools, that is incredibly useful for that is Test::MockTime.

I have always thought that Mocking things within a project for testing purposes is silly - if I need access to a database then I am not going to Mock that access I am going to provide a test database with real test data and test to see if the system behaves as I would expect.

At the office, we tend to use unit tests as ways of ensuring that re-factoring doesn't break old functionality. Write a unit test that passes when a class or method is used, refactor that class or method and ensure the test is still passed.

Here is what I dislike about unit tests:

They do not necessarily transfer when you refactor. For example, if you add a helper class or move methods from one class to another, your unit tests will have to change. now, if those classes are not exposed to your users and your "public" interfaces did not change, was that work needed?

If instead you had (block box) tests at the public interface level only, then you can change your implementation entirely without changing your tests or user facing docs.

So why add unit tests for "private", internal components if you can do all your testing at the public interface level? Several reasons:

1) speed and completeness - unit tests may be faster at testing an isolated piece of code. Especially if you are trying to be exhaustive and run a ton of example cases.

2) You are working on a team and the other components are not ready yet.

3) coverage - sometimes there are hard to hit corner cases that are simpler to hit at the unit level. Mocking also comes in handy here if there are error conditions that are had to simulate with real classes.

I am sure ewe can think of more, but my point is that if you can get 100% coverage using black-box system-level tests you will be able to keep those tests even in the face of major internal refactorings.

Perhaps Test::MockTime is the only useful mock? For other things like filesystem, database, just use the real thing.

One of the better articles I’ve read in this back&forth:

Gary Bernhardt: Test Isolation Is About Avoiding Mocks

Nice post. I also loathe mocked objects nearly universally. Also, some implementations are problematic -- e.g. Test::MockObject on the CPAN has a number of implementation issues that cause me to pull its use out of any project I can.

Here's one module I wrote which stands in for the server side of a client-server communication: https://metacpan.org/pod/Test::LWP::UserAgent - useful when you are writing a network client to communicate with a third-party server, and you want to simulate various behaviours of the server. Obviously it is only as useful as your ability to mimick the behaviour of the server, but it can be quite helpful for ensuring that your client can handle a variety of successful and failure responses.

Your boss told you to write a POS system (which you had no experience with) in three weeks? Really?

I would have either demanded an enormous bonus or invited him(her) to perform a certain anatomically impossible act.

Very nice "exposition" Ovid, and you are right in saying "So I don't really loathe unit tests; I loathe what they've become in the minds of many", because unit tests have their place, but the problems will surely start to appear once you rely only on them for your testing.

Martin Fowler published a summary of the full discussion between Kent Beck, David HH and himself, here:

http://martinfowler.com/articles/is-tdd-dead/

Well worth the read.

Great article, Love reading real world experiences.

We have had lots of bad mocking experiences when it comes to large databases. When you have 10TB of data and 3-4billion rows and the system have 500 active handlers each competing for resources. Your mocked test has an impossible job to predict a real world example. Very often people are lazy creating large joins of subqueries in subqueries not taking into account that in production joining large tables like this take out all the tempspace.

Leave a comment

About Ovid

user-pic Freelance Perl/Testing/Agile consultant and trainer. See http://www.allaroundtheworld.fr/ for our services. If you have a problem with Perl, we will solve it for you. And don't forget to buy my book! http://www.amazon.com/Beginning-Perl-Curtis-Poe/dp/1118013840/