A Standard Thought Process For Testing
Testing is, in some sense, a mess. Part of the problem is similar to the dynamic/static schism: nobody seems to agree on what those terms mean. Case in point: what's "integration testing"? Here's the definition from the excellent Code Complete 2:
Integration testing is the combined execution of two or more classes, packages, components, or subsystems that have been created by multiple programmers or programming teams. This kind of testing typically starts as soon as there are two classes to test and continues until the entire system is complete.
Read that carefully. Look for a flaw.
So, let's say that you're working for a company and you're the only programmer. By the aforementioned definition, you're not capable of doing integration testing because there are no multiple programmers or teams. In fact, the "component testing" definition has a similar flaw. Lone programmers are not capable of it if you accept the "Code Complete" definition.
Mind you, I don't want to use this as an excuse to rip into an otherwise excellent book. There are many areas of testing which are, um, not terribly well defined. BDD (Behavior Driven Development), for example, is often described in rather curious terms to the point where some people admit that they just don't understand it. I've never done it, but I've proposed similar things before and I think I understand it (I think it would be a nice way to kill off FIT testing).
I would love to see a standardised description of different testing techniques along with plenty of hard data (I've never seen this for FIT or BDD) to suggest which give more bang for the buck. For example, are there areas where QA is less useful than others? Having a single repository of this information, along with references, would be great.
Freelance Perl/Testing/Agile consultant and trainer. See http://www.allaroundtheworld.fr/ for our services. If you have a problem with Perl, we will solve it for you.
And don't forget to buy my book! http://www.amazon.com/Beginning-Perl-Curtis-Poe/dp/1118013840/
These terms are just hands pointing at the moon. They aren't the moon themselves, they just point towards something. It's in their nature to be loose & different to different people. We shouldn't expect it to be otherwise.
@Luke: then how can we really have a clear dialogue about testing if we can't even agree what underlying terms mean? In fact, if our meanings are close, but subtly different, it would be easy to not realize we're talking about different things, thus making meaningful conversation very difficult. (How many arguments stem from a similar problem?)
I don't think, in general, that the principles and definitions put forth in Code Complete are worded for the lone programmer. I don't imagine that you'd have enough of an argument on your hands if, for purposes of testing, you consider the me-who-wrote-the-existing-system to be different from the me-who-wrote-this-new-addition, and proceed to call the testing of these together "integration testing."
As Perl programmers, I think we are hyperaware of exceptions to the rule. When trying to write in a clear, accessible manner on a technical subject, it's almost inevitable that your descriptions will not align 100% with the real world. I'm not sure that an attempt to do so would produce satisfactory axoims. You'd sacrifice understandability for correctness.
@ovid it's a tough problem to be sure. It's do-able with a static group of people that can spend the time to listen to each others definitions and agree come to consensus on terms. It's radically difficult to do this with a dynamic group of people that always has new comers from different backgrounds.
Perhaps the best strategy is to use inquiry instead of advocacy as a first effort in discussion. Rather than using our built-in assumptions about what that person means by their terms, when we disagree or are confused by someone we should first inquire. Maybe they are using a term differently but their underlying point is what's interesting.
But I don't disagree with your point. Meaningful conversations are difficult - they are only possible with mutual understanding, respect and patience.
The first flaw I noticed was earlier in the sentence: "Integration testing is the combined execution of two or more classes, packages, components, or subsystems...". So if I write a unit test, and it doesn't mock out *everything* other than the code under test, that makes it an integration test?
For example, if I'm testing some calendar code, I'm probably not going to try to write a mock version of DateTime; I'm going to assume it works and is sufficiently predictable that it won't complicate my testing. But by the above definition, that's not a unit test, it's an integration test.
To me, the difference is a matter of focus. If the tests are focused on a single component, assuming other components function correctly, then they're unit tests. If the tests are focused on the interactions between several more or less equal components, then they're integration tests.
Look into Wittgenstein’s work someday. :-)