Software Test Podcast
I've recently been listening to Software Test Podcast. And I was particularly struck by episode 41, in which the presenters interview James Whittaker. He used to be Head Testing Honcho at Google until moving to Microsoft a year and a bit ago.
Almost all of what he says is good news for the perl community. The way we value testing and make it part of the community and how in our day jobs with perl most of us have "testing as an activity not a role", as he puts it, is not common elsewhere, but elsewhere is catching up to what we've been doing.
There is some bad news for us though. Almost all perl code has terrible diagnostics, so if we happen to ship something buggy, it can be awful hard to figure out what's going wrong even if you've got a shiny new test case written for the bug.
Of course, this is where breaking your code into small units with well-defined interfaces, interfaces that you can easily test, comes in handy. We rarely do proper unit testing, and if we wrote our code with unit testing in mind, it would make the sort of exploratory testing that makes it easier to find where a bug is a darned sight easier.
I've been thinking a lot recently about quality control versus quality assurance. And I think that we're still mostly in the world of quality control: we look for bugs, react to bug reports, and we fix 'em. Quality assurance is a much bigger concept. It includes quality control, but that's only a small part of it. Quality assurance is mainly about designing systems and working in such a way that it becomes harder for a bug to escape from your development environment in the first place, and that if it does it is easy to find. And it incorporates feedback. You learn from your mistakes and try to figure out how you can improve your practices so that that class of bug escapes less often. This is real software engineering, whereas most of us still work as artisans.
I'm convinced that unit testing is a big part of quality assurance. Another big part that I'm wrestling with is logging and diagnostics. Logging is, of course, easy to do - we have Log4perl. But doing it is only half - no, only a tenth - of the problem. Figuring out what to log so that the logs can help find bugs is something that we haven't figured out yet. It's something that we could learn from outside our community though, if only we could figure out where to ask for help.
I'm in yur test resultz analyzn yr failz
If only we could get actual users of our software to give meaningful reviews in a structured way, instead of just ranting about the bits they don't like..
Which makes me wonder if there ought to be some sort of structured review/survey site/system.
At the very least a kwalitee (or similar) score for "does this module document its error messages".. I keep meaning to attempt reviewing based on a structured set of points to cover.
I'm making this comment up as I go along, should sit down and think about it some more..
I think you hit the mark as to why Perl programmers are often thought of differently from other programmers with this quote: "This is real software engineering, whereas most of us still work as artisans." You see more artisans who primarily program in Perl than any other language I've seen.
Most programmers in almost all languages are artisans (the exceptions being those who use Ada and those who work in safety-critical areas where people can die expensively, like avionics and railway signalling), but I agree, perl programmers are even more likely than most to be artisans. I suppose there's a 99.9% chance that some random perl dev is, but it's only 99% if you don't limit yourself to perl. I don't consider that to be a significant difference.