Buddy Burden

So the most important takeaway from this blog post is that Test::File version 1.31 is now available on CPAN, with some new functions for your testing pleasure: dir_exists_ok, dir_contains_ok, file_contains_like, and file_contains_unlike.  Check it out.

But I thought it might also be enlightening if I went into a little more detail about the story behind this release.  Now, I already talked about how I became co-maintainer for this module.  At the time, I asked brian d foy if there was anything special he wanted me to do for releasing, any particular procedures he wanted followed.  As I said, it’s his car—I’m just driving it for a bit—so I want to do things the way he expects them done.  The only thing he really mentioned was that he typically puts out a developer release and makes sure it passes CPAN Testers before doing an official release.  Of course, I replied, I do that too.

Actually, I thought everyone did that.  But his statement got me thinking ... it reminds me of our vists to Chuck E. Cheese’s.  I know that seems like a stretch, but hear me out.  Every time we go to our local CEC, my elder son points out the prominent sign by the front door.  Yes, that’s right: firearms are not allowed in the kiddie arcade/pizza parlor.  What is the world coming to?  But my son’s point in drawing attention to this sign is that, somewhere, at some point, someone must have tried to bring a gun into Chuck E. Cheese’s.  Otherwise, they’d have no need for the sign, right?

And, likewise, the fact that brian d foy felt the need to tell me I should be running my releases through CPAN Testers before making them official means that there are still folks out there who don’t.  So, if you’re one of those folks, let me tell you how to make a CPAN release:

1. Write/fix your code.
2. Mark it as a developer release.
3. Upload it to PAUSE.
4. Wait a few days.
5. Go look at CPAN Testers and find out how bad you fucked up.
6. Repeat until you didn’t fuck up.
7. Change nothing but the version number, making it a production (i.e. non-developer) release, and reupload to PAUSE.

Now, typically when you screw something up, it’s blatant.  Narly every CPAN Tester will fail you, and it’s usually obvious what you did wrong.  Forgot to include a prerequisite.  Forgot to specify a particular version of a prerequisite.  Forgot to regenerate your MANIFEST file, or your META.yaml/META.json.  Et cetera.

But, every once in a while, you get a just a few weird errors that you can’t figure out.  If you’re lucky, they’re all from the same (typically older) Perl version, or maybe the same operating system, and you can either try to figure out how to fix that, or at the very least just update your POD to say you don’t support that version/OS.  If you’re not lucky ...

So, I wasn’t lucky.  I put out 1.28_02 of Test::File (after discovering the hard way there was already a 1.28_01) and I ended up with like 15 failures.  Out of like 200 or so ... I can’t figure out how to make CPAN Testers count the total reports for me (except on my author page, where it only lists totals for the most recent distros).  So we’re talking somewhere in the ballpark of 7% failures here.  That’s not much ... right?

Now, I wonder how other people would deal with this situation.  I bet we’re all different.  I bet it’s just like balancing your checkbook (assuming my readers are old enough to remember the days when you had to do that by hand) and coming up 3 cents short.  Some of us will just shrug and say “fuck it” and move on with our lives.  It’s 3 cents, man!  Who cares?  But others of us will go back and hunt down that lousy stinkin’ 3 cents like our lives depended on it.  Otherwise it will just mock us mercilessly, and we won’t be able to sleep at night knowing it’s out there.

I have a little of both in me.  If we really were talking about balancing checkbooks, I couldn’t let the 3 cents slide.  It’s a touch of OCD I inherited from my mother.  But a 7% failure rate on my software doesn’t really seem that bad ... I mean, I’m not talking about medical software, or airline autopilot programs.  And I actually looked at the failures, and they all look fine ... I can’t see any reason why they would be failing, actually.  Still, I looked up the email addy for one of the testers (one who provided 3 or 4 of the failures) and sent him off an email.  I’m sure he’s very busy with other stuff (like putting out a new version of Perl), so I wasn’t offended when he didn’t get back to me.

In the end, I decided I would check previous releases of Test::File and see what sort of error rate brian d foy was willing to live with ... after all, it’s his car, right?  So, by my rough estimate (still not being able to find the exact count anywhere—if there’s not something I’m blatantly missing, I wonder if the CPAN Testers folks would be willing to add this) the last production release had an error rate of around 5%.  I’m close enough to that, I figure.  So I went ahead and released version 1.30 (having discovered, not too late this time, that there was a 1.29 I didn’t know about).

And 1.30 had an error rate closer to 3%, which is pretty awesome.  I figured all was well.  Until I actually went to install my newly released version on my own box.

And it failed.

Now, this completely floored me.  I had run the exact same tests on this exact same box so many times it wasn’t funny.  I’m primarily a TDD developer, especially when it comes to code I release into the wild, so I was running those tests several times a minute at some points.  Certainly I’ve run the entire test suite over and over again ... hell, my script which I use to toss the tarfile up to PAUSE won’t run if the tests fail (which is as it should be).  There’s no way (I says to myself) that these tests can be failing now.

Yet they were.

I tried again.  I exited out of my CPAN shell and got back in.  I made sure my CPAN was the most recent version and reloaded it.  Still failing.  With errors that made no sense: the file called not_readable (and I’ll give you 1 guess as to why it’s named that) was apparently not failing the readable test.  How could this be?  Was the code that set the permissions to make it unreadble not working?  But, if that were the case, how had the code suddenly stopped working when I hadn’t changed anything?

I bet some of you have cottoned on to what’s going on here.  Please, feel free to be very smug about it.  I deserve it.

The problem, as it turned out, was that I was running all my tests, every time, as me.  But I was installing, via the CPAN shell, as root.  And, as you may know, root doesn’t have any problems reading files, even if they’re unreadable.  D’oh!  I had cribbed the non_readable file itself from brian d foy’s existing code, but I had failed to properly copy over his tests, all of which specifically had SKIP blocks and checked the UID (and EUID) to see if it was root running the tests.  I added those and put out a version 1.31 and that’s where we are today.  Thus far, my error rate is under 1%.  Woohoo!

Anyway, I’m not entirely sure what the ultimate moral of the story should be ... don’t discount CPAN Testers too quickly? always check to see what versions are out there before you release a module for the first time? always try to install your module as root before releasing? know your CPAN Testers error rate? something completely different that I missed myself?  Probably better if the reader decides that for themselves, I think.  But I figured it was illuminating enough to share with the group, at least.  Hopefully you agree.