die, you can now override the new signature_error_handler method in a subclass. Thanks to larryl for the patch.Hopefully you find these tidbits helpful. Let us know—here, on GitHub, or in an RT ticket—if you see any problems, or want to make any suggestions. Enjoy!
]]>Long Version:
If I’m known for anything on CPAN (and I’m not saying I am), it’s probably for my work on other people’s modules: my co-maintenance of Method::Signatures in particular, or perhaps my co-maintenance of Test::File (both of which I’ve talked about before). But the first CPAN module I ever released was Debuggit.
I wrote a post upon the occasion, on my other blog. I didn’t put it here, mostly because I hadn’t started this blog yet. But also because it was more of a philosophical musing on procrastination than a useful commentary about CPAN or my particular module.
In fact, I haven’t talked much about Debuggit on this blog. That’s partly because it’s one of those modules I wrote for me, and I don’t much care if anyone else uses it or not. I’m not in the business of evangelizing in general, but, if I were, I’d spend my time telling you how cool Method::Signatures is, not Debuggit. Debuggit is just a little thing, a bit nifty, but nothing you couldn’t write yourself, most likely. Nothing you haven’t already written yourself, most likely, or at least some version of it. But perhaps my version, which I’ve mucking about with for over 10 years now,* has something which your version(s) didn’t, or don’t, so perhaps it’s worth chatting about for a bit. Or you can just completely ignore me; I’m fine with that too.
So the idea behind Debuggit is, sometimes you need to put output in your Perl scripts to tell you the values of variables, or just to let you know how far your program is getting before it bombs out. You could use the debugger for this, but some people don’t like debuggers—I personally developed a distaste for them ever since I once spent 3 days tracking down a bug in a C++ program that it turned out didn’t even exist outside the debugger—or sometimes it’s hard to get to the code you need to debug (perhaps it’s in a BEGIN block, and you forgot how to make the debugger run through that, or perhaps it’s being run in mod_perl), or sometimes you just forget what all the single-letter debugger commands are and you just want an alternative. So you start putting in print statements, until you realize that STDOUT is buffered and you might not be seeing all the statements that are executed, so you change to print STDERR statements until you realize that warn would be much easier to type, so you do that until you realize that in some mod_perl setups warn and print STDERR can go to two different places, or what if you have a SIGWARN handler or you have Carp always doing stack traces? Anyways, you pepper your code full of output statements, of whatever stripe, until you find the problem, and then you take them all back out.
Until the next problem, when you have to put them all back in again.
So your next attempt is to create some global variable called $DEBUG which allows you turn all those output statements off at once, so now you just leave them in all the time. Of course, now they’re in all the time: they take up memory, and they take up execution time (even though the output never gets run, the if $DEBUG is always evaluated, over and over again). It’s not much memory or time, of course, and in most instances you’ll never even notice, but then again what if it’s in some tight loop, which is called by another loop? You can’t guarantee that it’ll never matter if you just leave them in ... right?
But then you discover use constant and the magic of compile-time folding, and warn("foo") if DEBUG turns out not to take any memory or execution time at all, if the DEBUG constant is set to 0. So that’s nifty. Of course, you may other have annoyances, like constantly having to remember to put spaces between your arguments so your output is readable, or what if your variable has leading or trailing spaces? that can be hard to see sometimes, and then if your variable is undefined, you get a nasty “uninitialized” warning, and, if you have warnings set to fatal, that will actually blow up your program itself, and what if your variable is an arrayref or hashref? then what you really want is to use Data::Dumper to print it out legibly, except you really don’t want to leave a use Data::Dumper lying around in your code for eternity because Data::Dumper can eat a healthy chunk of memory, depending on Perl version and DD version and various and sundry other things (phases of the moon? I have no idea, actually), so ...
So you really don’t want to have to think about all that crap, right? You just want to slap some debugging in your code, be able to leave it there forever, not have it take up any extra memory or CPU time, and not to have to remember too many special rules to make the output look nice.
You want Debuggit. You just never knew it.
Anyway, Debuggit handles all that, and more, like letting you trick out where your output goes, what it looks like when it gets there, and add little “debugging functions” like the default one that calls Data::Dumper for you:
use Debuggit;
my $hash = { some => 'very', complex => 'hashref' };
debuggit("hash is currently", DUMP => $hash) if DEBUG >= 3;
use Debuggit DataPrinter => 1;
my $hash = { some => 'very', complex => 'hashref' };
debuggit("hash is currently", DUMP => $hash) if DEBUG >= 3;
DataPrinter => 0 to get Data::Dumper. But this is good for now.
Anyways, I mostly put this on CPAN because I got tired of trying to send people a tarfile of it, not so I could convince the world to use my super-cool debugging module. But, then again, if you want to check it out, that’s cool too. Maybe it’ll be as useful for you as it has been for me.
* The original version of Debuggit was called Barefoot::debug, and my first commit to it was on August 28, 2000.
(By the way, most of these awesome new features came from the Damian, as I described in my post about Git and patches. The remainder come from schwern, who is of course the original author of MS. This time around, I really am just the guy who put it all together and uploaded it for all y’all.)
where ConstraintsOne thing that MooseX::Method::Signatures has always been able to do that MS (and consequently Method::Signatures::Modifiers) couldn’t was what are called where constraints. This is a Moose concept that’s a bit like on-the-fly typing. For instance, if you declare that your parameter is an Int, what you’re actually saying is, “please check the incoming value of this parameter and, if it’s not an integer, please throw a fit about it, thankyouverymuch.” If you wanted to make sure that you were getting an integer less than 10, say, what you could do is define a new type, say IntUnderTen. But that might be a huge pain in the butt, particularly if you had a lot of different maximum values, or if said maximum was parameterized. So an easier way to do it would be:
method foo (Int $bar where { $_ < 10 }) { do_stuff_with($bar) }
A while back, Ruz sent us a patch concerning default values. Your standard default value goes like so:
method foo ($bar = 'fred') { say $bar }
$obj->foo('sue'); # you get "sue"
$obj->foo(); # you get "fred"
$obj->foo(undef);
method foo ($bar = 'fred') { $bar //= 'fred'; say $bar }
method foo ($bar //= 'fred') { say $bar }
method foo ($bar ||= 'fred') { say $bar }
method foo ($bar ''= 'fred') { say $bar }
//= was a great idea, ||= was okay, but ''= was just too weird. But schwern and Damian came up with a way to satisfy everyone:
method foo ($bar = 'fred' when undef) { say $bar } # if it's not defined
method foo ($bar = 'fred' when 0) { say $bar } # if it's explicitly zero
method foo ($bar = 'fred' when '') { say $bar } # if it's the empty string
method foo ($bar = 'fred' when { !$_ }) { say $bar } # if it's false
method foo ($bar = 'fred' when { !/\S/ }) { say $bar } # if it's all spaces
method foo ($bar //= 'fred') { say $bar } # same as: = 'fred' when undef
We had long used Devel::BeginLift to make a MS method or function parse at compile-time, just like a real Perl sub. But it turns out this is a problem for our Windows brethren and sistren, because Devel::BeginLift doesn’t want to compile on Windows. Happily, schwern found another way to achieve the compile-time goodness and we’ve dropped DBL as a prerequisite. You should be able to MS to your heart’s content on Windows now.
Normally, if someone passes more arguments to you than you were expecting, that’s an error. But occasionally you find yourself in the position of needing to create a function or method that’s being called from someone else’s code, and perhaps they don’t play nice, or they are sending arguments that other people need but you don’t. In these sorts of cases, you could always just slap a slurpy parameter in there:
method foo ($bar, $baz, @stuff_I_dont_care_about) { do_stuff($bar, $baz) }
method foo ($bar, $baz, ...) { do_stuff($bar, $baz) }
method foo (...) { say ':-P'; }
Three final miscellaneous improvements: the first two from Damian, the final one from schwern.
This has always worked:
method foo (Str|ArrayRef $bar) { handle_multi_types($bar); }
method foo (Str|ArrayRef|HashRef $bar) { handle_multi_types($bar); }
Likewise, this always worked:
method foo (ArrayRef[Str] $bar) { process($bar); }
method foo (ArrayRef[ArrayRef[Str]] $bar) { process($bar); }
Finally, this didn’t used to work:
method foo ($bar, $baz, ) { do_stuff($bar, $baz); }
I haven’t seen any reports for this new version yet, but the last dev version that I saw reports for had a failure rate under 10%. Most of those failures I was able to identify and fix (just little buglets in the tests themselves). But there’s still a few reports of failures on Windows (about a third of the total Windows reports, last time I checked) that I can’t quite pin down. They don’t seem consistent based on Perl version, Windows version, or anything else I could find. If anyone tries out MS on Windows and sees a failure that looks like this, please open up a GitHub issue and let me know.
Thanks to Perl Weekly (which is absolutely great for discovering such things; you really should toddle off and sign up if you haven’t already), I see that MS has some new competition: Function::Parameters, by Mauke. It looks like FP has roughly the same syntax as MS; in fact, it looks like Mauke has used some of MS’s test suite in FP, which will help him guarantee that he maintains compatibility (a pretty good thought on his part). Of course, MS does a few things that FP doesn’t do (the biggest one being types, but also the new features above and a few other bits and bobs, such as explicit optional/required, alias parameters, readonly parameters, etc). But, then, FP does a few things that MS doesn’t do:
Of course, the big difference is that MS uses Devel::Declare, whereas FP uses the pluggable keywords API introduced in 5.12. I think that’s pretty much an invisible, under-the-hood difference, but perhaps it will matter to some. In any event, competition is always good, so I welcome the new face on the function signatures block.
What I’ll be working on next is a small-ish patch from Damian which will allow you to have named alias parameters (right now, you can have named parameters, and you can have alias parameters, but you can’t have named alias parameters). There are also a few minor GitHub issues I want to take a look at. Nothing major.
I’m also very interested in hearing from folks that are using MS. Have a problem? or a suggestion? Don’t hesitate to open a GitHub issue, even if it’s just to chat about something. I love hearing from people who are actually using this in the real world.
So install the latest version, give it a whirl, and let us know how it goes for you. Happy signature parsing!
]]>The Backstory: So, a couple months back, I look in my email and spy a message from Damian Conway. Now, some of you fine readers probably know Damian personally, and I’m sure I’ll get several comments pointing out how he’s just an ordinary bloke and all, but to you I say: hush! You may be all jaded and world-weary, but I’m still just a regular Perl schmoe, and when I see an email from The Damian addressed personally to me, I get all goosebumpy. So hush up and allow me my fanboy gushes.
Especially since the purpose of his email was to send me a patch for Method::Signatures (for which I’m a co-maintainer) and to tell me:
I adore Method::Signatures and am now using it in nearly all my own development...and in absolutely all my teaching.Ooohh ... high praise indeed, from a Perl luminary. Coolio, man!
Anyways, this missive just happened to have the ill fortune to land at the same time that $work began to (uncharacteristically) demand a larger chunk of my life than normal. So I didn’t have time to review the patch too closely, but I certainly didn’t want to lose the opportunity to discuss a few larger design issues with Mr. Conway. Thus began a long, winding discussion among me, Damian, and Schwern (who must be given full credit as the original author of MS), both in email and in GitHub issues, and even spilling over into my blog. And I learned something about Damian: he thinks in code. So, every time he comes up with a new idea, he sends me a new patch. By the time it’s all over, and $work is settled enough that I can really start devoting some time to this issue, I’ve got six, fairly hefty patchfiles from Damian—some of them overwriting or cancelling out the direction of others, the first four of them patched against current HEAD (of master) on GitHub, and the last two as both patches against current HEAD and against the previous patch.
What to do, what to do?
A Brief Tangent on Contributions: At some point, Schwern noted that we’d finally come to a general agreement on basic design, and that he’d made Damian a contributor, so Damian should just go ahead and make a branch, and put his code on it. To which Damian replied:
Re branching repo: Probably won’t. Expect annoyingly complete patches. (I know. Not properly social. Not a team player. Not git-ish. But also not a new phenomenon. Every single report card I ever received included some variation of: “Does not play well with others”.)Now, at the risk of putting words in The Damian’s mouth, I read this as “hey, I’ve been delivering patches to CPAN authors for years, and I’m not going to change now just because of the latest newfangled method.” Which leads me to ponder a bit on the relationship between author and contributor.
Like so many things in life, this involves a balance. The contributor has a feature they want added. The author wants people to fix their code for them (for free, even). This is a mutually beneficial arrangement, but there’s still all sorts of things to work out. Perhaps the author uses Dist::Zilla and the contributor doesn’t, or perhaps the author adheres to strict TDD and the contributor has sent no tests for the new feature. Or, maybe, the contributor has sent a patch when the author would really prefer a GitHub pull request.
Now the two can have some negotiation. In the TDD example, perhaps the author wants the new feature badly enough that they’re willing to write the tests themselves. Or perhaps the contributor wants the new feature badly enough that they’re willing to go back and add the tests upon request. But, the point is, everybody has to evaluate how bad they want it.
So ... how bad do I want a pack of new features and refactorings from The Damian? Yeah, you bet your ass.
The Meat of the Matter: So here’s what I suggested: I’ll make a branch, apply the patches to it, one at a time, review the code (not that I feel Damian’s code needs a lot of review, but, as he said: “Long and bitter experience indicates I’m every bit as fallible as a real human. Better check that code. ;-)”), make sure it all fits together, and then offer up the results for review.
This sounded simple enough at the time.
Of course, the tricky part is that, for the first four patches at least, they’re all patches against HEAD. That means that, while the first patch is no problem, the next three are going to be annoying, because the source I’m patching against isn’t the source originally used to generate the patchfile. Happily, I foresee this, and start to formulate a plan to work around it. Here’s how it went, and hopefully this is helpful to someone else out there.
First, I make a new branch off head, which I creatively name “damian_features”. Next, I apply patch #1 to my new branch. Since it was an exact copy of master’s HEAD, it applied cleanly ... once I remembered how to work patch. Been a while since I had any cause to apply a patchfile. Looking at the structure of the file helped me jog my memory: each file consists of a set of context diffs from several files. Each file is identified with a relative path. Within each file’s section are one or more “hunks”; that is, the individual context diffs themselves. The patch command (which, you may recall, was Larry Wall’s original claim to fame, before he invented Perl) simply finds the files indicated in the patchfile, then applies each “hunk” of context diffs, one at a time, to said files. If the source of the file being patched matches the source of the file the patchfile was generated from, the hunk “applies cleanly” and all is well. If not ... then you’ve got a problem to solve.
Since patchfiles are nearly always generated a directory above the source directory, but nearly always applied in the source directory, the filenames in the patchfile generally have an extra directory component in them that you need to strip off. Happily, patch can do that for you. In fact, it can strip off an arbitrary number of directories for you, but we’ll just tell it to strip one. The only other mildly tricky bit is that patchfiles are read from stdin, not as args:
patch -p1 </tmp/Method-Signatures-Conway.patch
git diff to review the changes—making notes of anything I want to change or ask about, but not actually changing anything yet—then I stage and commit. For the commit message, I make up a brief summary line, but I pull the guts of the message from Damian’s description of the patch in his email. Then I make any little tweaks I want to, as separate commits (that way Damian’s changes and my changes aren’t comingled). I could hold off on this until the end, but doing it in between each commit preserves the history better, and plus the next patch isn’t going to work straight up anyway, so there’s no advantage in waiting.
Okay, the next few patches aren’t going to apply cleanly, so let’s see how bad I want them. Looks like patch #3 pretty much wipes out everything patch #2 did (hey, when you’re thinking out loud in code, you’re going to go down a few blind alleys here and there), but, other than that, all the patches are pretty useful, at least in terms of showing a gradual progression to how we got to the end product. So I decide to toss patch #2 but keep the rest.
So, how do I proceed? Simple:
master branch.damian_features branch.Voilà. Now, this sounds simple(ish) in theory, but of course in practice it isn’t perfect. First, I want to stash any files which the patchfile adds as well as the ones it modifies. To do this, I need git stash -u, but that’s only available in git 1.7.7 or later, which I don’t actually have, so I have to upgrade git. That’s okay; I probably needed to do that anyway.
Next problem I run into is what happens when patch #1 adds a file, and then patch #3 adds the same file. This blows the unstash up (because it’s trying to add a file that already exists), and I have to move the existing file out of the way, then re-unstash, then do a manual diff to see if the two “new” files are identical or not. And, of course, if the first unstash added any really new files before it blew up, I have to remove those before the re-unstash or it’ll blow up again (this takes me a few tries to get right). Overall, this is an annoyance, but luckily none of the changes between patches to any such files are any big deal.
Final problem I run across is, what happens if patch #1 adds some code in a place, and patch #3 doesn’t touch the code in that same place? Well, the code really should be removed, because patch #3 doesn’t contain it, and it’s a full replacement for patch #1. But my method is going to leave it, because an unstash is essentially a merge, so it has no idea that code shouldn’t just stay right where it is. Luckily this only happens once throughout the process (if you think about it, it makes sense that this would be rare, since the patches are all messing with the same features), and I catch it easily.
So basically it’s just rinse and repeat this whole shebang for patch #4, and then patches #5 and #6 have versions against the previous patches, so I just apply those directly.
Now, you may be asking yourself: why go through all this rigamarole with the stashing and unstashing? If the patch doesn’t apply cleanly, you have a mess to clean up. On the other hand, if the unstash doesn’t merge cleanly, you have a mess to clean up there too. What’s the difference?
Well, when the patch doesn’t apply cleanly, it often keels over right there and doesn’t move on to the next file. Plus, what it generates is a file where as many of the hunks as it could apply cleanly are applied, the hunks it couldn’t apply cleanly are stuffed into a .rej file, and the original source is saved as a .orig file. Which is fine, as far as it goes. But it means that you have to pick the rejected hunks out of the .rej, find where they were supposed to go in the file based on the context (because the line numbers likely aren’t right if there are earlier hunks which were applied), insert the rejected hunk, which is actually a diff, which means it’s got little +‘s and -‘s in front of all the lines ... it’s just a little messy, is all. But if the unstash doesn’t apply cleanly, it’s just a merge conflict, and those are simple to fix.
At the end of this whole process, I’ve got 11 commits (5 substantial ones from Damian, and 6 little ones from me), and I review them all before I push:
git log origin/damian_features..HEAD
git has the concept of the author and committer being separate, which is perfect for this application. If only I’d remembered to use the --author switch when I committed!
But, wait: this is git we’re talking about. I can fix this ... right?
Of course I can:
git rebase -i HEAD^^^^^^^^^^^
git reapplies each commit, stopping at each one that I changed to edit (actually, stopping just after committing each of those), then I just have to:
git commit --amend --author "Damian's info here"
git rebase --continue
ZZ in between to resave the commit messages (which are fine), rinse and repeat 4 more times for the other Damian commits, and, Bob’s yer uncle, it’s all fixed. One last review to make sure it all looks good, then git push to GitHub and send Damian an email with a link to review (look at the commits on Sep 26, 2012, in case there happen to be more by the time you read this), and I’m all done.
Wrap-up: I hope you found something in this long post useful, or interesting, or at least entertaining. I found it to be an interesting and challenging exercise in making two different forms of contributing work together, and I bet it’s not the last time I’ll need to do that.
Certainly, as long as Damian is willing to keep sending me patches, I’ll keep integrating them. Look at the awesomeness we’ve gotten so far!
[Notes: 1) Excerpts from Damian’s emails used by permission. 2) If you’re interested in getting to use the actual code that Damian wrote for Method::Signatures, don’t worry: I’m going to be doing a bit more testing on different Perl versions, but you should see it hit CPAN soon. 3) If you didn’t catch the reference in the post title, you could read the classic American folktale online.]
First of all, we haven’t seen too many bugs. A couple of obscure things here and there, but nothing earth-shattering. So that’s positive. I noticed a fairly early adoption by Joel Berger, and he seems to be using it in many of his code snippets that he posts. So there’s a plus as well. And I just received a feature request from the Damian, who said in his opening remarks, “I adore Method::Signatures and am now using it in nearly all my own development...and in absolutely all my teaching.” So that’s pretty nice.
If you have found MS and/or MSM to be helpful in your development, I encourage you to leave a short note in the comments, and I’d also like to ask your opinion on a small design issue. This has been bugging me for a few months now, and I brought it up with Damian, and then schwern opened a GitHub issue for it. Well, technically he reopened an issue that I had opened way back when I was first digging into MS. Back then, I didn’t really have an opinion on it. But now I do.
Here it is. Let’s start with some basic stuff. This:
method foo ()
{
}
In call to Test::foo(), was given too many arguments, it expects 0 at ./test.pl line 16.
method foo (@_)
{
}
sub is that it still shifts off $self for you. (And also it maintains some consistency, in that it might look weird if every method is a method except for that one, which is a sub. But that’s just sugar.) You can call that with whatever parameters you like; MS doesn’t bother to check them.
With me so far? Good. Now here’s the sticky part.
What does this one mean?
method foo
{
}
method foo (): that is, it expects no arguments, and blows up if you try to pass any. But that’s not consistent with other systems that have signatures. For instance, in Perl 6, no signature just means ... well, no signature. That’s also true of MooseX::Method::Signatures, which means that this creates yet another difference between MXMS and MSM, which is always slightly irksome. That is, P6 and MXMS treat method foo as equivalent to method foo (@_), except that they don’t actually recognize that particular syntax. P6 will (I believe) just shift off $self for you, and MXMS won’t even do that (that is, it sets $self, but it doesn’t shift it off, so @_ will still contain $self).
This seemingly trivial change can trip you up in surprising places. For instance, this (which you may recognize from Moose::Manual::Construction):
around BUILDARGS
{
if ( @_ == 1 && !ref $_[0] )
{
return $self->$orig( ssn => $_[0] );
}
else
{
return $self->$orig(@_);
}
}
BUILDARGS results in the “too many arguments” error. Likewise, this, which is pretty much straight from the MooseX::App::Cmd POD:
method execute
{
my $result = $self->blortex ? blortex() : blort();
recheck($result) if $self->recheck;
print $result;
}
App::Cmd underneath is passing parameters to execute which you don’t actually need. At least in the BUILDARGS example, you were actually referencing @_; in this one, adding a signature of (@_) doesn’t seem to make much sense, but it’s what you need to do to make it work with MS or MSM.
So why wouldn’t we change it? Well, one thing to consider is backwards compatibility. We’ve had it working this way for the nearly-a-year that I’ve been involved, and I’m pretty sure it was working that way before I got here as well. So changing it might surprise some people that have gotten used to it working the way it does. Now, one good thing about it is that it’s not going to add a fatal error into your existing code. Rather, it would be removing a fatal error ... and, since it was, in fact, a fatal error, you probably already removed it, so most likely this change would impact any old code at all. Still, there could be something, somewhere, that perhaps does a method call in an eval block, then checks $@ and expects to find the “too many arguments” error ... okay, it’s a long shot, but, as Judy Tenuta used to say: it could happen!
So, if you have an opinion on whether we should make this change or not, we’d love to hear it. Comment below, or join the discussion on GitHub. And thanks for using Method::Signatures!
]]>My first experience with TDD was via XP (that is, Extreme Programming, not the horrid Windows version), specifically the original book by Kent Beck, where it’s referred to as “test-first programming.” Like many of the ideas therein, test-first programming was completely insane. I knew instinctively that it could never work. So I immediately decided to try it. Why? Probably because I often find myself in the position of telling other people about insane ideas that they’re positive can’t possibly work, and hypocrisy is one of my pet peeves. So I decided to pick a project (not a paid one; one that I was working on on the side) and use test-first programming at least until I reached my first milestone. And, let me tell you: I hated it. Utterly despised it.
For about two or three weeks.
Then, all of a sudden, the lightbulb went on. Just like it had when I suffered through learning vi for the first time. Just like it did about halfway through my first project using version control (RCS, it was, back in those days). Just like it did when I finally grokked spreadsheets using Lotus 1-2-3, or (as I mention in the other blog post) OOP while learning C++. And now I use TDD a lot, and I evangelize it to my co-workers. It really has changed the way I program.
Still, I’m fond of saying I’m not a TDD zealot. TDD is good sometimes, but it’s not for every project, I say. Heck, last time I mentioned (ever so briefly) TDD in this blog, I even referred you over to chromatic’s blog, where he gives a great, balanced view on using TDD (and not using TDD). And definitely go read that, if you didn’t last time I pointed you at it: it really does say everything I might want to. You use TDD when you should, and not when you shouldn’t. And I’m still going to stand by that.
But I’m starting to rethink how how often the “shouldn’t” comes up.
See, I had an experience recently (a couple months ago, but it’s taken a while for the idea to percolate into a blog post) where I had to do a “quick” thing for work. This thing was to enable our QA department to test some stuff that, right now, is a huge PITA to test. So my code would not actually be part of our codebase; it would not “go to production” and be run by our customers. It was strictly for use by QA, and it was all fiddling with data in the database (and not even the production database), and it was just a really quick thing I was going to hack together.
And, one of the downsides of using TDD is, it takes longer. Now, why would you ever use a developement technique (and, don’t forget: TDD is not a QA technique, but rather a development technique: that’s why it’s “TDD” and not “TDQ”) that slows you down? Well, there are two big reasons why the speed hit is not as awful as it sounds.
The first is that TDD often slows you down in a good way. Like, you know how sometimes you’re just ripping along, coding like a bat out of hell, and you realize you’ve been coding in completely the wrong direction? for several hours? Yeah, TDD eliminates a lot (but not all!) of that. When you have to write the tests first, it essentially forces you to design the interface first, and not just from a theoretical perspective: you have to write working (well, non-working, at first) code that actually uses your stuff. That helps settle your thinking and avoids a lot of (but not all!) blind alleys.
Secondly, you will produce software with fewer bugs when using TDD. This is not something that can be definitively proven, but case studies so far bear it out, and, if you’ve tried it both ways, you’ll know it’s true. Does the time saved fixing bugs exceed the time lost due to slower development? Probably ... almost certainly. Does it make up for any lost opportunity costs in being slower to market? Ah, that’s much trickier ... I’m not aware of any studies that even attempt to address that thorny issue. But it’s often the yardstick I use when deciding to TDD or not to TDD: do I want it fast, or do I want it right? Neither one is the “wrong” answer—just depends on the situation.
So, this particular time, I wanted it quick, so I chose not to TDD. But I forgot something crucial. Something that I often bring up when evangelizing TDD, as it happens. See, the primary advantage of TDD is not that it reduces bugs, that it forces you to design your interface first, that it magically creates a test suite as you develop, or that it can short-circuit some of your more insane ideas before they cost you time. No, the primary advantage of TDD is actually much simpler than that.
It reduces fear.
To explain what I mean, allow me a brief tangent. As I mentioned last time, at $work we’re working on refactoring a 10-year-old codebase with nearly 2 million lines of code. In order to be able to do this, our first step was to convince the business that we needed to slow down development of new features in order to put time into the refactor. So we had to make a business case for this. After all, just because code is old doesn’t make it bad, right? After all, as Joel Spolsky once wrote:
The idea that new code is better than old is patently absurd. Old code has been used. It has been tested. Lots of bugs have been found, and they’ve been fixed. There’s nothing wrong with it. It doesn’t acquire bugs just by sitting around on your hard drive. Au contraire, baby! Is software supposed to be like an old Dodge Dart, that rusts just sitting in the garage? Is software like a teddy bear that’s kind of gross if it’s not made out of all new material?Like so much of what Spolsky writes, this makes a lot of sense ... and yet is totally wrong. There is a problem with older code: it’s cobbled together with duct tape and bubble gum, a creaky Frankenstein’s amalgam of “there’s no time to fix this” and “here’s a better way to do that.” It has 3 or 4 different ways to implement objects, 2 or 3 different ways to read and write to the database, dozens of codepaths that are never reached at all, and dozens more that seem like they’re never reached but really are, in certain unlikely corner cases. And the end result? Fear. How many times have you started to fix something in that 10-year-old ball of spaghetti, only to pull yourself up short and say, “whoa ... if I mess this up, that’s really going to cost us some money”? How many times have you (or the business) put the kibosh on a plan to clean up this or that code, because it’s just too risky? How many times have you been yelled at because you tried to improve something and ended up inadvertently causing an obscure bug that went undetected for months? And what did you take away from those experiences?
Better to just leave it. Better to not risk improving. If it ain’t broke, don’t fix it, and, even if it is broke, that may be less costly than breaking it even worse.
Your estimates (and your actual implementation time) starts to creep up. “Oh, that part of the code is really scary; that’ll take much longer to fix.” Or, “well, there’s no one left who understands that code any more; it’ll take extra time to study it.” And this turns out to be a vicious cycle that just feeds on itself: we don’t have time to clean things up, or write good documentation, because just implementing new things takes so much longer now.
And TDD fixes that. It takes the fear out of refactoring, because you know that all your code is going to work the same way after the refactor, because all the tests will pass. If you wrote the tests after the code, passing tests wouldn’t mean that. As I constantly tell people: if I write some code, then write a test, and then run the test, and the test passes, I don’t know that my code works. Rather, I know that either my code works, or my test is broken. Sort of a 50-50 shot at correctness. But, with TDD, I know the test works (’cause it failed when I ran it before writing the code), and I know the code works (’cause the test passed after writing the code). And I can refactor with impunity, ’cause the tests guarantee the code still works. And, if I followed my TDD fairly strictly, and never wrote any code without writing a failing test first, I know all the code still works. The fear just ... melts away.
Now, realistically, does TDD guarantee that your code doesn’t have any bugs 100% of the time? No, of course not. There can be interactions between your code and other code that nobody foresaw or thought to test. There can be accidental side-effects introduced by passing code. There can be misunderstanding of the specs, so that the tests merely prove that you implemented the wrong thing correctly. There’s still plenty of places to go wrong. But, the point is, your bug rate drops dramatically. And your confidence in your code’s correctness goes to like 90%. Of course, 90% ain’t perfect. But it’s still a massive improvement over the 0% you’ve got now.
And I forgot all this, or more likely I thought it just didn’t matter. But my little task turned into 3 little tasks, and those 3 little tasks were all interrelated, and then I realized that I could build a common substructure for all 3, and that common substructure was a little complicated (not a lot, but not simple either), and all of a sudden I was making 4 or 5 Moose classes with dependency graphs in them and MAN I wished I could refactor all that crap. But I was too scared to. It was already taking longer than I meant for it to, and I didn’t want to be the hold-up on the testing, and if I tried to refactor and broke something, how would I ever know? Hell, even adding new features was starting to terrify me, because I had no way of guaranteeing that I wasn’t breaking all the stuff that I’d tested and proclaimed done.
In the end, I finished it up, and delivered it, and so far no one’s found any problems with it. But I’m not looking forward to any of my coworkers going into the code for any reason. (None of you guys read this blog, right? Ummm ...) I mean, there are parts that fairly elegant, and I think it’s moderately well commented, but still ... there are plenty of parts where someone could go “WTF did you do this for?” and I’ll just have to shake my head helplessly. ‘Cause it was too scary to try to change it, is the real answer, although I probably won’t say that. Makes you sound kinda sissy when you put it like that.
I think, next time, I’ll consider even more carefully whether to dispense with TDD or not. ‘Cause it turns out that even those “quick” projects can benefit from a lack of fear.
]]>
Let’s go in reverse order: first, the switch from MXMS to MSM. This is fairly standard stuff, and for the most part is detailed in MSM’s POD.
method foo (Str $class: ) to method foo ($class: ). Method::Signatures doesn’t believe in type checking invocants, and thus neither does MSM.@args to @_. MS / MSM allows that latter signature to mean “leave my arguments alone, just shift off $self for me.” I could have left them as @args and they would have worked fine, but I figured, why have the unnecessary array copy?
Next, the Moose upgrade. I can’t find anything in the logs that indicate that I did a single thing for this. All our existing Moose code Just Worked. Of course, Moose is a relatively recent thing for us, and it still doesn’t constitute a major percentage of the total codebase. So perhaps we just aren’t doing anything Moosively interesting enough. I seem to (vaguely) recall having to upgrade some MooseX modules to match the new Moose versions, and maybe having to wait on a few of those to catch up, but that’s to be expected.
Finally, the actual Perl version upgrade. I didn’t really expect to have anything to do for this, but it turns out there were a few things here and there. Remember, our codebase goes back to 2001 ... at least, that’s the earliest commit I see in the VC logs (currently Git, converted from Subversion, converted from CVS). So there were a few long-standing bugs that the good Perl folks have decided to expunge, and I can’t really blame them for that. But I’ll call them out here, in case they should prove to be pitfalls to you too, if you’re ever porting anything from the general vicinity of 5.8 to the neighborhood of 5.14.
Switch structures. We had some code using use Switch 'Perl6'; I had to change all those lines to use feature 'switch'; I thought that would be all it took, but apparently the Switch module allows you to omit parends for when clauses (most likely ’cause Perl 6 allows you to omit ’em), whereas the new core switch says you need them. Still, not a huge quantity of changes, and easy to search for.
qw in foreach loops. Here’s an interesting one. Apparently you used to be able to do this:
foreach my $id qw(timestamp person_id property_id unit_id) {
foreach list, and that’s not allowed. Apparently someone over in Perl development land agreed with me, because you’re not allowed to do it any more. So I just changed all those instances to something like:
foreach my $id (qw< timestamp person_id property_id unit_id >) {
qw delimiter to something other than parends wasn’t strictly necessary, but a) I think it makes it a bit clearer as to what’s going on here, and b) I just don’t like using parends for qw. We use parends all the time, for loads and loads of things already. I want my qw constructs to stand out a little more. My choice of angle brackets is mainly because that’s what Perl 6 will use (without the actual “qw,” even), so I feel like I’m somehow getting ready for that change. Probably I’m fooling myself. But, hey, I had to pick something, and that was as good a choice as any.
This was a big pain in the ass to search for, as it turns out. It was okay as long as the foreach and the qw were all on the same line, but not so much when they weren’t. I missed a few the first time around (and the second time around, to be honest), but luckily we have a very diligent QA team, and they had my back.
Named subs in Mason templates. Okay, first of all, let me categorically state that I don’t care for Mason. I’m sorry if it’s your favorite templating system or whatnot, but after working for 5 years in 2 million lines of code that use it, I just find that it not only allows tainting your presentation layer with back-end-specific code, it practically encourages it. Yes, yes, blah-di-blah, you-can-write-bad-code-in-any-language/toolset, but I’d rather my language/toolset made it hard for me to write bad code, not reward me with candy and kittens every time I do.
So, this problem, which we only had one instance of, involved having a named sub in the Mason template. Apparently, this produces one of those weird closure warnings: “variable will not stay shared,” I think it was. This has to do with how Mason eval’s the template and all that happy horsepucky. Anyway, solution was just to change the named sub to an anonymous sub and assign it to a variable, and then call the sub via said variable. Simple enough, once I understood the problem.
I’m not sure if this is a new warning (or, rather, a new instance of an old warning), or if perhaps the newer version of Mason just wasn’t willing to overlook the warning where the old one was. My brain hurt so much by the time I got this far that I just wanted to fix it and move on with life. (And, by the way: the mere existence of an entire subroutine in the middle of a Mason template just goes to prove my original point about Mason, as far as I’m concerned.)
UNIVERSAL isa. So, apparently, code like use UNIVERSAL qw( isa can ); is deprecated now. Whereas it used to be the recommended way to do things. See how times change? I’m not entirely sure what’s going on here, but I’m sure has to do with esoteric Perl OO guts and I probably don’t really need to know. Mainly this involved changing a lot of code like this:
unless isa($validator, 'CODE');
unless reftype($validator) eq 'CODE';
reftype to the import list for Scalar::Util. There was also a bit of changing this:
if isa($obj, $class);
if $obj->isa($class);
unless can($handler, 'new');
unless $handler->can('new');
lazy property of attributes functions almost like a cache. In fact, often when working in Moose I find that when I want something to be cached, I just make it lazy and call it a day.
Let me illustrate what I’m talking about with some code. Now, the real process of putting this code together involved a lot of blind alleys and false starts, but I’m going to present it like I came up with the final product smoothly and without interruption. I do this for two reasons: one, it makes for less frustrating reading, and, two, it makes me look smarter. ;->
So I’m creating a little command line utility to do some moderately complex data loading. I decide to use MooseX::App::Cmd for the basic structure—this is an excellent module that I’m using more and more these days. Definitely check it out.
So, I’m going to need to some support classes. Since this utility is going to need to know things about data structures, indices, foreign keys, etc, I’ll need a class to manage the database stuff.
use Company::Moose;
class Rent::Data::DB
{
use MooseX::Types::Moose qw< :all >;
has name => (isa => Str, is => 'ro', required => 1);
has user => (isa => Str, is => 'ro');
has Dbh => (isa => 'DBI::db', is => 'ro', lazy => 1, builder => '_connect_dbh');
}
Company::Moose module is a “policy module” that gathers up all the common Moose stuff we want for our company. See the second example in the synopsis of Method::Signatures::Modifiers for what it basically looks like.Rent because that’s what $work is. Then Data::DB because is it’s a class that deals with our database, but specifically our particular data structures within that DB. (In retrospect, I probably should have tried to work “schema” in there somewhere. Ah, well.)connect, and the builder for the Dbh attribute, _connect_dbh, which utilize a bunch of company-specific connection stuff.Rent::Data::DB for the primary key columns of a table, dependencies and reverse depdencies, and so forth.
Except ... maybe my “DB” class should really be a container. Just like a real database, it should contain a bunch of tables. Okay, fine, now I need a table class:
use Company::Moose;
class Rent::Data::DB::Table
{
use MooseX::Types::Moose qw< :all >;
has name => (isa => Str, is => 'ro', required => 1);
has db => (isa => 'Rent::Data::DB', is => 'ro', required => 1, weak_ref => 1, handles => ['run_query']);
}
class Rent::Data::DB
{
use MooseX::Types::Moose qw< :all >;
has name => (isa => Str, is => 'ro', required => 1);
has user => (isa => Str, is => 'ro');
has Dbh => (isa => 'DBI::db', is => 'ro', lazy => 1, builder => '_connect_dbh');
has _tables => (isa => HashRef, is => 'ro', lazy => 1, default => sub { {} });
# sort of a pseudo-attribute
method table (Str $tablename)
{
unless (exists $self->_tables->{$tablename})
{
$self->_tables->{$tablename} = Rent::Data::DB::Table->new(name => $tablename, db => $self);
}
return $self->_tables->{$tablename};
}
}
db attribute a weak reference. Then I make a HashRef attribute of Rent::Data::DB which will hold all the table objects. It’s lazy, of course, starts off as an empty hashref (tablename => table_obj), and then I create a table method which will create table objects on demand, and return the cached results.
This is pretty cool, right? But it gets even cooler. How about a way to get primary key columns?
use Company::Moose;
class Rent::Data::DB::Table
{
use MooseX::Types::Moose qw< :all >;
has name => (isa => Str, is => 'ro', required => 1);
has db => (isa => 'Rent::Data::DB', is => 'ro', required => 1, weak_ref => 1, handles => ['run_query']);
has pk_cols => (isa => ArrayRef, is => 'ro', lazy => 1, builder => '_get_pk_cols');
method _get_pk_cols
{
my $query = q{
select cc.column_name
from all_constraints c, all_cons_columns cc
where c.table_name = ?
and c.constraint_type = 'P'
and c.owner = cc.owner
and c.constraint_name = cc.constraint_name
};
return [ map { lc } $self->run_query($query, uc $self->name) ];
}
}
class Rent::Data::DB
{
use MooseX::Types::Moose qw< :all >;
has name => (isa => Str, is => 'ro', required => 1);
has user => (isa => Str, is => 'ro');
has Dbh => (isa => 'DBI::db', is => 'ro', lazy => 1, builder => '_connect_dbh');
has _tables => (isa => HashRef, is => 'ro', lazy => 1, default => sub { {} });
# sort of a pseudo-attribute
method table (Str $tablename)
{
unless (exists $self->_tables->{$tablename})
{
$self->_tables->{$tablename} = Rent::Data::DB::Table->new(name => $tablename, db => $self);
}
return $self->_tables->{$tablename};
}
# we can handle all methods that a Rent::Data::DB::Table can handle
# args will be the same except the table name is the first arg
foreach my $method (Rent::Data::DB::Table->meta->get_method_list)
{
next if $method eq uc $method; # special methods like BUILD and DESTROY
next if $method =~ /^_/; # private methods
next if $method eq 'new' or $method eq 'meta'; # the two special methods that are lc
next if grep { $_ eq $method } qw< name db run_query >; # not special, but not required here either
__PACKAGE__->meta->add_method($method, sub
{
my $self = shift;
my $tablename = shift;
my $table = $self->table($tablename);
return $table->$method(@_);
});
}
}
pk_cols and give it an appropriate builder (revealing my secret shame: yes, I’m using Oracle (no, it’s not by choice)). Now, every time I call pk_cols on a table object, I get the appropriate columns, and because it’s an attribute (instead of a method) it’s stored (essentially cached). The bit of trickery in the DB class just allows me a little shortcut:
# instead of this:
my $cols = $db->table($table)->pk_cols;
# I can just do this:
my $cols = $db->pk_cols($table);
use Company::Moose;
class Rent::Data::DB::Table
{
use MooseX::Types::Moose qw< :all >;
has name => (isa => Str, is => 'ro', required => 1);
has db => (isa => 'Rent::Data::DB', is => 'ro', required => 1, weak_ref => 1, handles => ['run_query']);
has pk_cols => (isa => ArrayRef, is => 'ro', lazy => 1, builder => '_get_pk_cols');
has indexes => (isa => HashRef, is => 'ro', lazy => 1, builder => '_get_indexes', traits => ['Hash'],
handles =>
{
is_indexed => 'exists',
},
);
method _get_pk_cols
{
my $query = q{
select cc.column_name
from all_constraints c, all_cons_columns cc
where c.table_name = ?
and c.constraint_type = 'P'
and c.owner = cc.owner
and c.constraint_name = cc.constraint_name
};
return [ map { lc } $self->run_query($query, uc $self->name) ];
}
method _get_indexes
{
my $query = q{
select distinct aic.column_name
from all_indexes ai, all_ind_columns aic
where ai.table_name = ?
and ai.index_name = aic.index_name
and aic.column_name not like '%$'
and aic.column_position = 1
};
return { map { lc $_ => 1 } $self->run_query($query, uc $self->name) };
}
}
class Rent::Data::DB
{
use MooseX::Types::Moose qw< :all >;
has name => (isa => Str, is => 'ro', required => 1);
has user => (isa => Str, is => 'ro');
has Dbh => (isa => 'DBI::db', is => 'ro', lazy => 1, builder => '_connect_dbh');
has _tables => (isa => HashRef, is => 'ro', lazy => 1, default => sub { {} });
# sort of a pseudo-attribute
method table (Str $tablename)
{
unless (exists $self->_tables->{$tablename})
{
$self->_tables->{$tablename} = Rent::Data::DB::Table->new(name => $tablename, db => $self);
}
return $self->_tables->{$tablename};
}
# we can handle all methods that a Rent::Data::DB::Table can handle
# args will be the same except the table name is the first arg
foreach my $method (Rent::Data::DB::Table->meta->get_method_list)
{
next if $method eq uc $method; # special methods like BUILD and DESTROY
next if $method =~ /^_/; # private methods
next if $method eq 'new' or $method eq 'meta'; # the two special methods that are lc
next if grep { $_ eq $method } qw< name db run_query >; # not special, but not required here either
__PACKAGE__->meta->add_method($method, sub
{
my $self = shift;
my $tablename = shift;
my $table = $self->table($tablename);
return $table->$method(@_);
});
}
}
my $indices = $db->indexes($table);
See? Instant caching. Now, every attribute I add to the table object from here on out gives me an on-demand, insta-cached property of tables for my Data::DB. It’s pretty damn cool, if I do say so myself.
Note that the real code includes a bit more stuff: more comments, more infrastructure, and my possibly-too-clever run_query method, which includes error checking and debugging, and makes snap decisions about whether you want a scalar, an array, or an array (or arrayref) of hashrefs, based on caller context and number of columns being requested. But this should give you the flavor of what I’m talking about for this post.
Hopefully this will inspire you to utilize Moose’s lazy property for some caching of your own. All we need now are Moose properties for impatient and hubristic and we’ll be all set.
tcsh for my interactive shell. Of course, I maintained that bash was the only real choice for shell scripts.
But then this brings us to another interesting point. I, of course, am a Perl programmer. A choice between Perl and shell scripts is not like a choice between C++ programs and shell scripts. Back when I was a C++ programmer, there was no question but that some tasks should be done in bash (or, actually, I was using ksh back in those days). But Perl is quite different: it’s not only the case that Perl can do anything that bash can do (that’s true of C++ as well), but Perl can also do it just as easily. Perl is often considered a scripting language, and, while we could argue about whether that’s true or not due to the fact that Perl is compiled while bash is not, we can’t (and shouldn’t) argue that deploying a Perl program is as easy as deploying a shell script, and that’s part of what being a “scripting language” entails.
But, in the end, I still, sometimes, choose to use bash over Perl, for certain tasks. I suppose you could argue there’s a certain amount of inertia involved: I got used to doing certain types of things with shell scripts back when my only other (viable) option was C++, or maybe awk. But the comparison to awk is quite appropriate. Before I learned Perl, I used awk a lot. Nowadays ... hardly ever. I certainly would never use awk inside a shell script: it would always be Perl there. At the command line, I still occasionally type awk when I mean perl, but it happens less and less often, and more and more I find myself just giving up on awk before I even get to the end of the line. Perl just really completely replaces awk.
But not bash.
Recently I was doing a personal scripting task (it involved fiddling around with MP3s, if you’re curious). I started out doing it in bash, and then ended up ripping it apart about halfway through and starting over in Perl. I had just made a bad decision on that particular task. But, while I was cursing myself out for not just using Perl in the first place, it occurred to me that maybe I ought to try to articulate the places where bash really is (or might be) better. If I had a checklist, maybe I could more easily identify where to put my efforts in from the get-go. If I had a checklist, and I posted it here on this blog, maybe even you other Perlites would come along and tell me why I’m wrong, and maybe I’ll learn something. ;->
For the impatient, the executive precis is this: I generally write bash scripts for tasks which are essentially job control scripts. Yes, Perl can call external programs just as well as any shell script can, but there are a few things bash gives us which Perl doesn’t. This is not surprising, really: bash (as ksh before it, and the venerable sh before that) was basically invented for doing job control. What sh lacked in that department, csh filled in, and then ksh and bash backported. Perl has other foci. Personally, I’m okay with Perl not being the answer for every job.
So, let’s take a look at the (few) places where bash beats Perl:
Job Failure
If I want to run a command in bash, I simply do it, like so:
run some command
system("run some command");
bash version has a problem—command not found, not enough memory, process table full—it stops and throws an error. The Perl version just blithely keeps going. Now, these days the situation is better than it used to be, because I can do this:
use autodie qw< :all >;
system("run some command");
bash version. Except, what if I care whether the command succeeded or not? Here’s the bash version:
if ! run some command
then
some recovery command
fi
use autodie qw< :all >;
use Try::Tiny; # TryCatch is nicer, but more overhead
try
{
system("run some command");
}
catch
{
system("some recovery command");
}; # do NOT forget this semi-colon!
Commands on Exit
In Perl, if you want to run something when your program exits, no matter where it’s exiting from, you can do this:
END
{
system("run some command");
}
bash looks like this:
trap "run some command" EXIT
bash command actually does what you want. That command in the bash script gets run no matter what. Normal exit, error exit, explosion, Ctrl-C, core dump ... unless someone does a kill -9 on you, you’re pretty much guaranteed to get your command run. Not so in Perl. In fact, the perlmod man page has this to say on the topic:
(But not if it’s morphing into another program via “exec”, or being blown out of the water by a signal—you have to trap that yourself (if you can).)Lame.
Now, admittedly, the vast majority of times I use trap in this fashion, it looks like this:
trap "/bin/rm -f $tmpfile" EXIT
File::Temp, and then I wouldn’t have to worry about removing my tempfile. But, then, I don’t think File::Temp handles signals either. Overall, trap is a much easier way to deal with signals than Perl’s %SIG hash, although I have to admit that I’ve never written a trap statement that didn’t end with EXIT.
Processing Job Output Lines
If I know my output lines won’t have any spaces in them, I’m golden:
for line in $(run some command)
do
process each "$line"
done
use autodie qw< :all >;
open(PIPE, "run some command|");
while ( <PIPE> )
{
chomp;
system(qw< process each >, $_);
}
close(PIPE);
bash is if my lines might have spaces. That complicates the shell script version to where it’s not particularly better than the Perl:
OIFS="$IFS"
IFS="
"
for line in $(run some command)
do
process each "$line"
done
IFS="$OIFS"
Here Documents
Sure, Perl has “here documents.” But they’re different. In Perl, a here doc defines a string. In shell scripts, it defines STDIN. So, in bash, I could say:
mysql <<END # assume ~/.my.cnf is set up
select count(*) from some_table;
END
use autodie qw< :all >;
open(PIPE, "| mysql");
print PIPE <<END;
select count(*) from some_table;
END
close(PIPE);
File Equivalencies
I have no idea why Perl doesn’t have something like this. Here’s some bash I’ve needed on several occasions:
if [[ ! $(dirname $0) -ef $(pwd) ]]
then
echo "must run this from its home dir" >&2
exit 1
fi
Cwd module includes a realpath function, but its original implementation only worked on directories (leading to a number of subs in my Perl code named really_realpath). Finally that was fixed, making it easier. Nowadays, I’d probably use Path::Class to do this in Perl:
use Path::Class;
if (file($0)->dir->absolute->resolve ne dir()->absolute->resolve)
{
die("must run this from its home dir");
}
bash version reads a lot more cleanly.
File Timestamp Comparisons
This one doesn’t come up that often, but, still. In bash I can do:
if [[ $last_run -ot $touchfile ]]
then
do it again
touch $last_run
fi
bash version is just cleaner.
Tilde Expansion
I know, I know ... it’s just a convenience. But it’s so very ... well, convenient. In bash:
rcfile=~/.me.rc
File::HomeDir, which once-upon-a-time had the vaguely nifty $~, but they went and deprecated it. Yeah, I’m sure it was a perfectly awful idea for multiple reasons. But it was a lot more convenient than:
use File::HomeDir;
my $rcfile = File::HomeDir->my_home . "/.me.rc";
Path::Class on it, for portability (not that I’m likely to care much about having most of my personal job control scripts run on Windows or whatnot). Yet another minor place where Perl just gives me more to type without significantly increasing any functionality I might actually use.
Now don’t get me wrong: Perl still beats the crap out of bash for most applications. Reasons I might prefer Perl include (but are not limited to):
basename and dirname being the most obvious examples, but generally cut, grep, sort, and wc can all be eliminated as well).bash is rudimentary at best, and the whole $IFS thing is super-clunky.bash’s case statement leaves a lot to be desired beyond simple cases (NPI).bash suck. Hashes in bash (assuming your bash is new enough to have them at all) suck even harder.bash.So it’s not like bash is going to take over for Perl any time soon. But I still find, after all these years, that many times a simple shell script can sometimes be simpler than a simple Perl script. As I say, I welcome all attempts to convince me otherwise. But, then again, there’s nothing wrong with having a few different tools in your toolbox.
MANIFEST glitch with 1.32, which I released three days ago. (I know it’s been discussed before, but I guess I never really appreciated it: it sure would be nice if CPAN Testers could report stuff like that). Version 1.32 fixes a number of CPAN RT tickets (in fact, it pretty much closes out all the open bugs), most of which you won’t care about. If you happen to be using Windows, this may fix a number of test failures, although there are still a few left that I’m working with schwern to fix. (If you are running Windows and you happen to see some mysterious errors which boil down to the fact that “skip” isn’t the same as “SKIP”, it’s definitely safe to ignore those.)
What may be less exciting to you, but is definitely more exciting to me, is that I finally, after weeks of trying, got Debuggit 2.02 released. CPAN Testers kept me on my toes on this one, and I would guess that about half of my dev releases since 2.01 were simply fixes for stupid mistakes that CPAN Testers exposed. Thanks again to Barbie for all his efforts; if you didn’t see his recent blog post, you should check it out. (He gave a shout out to one of my blog posts, so I figure I can at least return the favor.)
Anyway, the substantive changes in Debuggit 2.02 are, first off, a major reorganization of the POD. I am, in many respects, a compulsive documenter, and Debuggit is my first CPAN module, so it was overflowing with doco ... to the point where 80% of Debuggit.pm was POD. Which is awesome if you need to know every single detail about Debuggit, but perhaps a bit overwhelming if you’re trying to check it out for the first time. Taking a cue from Moose, I’ve split the POD up into Debuggit, Debuggit::Manual, and Debuggit::Cookbook. Of course, Moose is a multi-module behemoth that constitutes a complete object system for Perl 5, whereas Debuggit is a kinda cool way to put debugging statements in your code, so it could well be argued that it’s a bit presumptuous of me to be taking cues from Moose. But at least this way you can just ignore the extra documentation if you so desire.
Secondly, although I’ve been using various versions of Debuggit for over ten years now, I had never tried to use it in a CPAN-style module before, weirdly, so I never realized that, if you did that, and if you had a POD coverage test (such as many people recommend, and Dist::Zilla can provide automatically), it reports DEBUG and debuggit as “naked” subroutines. Well, it did. Now it doesn’t.
Finally, I fixed some “redefined” warnings in some corner cases, fixed a problem with overriding DEBUG in a single module, and then fixed the resulting problem of namespace::autoclean breaking stuff when it autocleaned DEBUG. And I added a second way to verify that useing Debuggit adds no extra memory to your program (when debugging is off).
Hopefully something in all that is useful for someone somewhere.
]]>dir_exists_ok, dir_contains_ok, file_contains_like, and file_contains_unlike. Check it out.
But I thought it might also be enlightening if I went into a little more detail about the story behind this release. Now, I already talked about how I became co-maintainer for this module. At the time, I asked brian d foy if there was anything special he wanted me to do for releasing, any particular procedures he wanted followed. As I said, it’s his car—I’m just driving it for a bit—so I want to do things the way he expects them done. The only thing he really mentioned was that he typically puts out a developer release and makes sure it passes CPAN Testers before doing an official release. Of course, I replied, I do that too.
Actually, I thought everyone did that. But his statement got me thinking ... it reminds me of our vists to Chuck E. Cheese’s. I know that seems like a stretch, but hear me out. Every time we go to our local CEC, my elder son points out the prominent sign by the front door. Yes, that’s right: firearms are not allowed in the kiddie arcade/pizza parlor. What is the world coming to? But my son’s point in drawing attention to this sign is that, somewhere, at some point, someone must have tried to bring a gun into Chuck E. Cheese’s. Otherwise, they’d have no need for the sign, right?
And, likewise, the fact that brian d foy felt the need to tell me I should be running my releases through CPAN Testers before making them official means that there are still folks out there who don’t. So, if you’re one of those folks, let me tell you how to make a CPAN release:
Now, typically when you screw something up, it’s blatant. Narly every CPAN Tester will fail you, and it’s usually obvious what you did wrong. Forgot to include a prerequisite. Forgot to specify a particular version of a prerequisite. Forgot to regenerate your MANIFEST file, or your META.yaml/META.json. Et cetera.
But, every once in a while, you get a just a few weird errors that you can’t figure out. If you’re lucky, they’re all from the same (typically older) Perl version, or maybe the same operating system, and you can either try to figure out how to fix that, or at the very least just update your POD to say you don’t support that version/OS. If you’re not lucky ...
So, I wasn’t lucky. I put out 1.28_02 of Test::File (after discovering the hard way there was already a 1.28_01) and I ended up with like 15 failures. Out of like 200 or so ... I can’t figure out how to make CPAN Testers count the total reports for me (except on my author page, where it only lists totals for the most recent distros). So we’re talking somewhere in the ballpark of 7% failures here. That’s not much ... right?
Now, I wonder how other people would deal with this situation. I bet we’re all different. I bet it’s just like balancing your checkbook (assuming my readers are old enough to remember the days when you had to do that by hand) and coming up 3 cents short. Some of us will just shrug and say “fuck it” and move on with our lives. It’s 3 cents, man! Who cares? But others of us will go back and hunt down that lousy stinkin’ 3 cents like our lives depended on it. Otherwise it will just mock us mercilessly, and we won’t be able to sleep at night knowing it’s out there.
I have a little of both in me. If we really were talking about balancing checkbooks, I couldn’t let the 3 cents slide. It’s a touch of OCD I inherited from my mother. But a 7% failure rate on my software doesn’t really seem that bad ... I mean, I’m not talking about medical software, or airline autopilot programs. And I actually looked at the failures, and they all look fine ... I can’t see any reason why they would be failing, actually. Still, I looked up the email addy for one of the testers (one who provided 3 or 4 of the failures) and sent him off an email. I’m sure he’s very busy with other stuff (like putting out a new version of Perl), so I wasn’t offended when he didn’t get back to me.
In the end, I decided I would check previous releases of Test::File and see what sort of error rate brian d foy was willing to live with ... after all, it’s his car, right? So, by my rough estimate (still not being able to find the exact count anywhere—if there’s not something I’m blatantly missing, I wonder if the CPAN Testers folks would be willing to add this) the last production release had an error rate of around 5%. I’m close enough to that, I figure. So I went ahead and released version 1.30 (having discovered, not too late this time, that there was a 1.29 I didn’t know about).
And 1.30 had an error rate closer to 3%, which is pretty awesome. I figured all was well. Until I actually went to install my newly released version on my own box.
And it failed.
Now, this completely floored me. I had run the exact same tests on this exact same box so many times it wasn’t funny. I’m primarily a TDD developer, especially when it comes to code I release into the wild, so I was running those tests several times a minute at some points. Certainly I’ve run the entire test suite over and over again ... hell, my script which I use to toss the tarfile up to PAUSE won’t run if the tests fail (which is as it should be). There’s no way (I says to myself) that these tests can be failing now.
Yet they were.
I tried again. I exited out of my CPAN shell and got back in. I made sure my CPAN was the most recent version and reloaded it. Still failing. With errors that made no sense: the file called not_readable (and I’ll give you 1 guess as to why it’s named that) was apparently not failing the readable test. How could this be? Was the code that set the permissions to make it unreadble not working? But, if that were the case, how had the code suddenly stopped working when I hadn’t changed anything?
I bet some of you have cottoned on to what’s going on here. Please, feel free to be very smug about it. I deserve it.
The problem, as it turned out, was that I was running all my tests, every time, as me. But I was installing, via the CPAN shell, as root. And, as you may know, root doesn’t have any problems reading files, even if they’re unreadable. D’oh! I had cribbed the non_readable file itself from brian d foy’s existing code, but I had failed to properly copy over his tests, all of which specifically had SKIP blocks and checked the UID (and EUID) to see if it was root running the tests. I added those and put out a version 1.31 and that’s where we are today. Thus far, my error rate is under 1%. Woohoo!
Anyway, I’m not entirely sure what the ultimate moral of the story should be ... don’t discount CPAN Testers too quickly? always check to see what versions are out there before you release a module for the first time? always try to install your module as root before releasing? know your CPAN Testers error rate? something completely different that I missed myself? Probably better if the reader decides that for themselves, I think. But I figured it was illuminating enough to share with the group, at least. Hopefully you agree.
]]>Also, big thanks to Gabor for including last week’s blog post in the Perl Weekly this past week. If you haven’t yet signed up for PW, you should do so now! It’s invaluable.
]]>Nothing, really.
I just contributed. I found two modules that I actually cared about, where the existing art was almost what I wanted, but not quite, and then instead of hacking on them in private, or creating my own extension, I reached out to their authors, and I said, “hey, how would you feel if I sent you a patch for thus and so?” and they said “sure, sounds good” and I did, and then I sent more stuff—stuff that I actually needed, not just stuff to be doing stuff ’cause it sounded cool—and eventually they both ended up saying, “say, you seem like you have a strong interest, and you’ve demonstrated some meager ability, and I’m a busy guy ... why don’t you handle the releases for a bit?” and I said “sure, why not?” In this latter case, the entire conversation happened to be held in public, if you seriously cared that much to go read it.
But the point I’m trying to make is, this wasn’t brain science. Anyone can do it. You could end up becoming a co-maintainer, or end up taking over an orphaned module like I also recently did. And I think you should. Why do I think that?
You know, just the other day a friend of mine (also co-worker, also currently boss) was remarking to me that he couldn’t find a decent CPAN module for XML validation. (Feel free to point me at one he may have missed in the comments.) He wasn’t necessarily upset by this in the specific case—we’ve all been in the position of not finding something you need on CPAN. He was commenting more on the big picture presented by this. Once upon a time, he opined (not necessarily as flowerily as I am waxing here, but you catch my drift), once upon a time, you could find anything you needed on CPAN. You’d have to be looking for some pretty obscure shit to come up blanks on a reasonably intelligent CPAN search. It’s not so much the case any more. Now there are gaps ... perhaps there aren’t any serious gaps yet, but the cracks are showing. We have to face the fact that we need to break out some putty and start filling in those cracks.
And, don’t forget: CPAN is Perl’s killer app. It’s been said dozens of times, but I still say it bears repeating. Python may be cool and all (personally, I swore off whitespace-dependent languages after Fortran, but whatever floats your boat), but there ain’t no CPAN for Python. Ruby may be elegant and whatnot (with or without the rails), but there ain’t no CPAN for Ruby. Javascript may be ... okay, look, Javascript is just insane. Who the hell thought that was a good idea? Is there anyone out there using Javascript for any reason other than because every browser on the planet implements it? Objects but no classes? It’s just weird. Oh, and there ain’t no CPAN for Javascript.
Now, there are movements looking to start changing that, of course. But those other languages have a bit of catch-up to play. We’re out in front, but you know what they say in the marketing biz: you don’t keep a number one spot by restin’ on your laurels. We all have to put some effort into it. What can we do? Simple: contribute. Step up, help out, take over, keep up. There is a small number of prolific CPAN authors. There is a large number of regular-Joe Perl programmers like you and me. Let’s rebalance that tree.
What is holding us back? Well, I talked about this a bit on my other blog, upon the occasion of releasing my first CPAN module. Inertia, procrastination, apprehension, perfectionism. Blah blah blah. Screw all that. Just release something. It doesn’t matter if it’s any good or not. There are these constant rants by people about how CPAN is full of crap (there was one recently on module-authors, which I refuse to link to because of its inanity), but screw those people: a lot of that crap is composed of first attempts by people who later went on to do great things. You know the coolest thing about crap? It makes awesome fertilizer.
So put something out there. Uploading a new module is good, and I encourage people to try it, but I’ve got to tell you: it’s a huge pain in the ass figuring out how to get started on that. Not because of a lack of tutorials on it: rather because of a plethora of tutorials, all of which say something different. I may relate my experiences on that, because I think I may have tried them all at this point, but I want to make sure I have something useful to say on the topic instead of becoming just one more tutorial with a different take on things. As Voltaire (supposedly) said, “It is far better to be silent than merely to increase the quantity of bad books.”
But you have another option besides creating and uploading your own module: you can submit a patch, you can add a feature, you can fix a bug. You live in the era of git, and GitHub, and that makes it easier to do than ever. (There are lots of articles on why that is, so I won’t repeat that here.) MetaCPAN has come along, and now you can often just hover over the “Clone repository” link on the left side and get a direct link to the repository where the code lives, be it on GitHub or elsewhere. Assuming it is on GitHub, you can (usually) just hop over there and open an issue. (A plea to GitHub authors: please make sure your “Issues” tab is turned on—just click that “Admin” button if it’s not. Finding the code on GitHub and not being able to submit an issue on it really derails the whole process.) Use that issue to talk about what you want to fix or change—note that I did not say to talk about what needs fixing or changing! Always approach it from the angle that this is work that you are willing to do. (And, if you’re not willing to do it, go back to the top and read this blog again, because it obviously didn’t take the first time.) Tell the author that you’ve put in the effort to see where the problem is, or where your new feature could be added. Show them that you’ve taken the time to read their documentation (yes, all of it, now matter how long-winded they are) and you’ve grasped what they want to achieve with this module, and you know that what you want to do isn’t contrary to that. Demonstrate that you’ve put in the time and effort to search elsewhere for solutions and you’ve still come to the conclusion that this is the right place for it.
Then wait for a response. You want to make sure the person agrees with the fix or feature and is willing to include it in their module. If they aren’t, then maybe you want to look at working on a subclass or plugin or extension to their module instead. But, you know what? that’s the exception, not the rule. Generally, they’re happy to have someone else taking an interest in their baby.
That’s what your module is, you know. It’s your baby. You’ll always be proud of it, even when you know it could have turned out better. You’ll always be a little touchy about other people messing with it. brian d foy compared giving me co-maintenance to giving me the keys to his car. And, you know, when you drive another person’s car, you better show some respect for it. You better make sure you understand what it needs, and what its quirks are, and where all the controls are (’cause sometimes the windshield wipers are in a weird place or something), and whether you need to goose the accelerator when you’re stopped at a red light, and whether that yellow light on the dashboard comes on all the time and it’s no big deal, or whether it means pull over RIGHT NOW! In other words, when I start working on someone else’s module, I adhere to their coding standards (even if I think they’re insane), and I make my POD style match theirs (even if I think it’s bizarre), and I organize it and comment it the way they would (even if I don’t understand it). To break out another analogy, you want to leave the campsite as clean as you found it, at the minimum. Preferably cleaner.
And you put in that effort, and you become a contributor. Maybe the author will ask you to be a co-maintainer. Or maybe not, but that doesn’t matter. The point isn’t to become co-maintainer so you can get on your blog and go “ooh! ooh! I’m a co-maintainer now!” That’s not what this blog is about. Really. The point is that you’re doing something to make CPAN a better place, and to make someone else’s life easier. They’ll tell you how you can best do that. Typically, they ain’t shy, so don’t worry about that. As long as they’re happy with your contributions, you’re doing something positive for CPAN, for us other regular-Joe Perl hackers, for everyone. And that’s what it’s all about.
]]>;->Data::Random.
I first came across this module when I was trying to find some way to test some date routines I was doing for $work. Funnily enough (or not, depending on your perspective), the date routines were themselves for testing—I’m working on introducing TDD to my department, and so I try to maintain a decent library to make testing easier. To that end, I decided to make it easier to set up records with certain dates:
# go far back in time to avoid conflicting with real rates
my $initial_date = '1/1/1990';
$prop->create_billing_period(
start_date => days_from($initial_date, 30),
end_date => days_from($initial_date, 60)
);
days_from method very thoroughly.
So I thought, why not come up with a function to create random dates? And then I thought, I’m sure someone’s already done that. So I looked on CPAN. And I found Data::Random. So I used it, and everyone lived happily ever after.
No, wait, that’s not right ...
Well, we all were living happily ever after for a while. Then, one day, I’m running some tests and I start getting these weird Moose errors in a new module one of my colleagues has written. I grabbed said colleague and we started debugging. My code called something which had a use Test::Rent::Config in it. Test::Rent::Config is a test object designed to replace a Rent::Config, naturally enough, so it subclasses that class and overrides a few methods here and there. But the code kept complaining about a method being missing. There it was, right there in Rent::Config, but the code couldn’t find it. What the heck?? Well, after nearly two hours of staring at the code, and digging into internal Moose code (’cause it had to be something tricky going on, and we all know there’s few things trickier than Moose guts), and putting debugging statements all over, and fiddling with the debugger (always dicey when trying to catch compile-time errors), we finally figured it out: Data::Random contained this line:
use lib qw(..);
lib/Test/t. Which means that when Test::Rent::Config tried to load Rent::Config, it actually loaded ... itself.
Crap.
Obviously the fix was simple: remove that line. What was it doing in there anyway? Probably something that had been put in during testing, and just never taken out. No one had noticed it in the past 8 years, apparently, because it takes a very special set of circumstances to trigger this bug. Lucky me.
So I put in a hack fix (something along the lines of shift @INC if $INC[0] eq '..') and decided to open an RT bug. Didn’t expect it to do me a lot of good, of course, since a) there had been no CPAN release in 8 years, b) the author only had one other module and it hadn’t seen a new release in 10 years, c) and there were 5 open RT bugs, ranging from 4 to 8 years old, none of which had any responses from the author.
Not that I’m casting any aspersions on the author, of course. People move on; their lives get complicated. Sometimes shit (and real life) happens. In fact, I thought I’d poke around Google and see if I could see what this particular author was up to these days, just for fun. (And also because I figured an 8-year-old email address didn’t have much chance of working.) And I found him: he’s currently the CEO of a successful company in San Francisco. Well, heck: no wonder he didn’t have time to fiddle with CPAN distributions any more. He’s busy building a business empire and whatnot.
So I sent him an email. I used 3 addresses: first, the one in the module’s POD, even though I felt pretty sure it was outdated; and secondly, two wild guesses based on the fact that I knew the author’s full name, and I now knew the domain name for his new company. My email said, hey, I see you haven’t been around CPAN for a while; perhaps you’re interested in passing on maintainership of this module? Lo and behold, he responded. Sure, he said, take it over; I don’t have much time for Perl any more. By the way ... how do I even do that?
Well, heck, I dunno ... but I figured I ought to figure it out for him. He was a busy CEO, after all; I was the one who wanted my bug fixed. So I got to poking around PAUSE. Didn’t take long to find it: “Change Permissions,” it’s called. So I explained to him how to do it (and prayed he could remember his PAUSE password), and, next thing you know, I was the primary maintainer.
Great, now to fix the bug. First thing I need, of course, is source code. I could ask the author if he had a repository I could start from, but it certainly wouldn’t be in Git (on account of, you know, Git didn’t exist 8 years ago), so best case scenario is I’d have to import it from another VCS. Alternatively, I could take advantage of the awesomeness that is gitPAN. So I started a discussion about it, and eventually just forked the gitPAN repo and called it a day.
Now I was ready to fix the bug. First, I created a failing test, then I simply removed the offending line. Simple enough. I made a few packaging changes for easier releasing and was ready to start.
First, a developer release. I don’t know much about how to release stuff on CPAN, but one thing I do know: first thing you do is make a developer release (i.e. a release number with an underscore in it), then you see how many FAIL reports you get back from CPAN Testers. This also helps make sure you’ve got all your packaging stuff just right and don’t end up with a half-broken official CPAN release ... which is exactly what I did on my first try. My MANIFEST file was wrong and the install failed for everyone. Okay, second try.
This time I got a more subtle problem. A few of the CPAN Testers reports showed a test failing, but that same test passed on all the others. I looked for patterns in versions of Perl, operating systems, etc, but found none. What could it be? I looked more closely at the test in question. It was a test of rand_time, which was a method I personally had not been using, but, looking at the test, I could easily see what was going on. The test generated a random time between now and now, which was essentially guaranteed to return the current time. It then compared the results against the current time. But the problem with that is that there were two different places where “the current time” was being captured, and, every once in a great while, it was possible for the clock to roll over to a new second in between the two places. Thus, intermittent failures. I’d seen this sort of error before when messing around with datetimes, luckily, so I recognized it fairly easily.
Okay, fine: rewrite that test. But there was another test having intermittent failures too: it turned out that the test was making sure the that the random time was defined by testing its truth or falsehood. But what if the random time returned 0 seconds after midnight? That is, 00:00:00, also known as: 0 seconds. A.k.a., “false.” The test was only generating a very small number of random samples to test, so it was very rare that it just so happened to get 0 back as an answer.
Obviously the whole test file needed to be rewritten. I decided to generate 5 times as many random numbers as there were possible values, figuring that that would practically guarantee me generating all possible values. I left the undefined/false bug to make sure I tripped it, then ran my new test several times. The current time bug was fixed, but I wasn’t getting that 0 seconds failure reliably enough. I kept on upping it, until I got the failure every time; I ended up at generating 10 random times for every possible value. Of course, now the problem was that the test took forever to complete. Don’t want people to have to sit and stare at the CPAN shell for 2 minutes while my tests are running, right? But isn’t there a way to have long-running tests only run sometimes? Well, sure there is! Just check $AUTOMATED_TESTING. Rock on. So now my test does 10x as many tests as possible values when $AUTOMATED_TESTING is on, but only half as many as possible when it’s off. Cool.
My third try had another packaging failure. When I rewrote the test file, I naturally updated it to use the done_testing feature of modern Test::More versions, but I forgot to change the Test::More version number (to 0.88) under test_requires in the Makefile.PL. D’oh!
My fourth try had a number of insane failures from CPAN Testers. You can read the full discussion if you really care about the details, but let me summarize it for you: There are 86,400 seconds in a day. If you run a test file that generates 10x as many random time values as there possible values, that’s 864,000 tests (and that’s only the first range I was testing). If you have a test file, using Test::More, that actually generates around a million tests, you will run out of memory. Bummer.
So I rewrote the test file such that every range of tests was one big test. And, finally, on dev release number 5, it worked. Thank <insert deity of choice>.
Then I changed nothing but the version number, to something without an underscore, and I was done. Except I still need to go back and remove all those extranneous dev versions from PAUSE so they’re not eating up disk space for no reason.
Anyhow, that’s the story of how I took over Data::Random, and I hope there’s something instructive in there for folks. Taking over a module can be a bit of a chore, but I think it’s worthwhile in the end. Perhaps if we all did that, we could clean up some of the cruft that’s out there and make our CPAN a better place to visit.