Smoke testing on Windows

2012-12-16T16:01:10Z

As a Perl programmer on Windows, I've often had moments of supreme frustration. I can't use SSH or SFTP from Perl, and I can't use some of the mail handling modules. When I got started back in the early years of the century, I couldn't even use CPAN; I had to compile my own Perl and accept the fact that tests just mostly had no chance of working. Over the past year or two, however, especially as I've come to rely on CPANtesters for my own modules, I've started to realize that one reason for this is the lack of dense testing of modules on Windows. (Yeah, Paul Evans had something to do with that realization, too.)

But then Gábor Szabó posted on Google+ that he hoped to make it to #20 on the Windows smoke tester leaderboard. (As of today, he's at #19 - congrats!)

Wait. There's a leaderboard?

]]> OK, so I'm a bad person, but that was the final straw.

As it so happens, I have a couple of antediluvian laptops lying around. The whole family upgraded this last year and our last mass laptop purchase was in 2004, so let me be the first to confirm that smoke testing doesn't require the latest in hardware. Two laptops running XP and nothing else but the smoke tester can crank through a lot of smoke in less time than you think - I started testing five days ago, and I'm #6 on the leaderboard for Windows for the month.

Sadly, that says more about the number of people smoke testing on Windows than about how cool I am.

And the pathetic thing is this: it ain't hard. Not even close to hard. The CPANtester Wiki Quick Start instructions work perfectly. Just do that, and seriously, you're good to go.

In my case, I also set up TightVNC on both of the testing laptops so I could just stick them downstairs in the nice cool garage and check in on them periodically from my desk where it's warm.

There are still some adjustments. CPAN::Reporter::Smoker is very, very close to being unattended-capable, but it's not 100%. I'm seeing test processes hang every now and then (POE seems to be a problem for some reason, but I haven't been organized enough to keep records yet - hey, I'm still in my first week!), and the only recourse is to quit the smoker and start again.

By default, CPAN::Reporter::Smoker loads every prerequisite anew for every test. For frequently-used prerequisites, this is a colossal waste of time. If you want to just crank through CPAN, by all means set trust_test_report_history.

But even with that, so far neither of my smoker machines has gotten further than about 170 modules into CPAN without having to be restarted. This is a serious problem. The smoker sorts by age; new modules are tested first, which normally makes sense, as newly submitted modules need testing and are more likely to be fixed if tests fail. (For that reason, CPAN::Reporter::Smoker checks for new submissions every 12 hours and runs those before continuing down the list.)

But if your goal, like mine, is to increase the test coverage for all of CPAN, then 170 modules out of over 27,000 isn't quite enough. My current plan involves trying two tweaks: first, I need to stop testing modules that hang (for that to work, I have to detect hangs better). Second, I probably need to ignore new submissions; to do that, I'll have to use MiniCPAN to make a local mirror.

To do that right, I should actually download a MiniCPAN and then delete modules once tested so I can restart a smoker and not retest everything it's already done. But I can't delete a prerequisite for another module - so before doing that, I'll have to do a prerequisite analysis of CPAN.

So that's the state of Windows smoking here at headquarters, currently in sunny Budapest. I wouldn't say no to suggestions.

Michael

Smoke testing on Windows