Smoke testing on Windows

As a Perl programmer on Windows, I've often had moments of supreme frustration. I can't use SSH or SFTP from Perl, and I can't use some of the mail handling modules. When I got started back in the early years of the century, I couldn't even use CPAN; I had to compile my own Perl and accept the fact that tests just mostly had no chance of working. Over the past year or two, however, especially as I've come to rely on CPANtesters for my own modules, I've started to realize that one reason for this is the lack of dense testing of modules on Windows. (Yeah, Paul Evans had something to do with that realization, too.)

But then Gábor Szabó posted on Google+ that he hoped to make it to #20 on the Windows smoke tester leaderboard. (As of today, he's at #19 - congrats!)

Wait. There's a leaderboard?

OK, so I'm a bad person, but that was the final straw.

As it so happens, I have a couple of antediluvian laptops lying around. The whole family upgraded this last year and our last mass laptop purchase was in 2004, so let me be the first to confirm that smoke testing doesn't require the latest in hardware. Two laptops running XP and nothing else but the smoke tester can crank through a lot of smoke in less time than you think - I started testing five days ago, and I'm #6 on the leaderboard for Windows for the month.

Sadly, that says more about the number of people smoke testing on Windows than about how cool I am.

And the pathetic thing is this: it ain't hard. Not even close to hard. The CPANtester Wiki Quick Start instructions work perfectly. Just do that, and seriously, you're good to go.

In my case, I also set up TightVNC on both of the testing laptops so I could just stick them downstairs in the nice cool garage and check in on them periodically from my desk where it's warm.

There are still some adjustments. CPAN::Reporter::Smoker is very, very close to being unattended-capable, but it's not 100%. I'm seeing test processes hang every now and then (POE seems to be a problem for some reason, but I haven't been organized enough to keep records yet - hey, I'm still in my first week!), and the only recourse is to quit the smoker and start again.

By default, CPAN::Reporter::Smoker loads every prerequisite anew for every test. For frequently-used prerequisites, this is a colossal waste of time. If you want to just crank through CPAN, by all means set trust_test_report_history.

But even with that, so far neither of my smoker machines has gotten further than about 170 modules into CPAN without having to be restarted. This is a serious problem. The smoker sorts by age; new modules are tested first, which normally makes sense, as newly submitted modules need testing and are more likely to be fixed if tests fail. (For that reason, CPAN::Reporter::Smoker checks for new submissions every 12 hours and runs those before continuing down the list.)

But if your goal, like mine, is to increase the test coverage for all of CPAN, then 170 modules out of over 27,000 isn't quite enough. My current plan involves trying two tweaks: first, I need to stop testing modules that hang (for that to work, I have to detect hangs better). Second, I probably need to ignore new submissions; to do that, I'll have to use MiniCPAN to make a local mirror.

To do that right, I should actually download a MiniCPAN and then delete modules once tested so I can restart a smoker and not retest everything it's already done. But I can't delete a prerequisite for another module - so before doing that, I'll have to do a prerequisite analysis of CPAN.

So that's the state of Windows smoking here at headquarters, currently in sunny Budapest. I wouldn't say no to suggestions.

10 Comments

Alright, since i've been doing smoke testing a while ago, here's my current setup:

https://github.com/wchristian/perl16_smoker

Do note the commits changing the smoking modules and CPAN and such. :)

Also, some things to note:

- this file is your friend: https://github.com/wchristian/perl16_smoker/blob/master/cpan/prefs/disabled.yml
- you can tell the smoker to run in reverse order, i.e. smoke old stuff first
- trust_test_report_history is nice, but can result in bogus fails. if PERL5LIB gets too long, depending on your Perl you'll either get an empty @INC or it'll outright crash
- stuff that hangs is usually outright borked, if you get hangs that don't reproduce, then you have a problem

Hmm, what happened to the Microsoft CPAN Authors Lab. There was an announcement on use.perl.org (link now dead), and a /. article, and a couple of other things.

But other than that, nothing. I have no idea how to get one onto of these VMs...

http://tech.slashdot.org/story/08/12/06/227246/free-resources-for-windows-perl-development
http://www.perlmonks.org/?node_id=728667

I'm #7 on the leaderboard for Win32; I have more or less the same experience.

BINGOS, who is the über-smoker on Linux and BSD platforms, uses some tooling to automatically cancel modules after they exceed a timeout; I have not tested these for Win32 but still mean to: I think it will be the key to good smoker results on Win32.

http://search.cpan.org/~bingos/POE-Component-SmokeBox-0.48/lib/POE/Component/SmokeBox/Backend.pm

CPAN::Reporter::Smoker can use Win32::Job to kill hanging processes. Install it and add "command_timeout=2000" (or some other value in seconds) to .cpanreporter/config.ini. Any installations that takes more that this amount will be killed. distroprefs can be used to skip hanging modules or any modules that you don't want to test, for ex. Acme-Mom-Yours.

CPAN::Reporter::Smoker should automatically skip distributions that were already tested, if it doesn't, it's a bug.

Testing newest modules is most useful, they have better probability of being fixed.

Forgot to mention that modules, that were killed by using command_timeout, will not be tested again unless they will be tested as a dependency for other module.

I need a more principled way of queuing my tests.

Honestly, you don't. When you use a decent disabled.yml file, and remove some of the unnecessary delays in CPAN.pm and the testers stuff, like my repo shows, you can burn through all of CPAN in a week.

I've been smoke testing on Windows since reading this blog post. I'm also getting hangs. I've used Process Explorer to see what is hanging and it's always due to Capture::Tiny. When things stop I'll see two Perl processes that look like this but with different filenames used for the last argument.

C:\strawberry\perl\bin\perl.exe -e "$SIG{HUP}=sub{exit}; if( my $fn=shift ){ open my $fh, qq{>$fn}; print {$fh} $$; close $fh;} my $buf; while (sysread(STDIN, $buf, 2048)) { syswrite(STDOUT, $buf); syswrite(STDERR, $buf)}" C:\Users\smoker\AppData\Local\Temp\2\qdJ32OEk0b

Killing one or both of the processes will let the build continue without a problem.

Both of the processes are children of the following where the tested distribution varies, of course.

C:\strawberry\perl\bin\perl.exe -MCPAN -e "$CPAN::Config->{test_report} = 1; test( 'KTAT/Teng-Plugin-SearchBySQLAbstractMore-0.10.tar.gz' )"


I installed Win32::Job and set command_timeout=1000 in my config.ini file, but it doesn't work. The processes will wait forever without being killed.

If a solution can be found to Capture::Tiny's hangs on Windows then smoketesting on Windows could be easy. But right now it requires babysitting.

My solution so far has been: Smoke on ActivePerl. I haven't tried smoking with Strawberry yet, but ActivePerl smoking has never hung in the manner you described.

Leave a comment

About Michael Roberts

user-pic I blog about things - things that help me go. Perl helps me go.