confuseAcat

Having continuous integration is incredibly helpful and setting up a Jenkins server is surprisingly easy. However, configuring Jenkins to run your Perl unit tests is a wee bit harder, although it may seem easy at first. Here are a couple of issues I ran into and some things I learned:

We all know that Perl unit tests, aided by Test::More and prove produce results in TAP format. But since Jenkins is Java and someone once dictated that Java shall only read XML and only write XML, Jenkins expects test results to conform to the JUnit XML format. Well, OK, if they want XML, let's give them XML. So you turn to your favorite search engine and ask it how to transform TAP to JUnit. You will, of course, find something. And since that something was actually written by Justin Mason of SpamAssassin fame, you stop searching at that point and download that conversion script.

Well, the first mistake was to use Google. It's always better to search CPAN first in a case like that. A quick look at those CPAN search results goes a long way. It turns out that you don't have to transform TAP to JUnit, you can simply tell prove to use a different formatter for its output:

prove --formatter=TAP::Formatter::JUnit -l t > test_results.xml

It's really that simple. Now go ahead and look at the documentation for prove. It's full of goodness! For example, you might want to add --timer to the line above, and you will not only see which tests failed or didn't, but also how long each test took.

prove --timer --formatter=TAP::Formatter::JUnit -l t > test_results.xml

There is really no need to save temporary TAP output and then transform it into JUnit later.

Moving along, you will find that some people out there are actually trying to tell you that you should run a find to gather your test files and then loop over the results to pass them to prove. Of course that's possible, but it's also quite clumsy, completely unnecessary and it will run your tests in no particular order. So don't do this:

for t in $(find t -name '*.t') ; do prove ....; done

prove will find you test files if you tell it to, simply use -r for that.

prove -r --timer --formatter=TAP::Formatter::JUnit -l t > test_results.xml

Still relying on Google, you might come up with some clever Unix sexyness:

prove -r --timer --formatter=TAP::Formatter::JUnit -l t | tee test_results.xml

I guess people use the tee because that lets them see the test output when looking at the console output of their build jobs, but tee has a nasty side effect: It will hide prove's exit value from Jenkins. Your builds will not fail when you tee their output. At worst, they will be marked as unstable. So if you want your build to fail when some of your tests fail, stay away from tee! Use it for the initial configuration of your jobs to make sure you don't miss any error messages in the console output, but then get rid of it.

Perl is elegant. Or can be. For me, what I learned about prove when working with Jenkins, confirmed this once again.

This is what I started with:

for t in $(find t -name '*.t') ; do prove -l $t 2>&1 | tee -a $OUTPUT/test-$BUILD_NUMBER.tap; done
tap-to-junit-xml --input=$OUTPUT/test-$BUILD_NUMBER.tap --output=test_results.xml

And this is what I ended up with:

prove -r --timer --formatter=TAP::Formatter::JUnit -l t > test_results.xml

Not only is this easier to read, it also does what I want.