Perl testing with Jenkins/Hudson: avoiding some pitfalls

Having continuous integration is incredibly helpful and setting up a Jenkins server is surprisingly easy. However, configuring Jenkins to run your Perl unit tests is a wee bit harder, although it may seem easy at first. Here are a couple of issues I ran into and some things I learned:

We all know that Perl unit tests, aided by Test::More and prove produce results in TAP format. But since Jenkins is Java and someone once dictated that Java shall only read XML and only write XML, Jenkins expects test results to conform to the JUnit XML format. Well, OK, if they want XML, let's give them XML. So you turn to your favorite search engine and ask it how to transform TAP to JUnit. You will, of course, find something. And since that something was actually written by Justin Mason of SpamAssassin fame, you stop searching at that point and download that conversion script.

Well, the first mistake was to use Google. It's always better to search CPAN first in a case like that. A quick look at those CPAN search results goes a long way. It turns out that you don't have to transform TAP to JUnit, you can simply tell prove to use a different formatter for its output:

prove --formatter=TAP::Formatter::JUnit -l t > test_results.xml

It's really that simple. Now go ahead and look at the documentation for prove. It's full of goodness! For example, you might want to add --timer to the line above, and you will not only see which tests failed or didn't, but also how long each test took.

prove --timer --formatter=TAP::Formatter::JUnit -l t > test_results.xml

There is really no need to save temporary TAP output and then transform it into JUnit later.

Moving along, you will find that some people out there are actually trying to tell you that you should run a find to gather your test files and then loop over the results to pass them to prove. Of course that's possible, but it's also quite clumsy, completely unnecessary and it will run your tests in no particular order. So don't do this:

for t in $(find t -name '*.t') ; do prove ....; done

prove will find you test files if you tell it to, simply use -r for that.

prove -r --timer --formatter=TAP::Formatter::JUnit -l t > test_results.xml

Still relying on Google, you might come up with some clever Unix sexyness:

prove -r --timer --formatter=TAP::Formatter::JUnit -l t | tee test_results.xml

I guess people use the tee because that lets them see the test output when looking at the console output of their build jobs, but tee has a nasty side effect: It will hide prove's exit value from Jenkins. Your builds will not fail when you tee their output. At worst, they will be marked as unstable. So if you want your build to fail when some of your tests fail, stay away from tee! Use it for the initial configuration of your jobs to make sure you don't miss any error messages in the console output, but then get rid of it.

Perl is elegant. Or can be. For me, what I learned about prove when working with Jenkins, confirmed this once again.

This is what I started with:

for t in $(find t -name '*.t') ; do prove -l $t 2>&1 | tee -a $OUTPUT/test-$BUILD_NUMBER.tap; done
tap-to-junit-xml --input=$OUTPUT/test-$BUILD_NUMBER.tap --output=test_results.xml

And this is what I ended up with:

prove -r --timer --formatter=TAP::Formatter::JUnit -l t > test_results.xml

Not only is this easier to read, it also does what I want.

11 Comments

I've done similar for a while, but there's a new TAP plugin. I've only had a chance to look at it very briefly, but it looks promising.

Regarding the pipe to tee you can, at least in Bash, enable the pipefail option to resolve the issue with exit code. Here's an example:


$ set +o pipefail && $(exit 200 | tee); echo $?
0
$ set -o pipefail && $(exit 200 | tee); echo $?
200

Nice article.

We just tried the TAP plugin - unfortunately it doesn't work if you use done_testing();

Apparently it use http://www.tap4j.org/ which doesn't support it yet, I've opened a ticket

I haven't used the TAP plugin yet, but it looks like you can't use a standard tool like prove to run the tests. Instead you'd have to find all the tests yourself and run the manually with the output going to a file, something like:

   for tfile in `find t -name '*.t'`; do
       perl -Ilib $tfile > $tfile.tap 2>&1
   done

And then you'd tell the TAP plugin to look for t/*.tap. This is rather annoying. Maybe I'm mistaken?

—Theory

You can Point TAP::Harness::Archive at an empty directory to get per-script output in the form expected by the plugin.

Hi there David!

I have just created an issue for that: https://sourceforge.net/tracker/?func=detail&aid=3489038&group_id=351793&atid=1470127

I will work on this in the next days. Feel free to add any comments, suggestions.

When I wrote tap4j, I had a short deadline to deliver it for an internal project too. So probably there is room for improvements in both, tap4j and TAP Plug-in (in the latter, mainly due to the fact that I code Java everyday, but still have a loooot to learn about Perl :)

Thanks!
Bruno

Thanks for reporting the issue! I'm back working on my Open Source projects, probably until July/August.

I'm currently working on a YAMLish issue in tap4j (this was reported in GitHub), and after this one I'll focus in integration tests for Perl x tap4j.

Feel free to drop a message and check the status if I take too long to update the issue.

All the best,
Bruno

I also tried the TAP plugin for Jenkins/Hudson and couldn't get it to work.

We are doing parallel testing with -j4, so the hack of manually running each script with "perl" won't work. and I couldn't get --archive to produce any results for me. (Perhaps it's not compatible with parallel testing). We also use "done_testing()", so perhaps that it is part of the issue as well.

However, emitting JUnit works fine, I was just hoping to get TAP integration to work!

Leave a comment

About confuseAcat

user-pic Random observations that may in some way be related to Perl.