A fast pragmatic test runner
After breaking the build twice in the space of a week for more or less the same reason each time, I figured that my team needed a continuous integration rig, and we needed it now.
For web application stuff it's not that unusual to have tests that collide with each other, at least in the short term due to the conflict between the need to "do it right' and the need to "do it now". And generally it's desirable to run your test suite in parallel wherever possible. At the moment, our tests are generally inadequate, so I'm only saving around 2 minutes on a parallel versus series run. However I'm doing bulk changes (with the help of PPI) to this largeish existing codebase at the moment, and get into situations where I want to run the whole test suite every 10 minutes or so, at which point saving 12 minutes in an hour becomes significant.
Now that we have a convenient testing rig, I expect the number of tests to increase exponentially, so sorting out parallel versus series early in the process means we have one less problem the next time some test infrastructure issue arises.
So I came up with this short rig based on App::ForkProve so that parallel and series test results are emitted in the same TAP output. To mark a test as only suitable for running in series, the text SERIES_TEST should be present in the file. Or you can pick some other distinguishing string.
#!/usr/bin/env perl;
use strictures 1;
use IPC::System::Simple qw/capture/;
use FindBin qw/$Bin/;
# use lib::require::all qw{modules lib $Bin/t/lib}; # would be nice but not working right now.
use App::ForkProve;
=head2 run_tests.pl
Runs all parallellisable tests 9 at a time, then runs the serial
tests separately, sending all to the same TAP report.
# TODO consider if adding an option for -v makes sense at all
=cut
use File::chdir;
$CWD = $Bin;
my @parallel = capture ("ack -L --perl --ignore-dir t/data --ignore-dir t/lib SERIES_TEST t");
my @series = capture ("ack -l --perl --ignore-dir t/lib --ignore-dir t/data SERIES_TEST t");
chomp $_ for @series;
chomp $_ for @parallel;
App::ForkProve->run(qw/-j 9/, @parallel);
App::ForkProve->run(qw/-j 1/, @series);
Australian perl hacker. Lead author of the Definitive Guide to Catalyst. Dabbles in javascript, social science and statistical analysis. Seems to have been sucked into the world of cloud and devops.
No it doesn't. I thought it merged concatenated the TAP output, but it doesn't. Seems that something to do with TAP::Parser ought to sort it, but that will have to wait (unless someone else knows).
I wonder if this could be achieved with a lockfile of some kind? Test files that work in parallel try to open a shared lock on a particular file (say
t/lock), and wait until they obtain the lock. Test files that must be run in series do the same, but ask for an exclusive lock.The exact logic for handling the lock file could be stuffed into a module, say, Test::Lock, and the test scripts themselves would just contain:
It is not clear for me if "series" tests might conflict with "parallel" tests, or just that "series" tests conflict with each other?
If "parallel" tests do not interfere with any other tests then it would be best I think to join "series" tests into single .t entry point (possibly using Test::Aggregate, but it is not necessary).