Ordering Your Tests
By default, the test actions of both ExtUtils::MakeMaker and Module::Build test t/*.t in lexicographic order (a.k.a. ASCIIbetical order). Under this default, some Perl module authors who want tests performed in a given order have resorted to numbering tests: t/01_basic.t, t/10_functional.t, and so on.
My personal preference is to take the lexicographic ordering into consideration when naming test files: t/basic.t through t/whole_thing.t. But the price of this choice is a certain number of contrived test names, and even the occasional thesaurus lookup.
But there is a better way. Both ExtUtils::MakeMaker and Module::Build allow you to specify tests explicitly.
Under ExtUtils::MakeMaker version 6.76 or above, you call WriteMakeFile() thus:
WriteMakeFile(
...
test => {
TESTS => 't/one.t t/two.t t/three.t t/four.t',
},
...
);
If you do this, the tests specified (and only the tests specified) are performed in the order specified.
ExtUtils::MakeMaker version 6.76 was released September 5 2013 and shipped with Perl 5.19.4, so any reasonably modern Perl should support this.
The equivalent incantation under Module::Build version 0.23 or above is:
Module::Build->new(
...
test_files => [ qw{
t/one.t
t/two.t
t/three.t
t/four.t
} ],
...
)->create_build_script();
Module::Build version 0.23 was released February 9 2004.
I blog about Perl.
Test::Manifest uses this feature to take the test order from a separate file (t/test_manifest)
FWIW, I think this practice is somewhat questionable. It shouldn't matter what order the test files are run in. It should be possible to run any test file in a distro on its own, without any other test file being run at all, and it should be possible to run all the test files simultaneously without any of them conflicting.
Adding logic that expects a specific order seems like a good way to end up violating one of these expectations.
Karl and I put a lot of work in this last dev cycle to make sure that all of the tests in core can be run in parallel, and the result is a massive reduction in how long it takes to test perl. For instance I typically run with 16 processes at a time.
I have also noticed recently that there are certain modules out there which are not parallel safe, which makes it very frustrating to install a large number of modules in parallel. Anything that depends on something that needs serial testing essentially needs to be installed in series as well.
So I would recommend you *don't* do this. Or if you do, that you *also* run your tests regularly in parallel by using HARNESS_OPTIONS=j8 or j16, so that you can detect any race conditions in your tests.
Eg, I would do this:
HARNESS_OPTIONS=j16 make test
And I routinely do this:
HARNESS_OPTIONS=j16 cpanm Some::Module::Name
I see your point. A test suite should succeed regardless of the order in which tests are run. Individual test files should be independent, I think of them as independent, and I will see about periodically testing in parallel.
The thing is, the purpose of testing is two-fold: to demonstrate success and to find failures. But in my experience failures are not often solitary -- more commonly they occur in cascade. When a cascade occurs, it is very handy to the maintainer to have the problem that initiated the cascade come out first, rather than buried in a welter of subsidiary failures. Unfortunately I see no way to make this happen without testing the more fundamental code first, and I see no way to test the more fundamental code first without specifying the test order.
The activity that triggered this post was my attempt to understand Chinese New Year by re-implementing the algorithms from Calendrical Calculations in Perl. It seemed logical to have a module per chapter. If something from the Gregorian chapter/module fails, not much else has a chance to succeed. Diagnosing a cascade involves running down (or up) the internal river. Why not test in river order, at least while under active development?
P.S. The
HARNESS_OPTIONSthing should work on whatever tool chain you use, because it acts directly on TAP::Harness.I do get the point that it may be helpful to have a simple test fail early in certain types of testing, and for instance you can hit CTL-C and then start debugging just that test file.
But that is also serial thinking. :-) If you run your tests in parallel, they run quickly and you don't need to think abut CTL-C, and then you can look at the failure reports from the parallel run and then drill into the one that is relevant based on the report of them all failing.
Anyway, I said "somewhat questionable" and not something stronger because I recognize it is a matter of personal preference and provided you do /something/ to avoid creating test files with inter-dependencies, or tests that are not parallel safe because of race conditions then all is good and it is really none of my business to tell you how to run your tests.
Regarding your PS, yes, HARNESS_OPTIONS affects Test::Harness so it is helpful in many situations. One of the few it does not affect is the core tests for perl itself. There you have to say TEST_JOBS=N instead, and you have to use make test_harness instead of make test. :-) HARNESS_OPTIONS was designed to be a bit more future proof and allow multiple different options at once, whereas test_harness in the core does not respect that and expects a specific single parameter env var instead.
Who said life had to be easy huh? :-)
Anyway, thanks for the discussion, I do realize there are legit reasons to order your tests, but after having been stung recently by a number of distributions that have race conditions, I thought I would pipe up with the contrary view.
Don't know if it is allowed to ask questions here. I am looking for a way to tell EU::MM to shuffle test scripts. An even more challenging setup would be to always execute load.t (bails out test suite run if a module cannot be loaded) first and shuffle only the remaining test scripts.