I wrote about speeding up Test::Class with subtests. That's worked extremely well for us, but it's raised an interesting problem: how can I filter subtests the way I can filter test methods? Explanation follows, suggestions welcome.
Imagine a test class with the following test methods:
genres_and_formats lazy_columns partner_episodes search_partner_id_not_defined partner_quits
If the setup method which runs before every test takes 15 seconds to run, then these five test methods cannot complete in under a 1m15s. Assuming they take an average of two seconds each to run, let's say that this test class takes 1m25s to run.
Now imagine that the first four test methods are "read-only". That is to say, they don't change the state of the world. You could push all of those down into a single test method with four subtests:
read_only genres_and_formats lazy_columns partner_episodes search_partner_id_not_defined partner_quits
Your tests will now take about 0m40s to run, or a savings of over half a minute. Do this to 30 test classes and you could shave 15 minutes off of your test suite run time (and we did).
The problem is that the Test::Class trick of running individual test methods no longer works for the test methods converted to subtests. Thus, it would be nice to be able to "filter" subtests. That is to say, I'd like to only run subtests which match a specific criteria.
The only criteria I can currently think of is the name of the subtest, but even that isn't great if you want to manage your test suite with this technique (see chromatic's discussion of error messages as strings to get an idea of the issue here).
Anyone have any particularly brilliant suggestions on how to address this issue? How could one easily write "filterable" subtests and what mechanism would one use to filter them?