Filtering Subtests?
I wrote about speeding up Test::Class with subtests. That's worked extremely well for us, but it's raised an interesting problem: how can I filter subtests the way I can filter test methods? Explanation follows, suggestions welcome.
Imagine a test class with the following test methods:
genres_and_formats
lazy_columns
partner_episodes
search_partner_id_not_defined
partner_quits
If the setup method which runs before every test takes 15 seconds to run, then these five test methods cannot complete in under a 1m15s. Assuming they take an average of two seconds each to run, let's say that this test class takes 1m25s to run.
Now imagine that the first four test methods are "read-only". That is to say, they don't change the state of the world. You could push all of those down into a single test method with four subtests:
read_only
genres_and_formats
lazy_columns
partner_episodes
search_partner_id_not_defined
partner_quits
Your tests will now take about 0m40s to run, or a savings of over half a minute. Do this to 30 test classes and you could shave 15 minutes off of your test suite run time (and we did).
The problem is that the Test::Class trick of running individual test methods no longer works for the test methods converted to subtests. Thus, it would be nice to be able to "filter" subtests. That is to say, I'd like to only run subtests which match a specific criteria.
The only criteria I can currently think of is the name of the subtest, but even that isn't great if you want to manage your test suite with this technique (see chromatic's discussion of error messages as strings to get an idea of the issue here).
Anyone have any particularly brilliant suggestions on how to address this issue? How could one easily write "filterable" subtests and what mechanism would one use to filter them?
Freelance Perl/Testing/Agile consultant and trainer. See http://www.allaroundtheworld.fr/ for our services. If you have a problem with Perl, we will solve it for you.
And don't forget to buy my book! http://www.amazon.com/Beginning-Perl-Curtis-Poe/dp/1118013840/
I'm not entirely sure I'm understanding you right there, but it seems like you're thinking about the same problem i was thinking about yesterday, only in a reverse manner.
Mainly, i was looking for a way to apply Test::More's subtest function to Test::Class without having to write a lot of extra stuff each time.
This was the result: http://gist.github.com/527670
To provide some context, a discussion i had with kentnl about it last night: http://gist.github.com/529083 (thanks for providing the log, kent :))
Mithaldu, bug Adrian Howard. He had a fork of Test::Class which used subtests. He said it was much easier and cleaner, but he was waiting on stable subtests. He might still have it on his laptop :)
Why subtests for this? The interesting part of the speedup seems that you've identified which groups of tests can run concurrently.
If subtests are the only way to gather and report this information that's one thing, but that seems like an implementation detail that doesn't necessarily have to be so.
@chromatic: Many times people will group tests in curlies to get a block scope (for example, they want a lexical var to not leak and locally override a method). Often when I see this, there's even a comment at the beginning of the block, explaining what is being tested. In this case, however, we're naturally fall back to a subtest where the comment is turned into a name and the lexical scoping is just part of the actual test.
In short, it's a way of organising tests in a very natural style. As subtest parsing gets more robust with Test::Harness, this technique will become more popular.