Ovid

Back in September 2008, I had a list of the most popular testing modules on the CPAN. I created this list because I was writing Test::Most and I needed to know them. Tonight, after hearing that the next Perl-QA hackathon is in Vienna, I thought about what I might want to accomplish and decided to see if the most popular modules had changes. In 2008, out of 373 test modules, we had the following top 20:

 1  Test::More                          69396
 2  Test                                10912
 3  Test::Exception                     2314
 4  Test::Simple                        962
 5  Test::Base                          610
 6  Test::NoWarnings                    386
 7  Test::Builder::Tester               308
 8  Test::Deep                          254
 9  Test::Pod                           223
10  Test::Warn                          217
11  Test::Differences                   213
12  Test::MockObject                    187
13  Test::Pod::Coverage                 157
14  Test::Builder                       131
15  Test::WWW::Mechanize::Catalyst      118
16  Test::XML                           113
17  Test::Block                         112
18  Test::Perl::Critic                  110
19  Test::Distribution                  107
20  Test::SQL::Translator               101

In January 2010, out of 493(!) Test:: modules, we have the following top 20:

 1  Test::More                          86194
 2  Test                                10789
 3  Test::Exception                     3548
 4  Test::Base                          1006
 5  Test::Simple                        962
 6  Test::NoWarnings                    884
 7  Test::Deep                          413
 8  Test::Warn                          350
 9  Test::Pod                           335
10  Test::Builder::Tester               322
11  Test::Differences                   311
12  Test::Pod::Coverage                 277
13  Test::Perl::Critic                  256
14  Test::MockObject                    205
15  Test::Builder                       203
16  Test::Most                          170
17  Test::Block                         169
18  Test::WWW::Mechanize::Catalyst      162
19  Test::Distribution                  150
20  Test::Kwalitee                      143

That's when I noticed something very interesting. My Test::Most module, the one I created this list for, was already in the number 16 slot (it's currently at the number 13 slot if you ignore modules in the Test:: namespace but which don't export test functions)! That's when my failure dawned on me. Given that people who use Test::Most are more serious about testing, it's reasonable that they're more serious about testing, thus artificially inflating my stats. I was counting the number of times the modules were being used. However, if I write, say, an extra 160 test programs for my modules, each of which uses Test::Most (or indeed, just type use Test::Most 160 times in a single module), then I could push my module into the top ten. So I reran my stats per distribution:

1   Test::More                          14111
2   Test                                 1736
3   Test::Exception                       744
4   Test::Simple                          331
5   Test::Pod                             328
6   Test::Pod::Coverage                   274
7   Test::Perl::Critic                    248
8   Test::Base                            228
9   Test::NoWarnings                      155
10  Test::Distribution                    142
11  Test::Kwalitee                        138
12  Test::Deep                            128
13  Test::Warn                            127
14  Test::Differences                     102
15  Test::Spelling                        101
16  Test::MockObject                       87
17  Test::Builder::Tester                  84
18  Test::WWW::Mechanize::Catalyst         79
19  Test::UseAllModules                    63
20  Test::YAML::Meta                       61
21  Test::Synopsis                         57
22  Test::Compile                          56
23  Test::Portability::Files               54
24  Test::Most                             49

OK, so I dropped down to a more reasonable 24th place. But a quick check of CPANTS showed that I wasn't finding as many uses of Test::Most as I should because there were cases where:

• Authors declared it in their Makefile.PL or Build.PL but didn't use it
• In at least one case (probably more), the author has tests in their lib/ directory that I was not searching!

There was also the case where it turns out that Robert Krimen is a huge fan of Test::Most (Hi Robert!) and has enough modules out there to artificially inflate my stats. So here's yet another list, showing test modules per author:

1   Test::More                          2669
2   Test                                 685
3   Test::Exception                      260
4   Test::Simple                         161
5   Test::Pod                             76
6   Test::Warn                            66
7   Test::Perl::Critic                    65
8   Test::Deep                            60
9   Test::Pod::Coverage                   55
10  Test::NoWarnings                      51
11  Test::Differences                     50
12  Test::Builder::Tester                 49
13  Test::MockObject                      49
14  Test::Base                            47
15  Test::WWW::Mechanize::Catalyst        36
16  Test::Builder                         27
17  Test::Kwalitee                        26
18  Test::MockObject::Extends             24
19  Test::Distribution                    21
20  Test::Output                          19
21  Test::XML                             17
22  Test::Moose                           16
23  Test::Harness                         16
24  Test::LongString                      15
25  Test::Tester                          13
26  Test::Spelling                        12
27  Test::MockModule                      11
28  Test::UseAllModules                   10
29  Test::TCP                             10
30  Test::Most                            10

That seems a touch more reasonable, though I confess I'd like to see more folks using this module. It really does make your testing life simpler.

And as for the beautiful Test::Class which more people should be using:

• 95 test programs (25th place)
• 19 distributions (42nd place)
• 9 authors (33rd place)

Come on, folks! You can do better than that, right?