CPAN Testers

With 20 million reports, CPAN Testers is very definitely one of the biggest online repositories of test reports for both programming languages and software applications. While other languages and applications may have larger communities than CPAN Testers, the Perl community's commitment to testing has uniquely enabled CPAN Testers to build on the benefits from many areas of the test community. The TAP protocol has now been incorporated into several other language and application testing frameworks, and stand-alone applications, such as Smolder, have been able to harness the output to present results in a way which best highlights problem areas. CPAN Testers too has been striving to improve the way we present and provide analysis of reports.

With regards to the analysis CPAN Testers provides, a notable example has been the work by Andreas König with the CPAN Testers Analysis site. By taking a selection of test reports, his tools have the ability to find common areas which help authors to pin-point unusual problems. One aspect this has recently helped with, is identifying a problem with development releases, which in themselves don't necessarily fail their tests (possibly because the don't include tests which test all scenarios), but when used within other modules may highlight faults in the prerequisite. The problem with this is that the report is associated with the calling distribution, not the prerequisite which has the fault. Andreas' tools recently were able to identify a prerequisite that was causing concern for a number of distributions, even though the fault lay elsewhere. This in turn led to a lengthy discussion on the CPAN Testers Discussion List, of how we could best approach this. The difficulty though is writing parsers that would automatically determine what was a fault. With many of the test reports it requires a human to determine this for single reports, while Andreas' tools can only highlight where further human investigation is needed, and needs many reports to determine if there might be a cause that may need investigating.

In the short term it is possible to re-associate reports with other distributions, by requesting to me, but in the longer term it would be better to enable a automated process. Which is where the Admin site comes in again. Although that's been on hold for a while, I think this is another feature which could simplify the process of correcting mis-filed reports. As such, I hope to continue on that site soon.

The discussion on the mailing list asked whether we could more appropriately filter these types of reports, but didn't lead a conclusive answer. However, it will be a discussion topic for the forthcoming QA Hackathon. At the end of the month, a number of testing hackers will be assembling in Paris to further Perl's testing infrastructure, processes and code. It will be a good opportunity to have a number of different views discussed in one room with all those likely to be actioned with writing the solutions. This QA Hackathon looks to be the biggest so far with roughly 40 people attending. There are a diverse set of projects being planned for the 3 days, so expect lots of output afterwards.

Gabor Szabo highlighted an issue running CPAN-Reporter with Net-HTTPS-6.02 last month on the mailing list. Thanks to Manoj Kumar, if you experience a "fact submission failed" error, try rolling back to Net-HTTPS-6.01. Tim Bunce also asked about testing on Windows. Although Windows is still a regularly tested platform, the volumes are considerably less than for other more Unix based OSes. David Golden put in a lot working getting CPAN-Reporter and Strawberry Perl working on Windows, so if you have the spare capacity and want to get involved with CPAN Testers, Windows is certainly an ideal platform to make a difference. And if you're running Windows, Cygwin would welcome some more regular reporting.

Lastly, for now, the problems with the SQLite Database have surfaced again. Although I can't definitely say why there is a problem, it would seem to be related to the sheer size of the data CPAN Testers stores. The uncompressed version is now over 8GB. To reduce the bandwidth, and better provide data in a way that consumers can store and use it, an API is being written to supply a record set, rather than the full data set. Expect further work on this during the QA Hackathon.

Cross-posted from the CPAN Testers Blog.