CPAN Testers

After 6 months of development work, following 2 years worth of design and preparation, CPAN Testers 2.0 is finally live.

With the rapid growth in CPAN Testers environments and testers over the past few years, the previous method of posting reports to a mailing list had reached a point where the scalability was no longer viable. This was recognised several years ago and discussions for a new system had already begun, with the view that reports should be submitted via HTTP.

At the Oslo QA Hackathon in 2008, David Golden and Ricardo Signes devised the Metabase, with the design work continuing at the Birmingham QA Hackathon in 2009, where David and Ricardo were able to bring others into the thought process to work through potential issues and begin initial coding. A number of releases to CPAN and Github followed, with more people taking an interest in the project.

The Metabase itself is a database framework and web API to store and search opinions from anyone about anything. In the terminology of Metabase, Users store Facts about Resources. In the Metabase world, each CPAN tester is a User. The Resource is a CPAN distribution. The Fact is the test report. Today that's just the text of the email message, but in the future it will be structured data. The Metabase specifies data storage capabilities, but the actual database storage is pluggable, from flat files to relational databases to cloud services, which gives CPAN Testers more flexibility to evolve or scale over time.

Meanwhile the CPAN Testers community was also attracting more and more interest from people wanting to be testers themselves. As a consequence the volume of reports submitted increased each month, to the point that the perl.org mail server was struggling to deal with all the mailing lists it hosted. The cpan-testers mailing list was submitting more posts in one day than any other list submitted in a month (in a year in some cases). Robert and Ask, very reasonably, asked if the testers could throttle their submissions down to 5k report posts a day, and set a deadline of 1st March 2010 to switch off the mailing list.

David Golden quickly took on the task to envisage a project plan, and work began in earnest in December 2009. With less than 3 months to the cut-off date, there was a lot of work to do. David concentrated on the Metabase, with Barbie working on ensuring that the current cpanstats database and related websites could move to the Metabase style of reports. Despite a lot of hard work from a lot of people, we unfortunately missed the 1st March deadline. Having throttled report submissions to a more manageable level, and although not complete, the target for HTTP submissions was in sight, Robert and Ask were very understanding and agreed to keep us going a little while longer.

Throughout March and April a small group of beta testers were asked to fire their submissions at the new system. It ironed out many wrinkles and resulted in a better understanding of what we wanted to achieve. The first attempts at retrieving the reports from the Metabase into the cpanstats database began in April, and again highlighted further wrinkles that needed to be addressed. After a month of hard testing and refinement, we finally had working code that went from report submission by a tester, storage into the Metabase, retrieval into the cpanstats database and finally presentation on the CPAN Testers family of websites.

During June the process was silently switched from testing to live, allowing reports to be fed through into the live websites. Due to the ease with which the new style reporting fit into the existing system, the switch largely went unnoticed by the CPAN testers community as well as the Perl community. A considerable success.

The CPAN Testers eco-system is now considerably larger than those early days of simply submitting handwritten reports by email to a mailing list, and the work to get here has featured a cast of thousands. Specifically for CPAN Testers 2.0, the following people have contributed code, ideas and effort to the project over the past six months:

• Andreas König
• Apocalypse
• Ask Bjørn Hansen
• Barbie
• Chris Williams
• Dan Collins
• David Cantrell
• David Golden
• Florian Ragwitz
• H.Merijn Brand
• Jon Allen
• Lars Dɪᴇᴄᴋᴏᴡ 迪拉斯
• Léon Brocard
• MW487
• Nigel Horne
• Ricardo Signes
• Richard Dawe
• Robert Spier
• Serguei Trouchelle
• Shlomi Fish
• Slaven Rezić

Barbie and David would like to thank everyone for their involvement. Without these guys CPAN Testers 2.0 would not have been possible. Thanks to everyone, we can now look forward to another 10 years and more of CPAN Testers.

CPAN Testers now holds over 7.5 million test reports covering nearly 11 years worth of testing Perl distributions. There have been over 1,000 testers in that time, and every single one has helped the CPAN Testers project to be the largest single community supported testing system of any programming language. For a full list of everyone who has contributed, visit the CPAN Testers Leaderboard. A huge thank you to everyone.

With the Metabase now online and live, we can now announce an absolute deadline to close the mailing list. This is currently set as 31st August 2010. After this date all submissions via email will be rejected, and testers will be encouraged to upgrade their testing tools to take advantage of the new HTTP submission system. Many of the high volume testers have already moved to the new system, and we expect nearly everyone else to move in the next month. We will be tailing the SMTP submissions to catch those who haven't switched, such as some of the more infrequent testers, and warn them of the deadline.

More work is planned for CPAN Testers, from further validation and administration of reports, to providing more functionality for alternative analysis and search capabilities. Please check the CPAN Testers Blog for our regular updates.

If you'd like to become a CPAN Tester, please check the CPAN Testers Wiki for details about setting up a smoke testing environment, and join the cpan-testers-discuss mailing list where many of the key members of the project can offer help and advice.

You can find out more about CPAN Testers at two forthcoming conferences. David Golden will be presenting "Free QA! What FOSS can Learn from CPAN Testers" at OSCON and Barbie will be presenting "CPAN Testers 2.0 : I love it when a plan comes together" at YAPC::Europe.

CPAN Testers is sponsored by Birmingham Perl Mongers, and supported by the Perl community.

You can now download the full and complete Press Release from the CPAN Testers Blog. If you have access to further IT news reporting services, please feel free to submit the Press Release to them. Please let us know if you are successful it getting it published.

Cross-posted from the CPAN Testers Blog