preaction

I made a lot of progress on CPAN Testers at this year's Perl Toolchain Summit (PTS). The PTS is an annual event devoted to maintaining and improving the Perl toolchain. The Perl toolchain includes things like:

• How CPAN distributions are built and installed (modules like ExtUtils::MakeMaker and Module::Build)
• How CPAN distributions are developed and released: PAUSE, the Perl Authors Upload Server, the primary source for CPAN mirrors, but also modules like Dist::Zilla
• How CPAN distributions are downloaded: CPAN clients like CPAN.pm
• And other tools around the CPAN ecosystem: Search engines like MetaCPAN, test tools like CPAN Testers, and coverage tools like CPAN Cover

For me, this year's Toolchain Summit was wildly productive, and as always, for every task I completed, two new tasks are revealed to take their place. If anyone would like to help, we could use web developers, backend data developers, devops help, API documentation help, and more. There are little tasks to do over a weekend, or big tasks to take ownership of. Contact me at doug@preaction.me and let me know what you'd be interested in.

Before I get into the full report of what I completed at the summit, I'd like to thank all of the sponsors for this event: NUUG Foundation,Teknologihuset, Booking.com, cPanel, FastMail, Elastic, ZipRecruiter, MaxMind, MongoDB, SureVoIP, Campus Explorer, Bytemark, Infinity Interactive, OpusVL, Eligo, Perl Services, Oetiker+Partner. Without sponsorship, this important work could not get done.

New APIs

While CPAN Testers primary goal is testing CPAN modules on a variety of different OSes and Perl versions, it is also used to test development Perl versions to see if they are backwards compatible with existing CPAN code. This project is called "Blead Perl Breaks CPAN", and has been helpful in keeping Perl 5 stable.

Before the summit, Todd Rinaldo announced that he and Nicolas R. were building a dashboard to show the current state of Blead Perl against CPAN, and track the work being done. In order to help make this dashboard work, I added some new APIs to http://api.cpantesters.org:

• You can now request test summary data by distribution, and filter those results by OS name and Perl version.
• You can now ask for test summary data across all CPAN distributions, though you must limit your request to either certain Perl versions or certain OS names.
• You can now get a single release's pass/fail statistics as an API, making it very easy to show CPAN Testers data next to your releases
  • In this same vein, I would love it if someone built an API to have a CPAN Testers badge API like one of these at shields.io

These APIs, along with all APIs on api.cpantesters.org are hosted on multiple servers behind a load balancer, and so are more reliable.

Performance Improvements

For new code, I will always recommend using api.cpantesters.org, but I am determined to ensure that the existing CPAN Testers ecosystem continues working so that the real work of keeping Perl and CPAN stable and useful can continue.

To that end, I built a new Mojolicious application to serve the individual test reports (like this example report for Minion-Backend-mysql). This has greatly improved performance for this page, which was the most expensive and most heavily-trafficked page on the site.

I plan to continue improving the performance of some of the current website's pages, including any JSON, XML, and RSS pages. I also have a plan to start banning the robots that are widely known to be poorly-behaved (ignoring robots.txt for example), which should also help system stability and performance.

Increased Stability

Stability maintenance is still my primary role on the CPAN Testers project. The first day of the summit, Todd and Nicolas helped me clear out some of the error messages the database has been getting. I spent some time on the second and third days cleaning things up.

1. The Minion worker was not gracefully closing connections to MySQL, which causes an error message to appear "Aborted connection (Got an error reading communication packets)". The culprit appears to be the use of POSIX::_exit() in Minion::Job to prevent DESTROY blocks from running. I patched the Minion code on CPAN Testers, but upstream is not yet fixed.
2. A very small percentage (<1%) of the incoming test reports exceed the default, 16 megabyte limit Mojolicious has on incoming requests. Now, I've increased the limit to 32 megabytes. I likely will not be increasing that limit further, so try to make sure your distribution's test suite output can fit.

3. The CPAN::Testers::Report Metabase JSON object contains strings that contain JSON objects which also contain strings that contain JSON objects (Metabase::Facts are recursive data structures). The amount of backslash escaping required to deserialize this JSON was making Mojo::JSON take up gigabytes of memory to parse, eventually being killed by the system's out-of-memory killer. The Cpanel::JSON::XS module does not have this limitation, and now neither does CPAN Testers.

   Additionally, the process being killed by the OOM killer was leaving multi-megabyte /tmp files around, which led to an outage a few weeks ago when the /tmp directory filled up (800 Gigabytes of 30 Megabyte files). One outstanding task is to add monitoring around disk space usage on all the CPAN Testers machines to prevent this problem happening again.

All told, the event was a great success for me. A special thanks to all the organizers, whose tireless work in the months before the event makes sure everything goes smoothly, and whose work during the event makes sure everyone is fed, comfortable, and productive. I'm looking forward to the next one!