CPAN Testers at the Perl Toolchain Summit 2018

I made a lot of progress on CPAN Testers at this year's Perl Toolchain Summit (PTS). The PTS is an annual event devoted to maintaining and improving the Perl toolchain. The Perl toolchain includes things like:

For me, this year's Toolchain Summit was wildly productive, and as always, for every task I completed, two new tasks are revealed to take their place. If anyone would like to help, we could use web developers, backend data developers, devops help, API documentation help, and more. There are little tasks to do over a weekend, or big tasks to take ownership of. Contact me at doug@preaction.me and let me know what you'd be interested in.

Before I get into the full report of what I completed at the summit, I'd like to thank all of the sponsors for this event: NUUG Foundation,Teknologihuset, Booking.com, cPanel, FastMail, Elastic, ZipRecruiter, MaxMind, MongoDB, SureVoIP, Campus Explorer, Bytemark, Infinity Interactive, OpusVL, Eligo, Perl Services, Oetiker+Partner. Without sponsorship, this important work could not get done.

New APIs

While CPAN Testers primary goal is testing CPAN modules on a variety of different OSes and Perl versions, it is also used to test development Perl versions to see if they are backwards compatible with existing CPAN code. This project is called "Blead Perl Breaks CPAN", and has been helpful in keeping Perl 5 stable.

Before the summit, Todd Rinaldo announced that he and Nicolas R. were building a dashboard to show the current state of Blead Perl against CPAN, and track the work being done. In order to help make this dashboard work, I added some new APIs to http://api.cpantesters.org:

These APIs, along with all APIs on api.cpantesters.org are hosted on multiple servers behind a load balancer, and so are more reliable.

Performance Improvements

For new code, I will always recommend using api.cpantesters.org, but I am determined to ensure that the existing CPAN Testers ecosystem continues working so that the real work of keeping Perl and CPAN stable and useful can continue.

To that end, I built a new Mojolicious application to serve the individual test reports (like this example report for Minion-Backend-mysql). This has greatly improved performance for this page, which was the most expensive and most heavily-trafficked page on the site.

I plan to continue improving the performance of some of the current website's pages, including any JSON, XML, and RSS pages. I also have a plan to start banning the robots that are widely known to be poorly-behaved (ignoring robots.txt for example), which should also help system stability and performance.

Increased Stability

Stability maintenance is still my primary role on the CPAN Testers project. The first day of the summit, Todd and Nicolas helped me clear out some of the error messages the database has been getting. I spent some time on the second and third days cleaning things up.

  1. The Minion worker was not gracefully closing connections to MySQL, which causes an error message to appear "Aborted connection (Got an error reading communication packets)". The culprit appears to be the use of POSIX::_exit() in Minion::Job to prevent DESTROY blocks from running. I patched the Minion code on CPAN Testers, but upstream is not yet fixed.
  2. A very small percentage (<1%) of the incoming test reports exceed the default, 16 megabyte limit Mojolicious has on incoming requests. Now, I've increased the limit to 32 megabytes. I likely will not be increasing that limit further, so try to make sure your distribution's test suite output can fit.
  3. The CPAN::Testers::Report Metabase JSON object contains strings that contain JSON objects which also contain strings that contain JSON objects (Metabase::Facts are recursive data structures). The amount of backslash escaping required to deserialize this JSON was making Mojo::JSON take up gigabytes of memory to parse, eventually being killed by the system's out-of-memory killer. The Cpanel::JSON::XS module does not have this limitation, and now neither does CPAN Testers.

    Additionally, the process being killed by the OOM killer was leaving multi-megabyte /tmp files around, which led to an outage a few weeks ago when the /tmp directory filled up (800 Gigabytes of 30 Megabyte files). One outstanding task is to add monitoring around disk space usage on all the CPAN Testers machines to prevent this problem happening again.

All told, the event was a great success for me. A special thanks to all the organizers, whose tireless work in the months before the event makes sure everything goes smoothly, and whose work during the event makes sure everyone is fed, comfortable, and productive. I'm looking forward to the next one!

Leave a comment

About preaction

user-pic