The Perl Toolchain Summit 2019

This year I was again privileged to attend the Perl Toolchain Summit, held at Bisham Abbey, near Marlow in southern England.

The Perl Toolchain summit is an opportunity for the developers and maintainers of Perl's infrastructure to get together and do whatever is necessary to keep that infrastructure running well and improve its services.

I arrived at PTS with a TODO list which I knew was already more than I would be able to complete. But PTS is not just about getting your own work done. A major benefit of having thirty perl hackers together in a room is that often when someone hits a problem or needs guidance or information they can simply wander over to the table where the author of the software they are using is sitting and ask them directly.

On the first day in the standup meeting I mentioned that I was looking for a solution to install system dependencies of CPAN modules before generating their coverage and that I was sure someone must have solved that problem. Immediately thereafter Slaven came up to me and told me about his module CPAN::Plugin::Sysdeps which does just that. Philippe offered to integrate it into cpancover and before the end of the summit I had a pull request. This type of interaction is a major benefit of the PTS.

Similarly I have (along with many others), for years, wanted to be able to gather coverage information for top-level statements in modules. This is not easy because perl executes these statements on loading the module and then, with no further use for the optree, throws it away, so that Devel::Cover can't get to it later.

This year, I managed to enlist the help of Aaron Crane, who knows far more about the perl internals than I do. Aaron had previously suggested that the DB::postponed subroutine might be part of a solution and this year he spent some time determining the rest of the solution. I was then able to program it up as a proof of concept. The full solution will require changes to the perl core which I will still need to properly implement, but without Aaron's help I'd never have got to a solution. (I know this because I have spent a long time trying and I was on the wrong path(s).)

For a while I have been discussing closer integration of cpancover with metacpan. The fruit of part of these conversations came online a short time ago when metacpan started showing module coverage figures and linking into cpancover for more detailed information. We have also been discussing matters of infrastructure and investigating backup/failover/redundancy for cpancover. Based on some plans made last year and implemented recently, along with Leo's work during PTS, we now have cpancover data backed up to the metacpan infrastructure, and we have a plan for dealing with downtime on the main cpancover server. This is the sort of thing which is so much easier to do by pulling three or four people into a discussion rather than trying to coordinate over mail or IRC.

PTS has become the most important Perl event of the year for me, and I am very grateful to everyone who attends, to the organisers, Neil, Philippe and Laurent, who did a wonderful job, and to all the sponsors without whom it would not have been possible:, cPanel, MaxMind, FastMail, ZipRecruiter, Cogendo, Elastic, OpenCage Data, Perl Services, Zoopla, Archer Education, OpusVL, Oetiker+Partner, SureVoIP, YEF.

The Perl Toolchain Summit 2018

This year marks the 10 year anniversary of the Perl Toolchain Summit, formerly known as the Perl QA Hackathon.

The Perl Toolchain Summit provides an opportunity for around 30 Perl developers to get together for four days or so to talk about and develop the infrastructure which surrounds Perl. This includes being able to build modules, being able to test them on multiple systems, being able to find out about them, being able to manage them, and many other related areas.

Salve Nilsen started the QA Hackathon back in 2008 in Oslo, and we returned in 2018 with Salve again being involved in the organisation.

This year I was very happy to discover that a number of people arrived in Oslo with the idea of working on some area of Devel::Cover (the Perl code coverage module) or cpancover (the service that provides coverage information for all of CPAN).

This meant that for me much of the summit was spent in collaboration with people who were doing the real work, and I was very pleased about that - such collaboration, which is facilitated and sometimes only made possible by being together physically, is the purpose of the summit.

One such collaboration was with Joel Berger who implemented a queueing system for Currently I have a crude shell script which does the work it thinks it needs to do, sleeps for 10 minutes and then looks for new work. This means that there can be a delay of a couple of hours or more between a module being uploaded and the coverage being calculated, especially if the module being uploaded happens to have been uploaded just after another module for which the coverage run takes a particularly long time. With the queueing system Joel has implemented the coverage will usually start to be calculated almost immediately. There are still a few nits to be sorted out, but I hope to put this live in the near future.

For quite a while now I have been trying to find a way to get the coverage information from top-level statements in modules. This has become more important as Dancer, Moose and other frameworks have had more code in top-level statements. Aaron Crane spent some time on working out how this might be done and has come up with a solution which is tantalisingly close, but so far we've not quite got to the finishing line. The problem here is that perl, quite reasonably, recognises that this code will only ever be run once, at the start of the execution and so it throws away the optree as soon as it has been executed. The optree is Devel::Cover's hook into what code has been executed, and persuading perl to leave to optree lying around where we can find it later has proven to be remarkably difficult. I will continue looking at this but thanks to Aaron's knowledge and assistance we may shortly* have a solution. *In relation to the length of time I have wanted a solution.

The UI on is awful, in many ways. Babs has be doing wonderful work on making Perl websites look as if they were created this century. We looked at cpancover and she sees a lot of potential for improvement. However, I do not want the front page of cpancover to be a page that is visited very often. Rather, I would prefer that metacpan linked to the internal cpancover pages and thus metacpan becomes the site to search for coverage data.

So I discussed with the metacpan team how we might incorporate coverage data into metacpan, and use its UI as the main starting point for getting access to coverage data. Mickey spent some time building a system to take the data from cpancover and add it into the the metacpan data. The data still needs to be incorporated into the metacpan UI, but we are almost there.

I talked with Garu about uploading users' local coverage data to cpantesters via Test::Reporter and how we might be able to eventually provide coverage data from many users on different systems in consistent, merged coverage reports. There is a lot of work to be done here, but we might be able to get a simple first step in place without too much difficulty . (By we, I mean Garu, of course. And thus I mean without too much difficulty for me.)

As far as my individual work was concerned, I have managed to get the cpancover reports flowing again. I had been storing and serving all the coverage data I had ever calculated, and I managed to run out of inodes on the partition where that data was stored. I have now implemented a retention policy of only serving the latest three module versions, but older versions are being archived. I also managed to bring the server down for a number of hours, so sorry about that if you noticed.

All in all, this was an extremely productive summit for me with respect to Devel::Cover and cpancover. I consider the Perl Toolchain Summit to be the most important event in my technical calendar and I am extremely grateful to the organisers of this year's summit, as I am to the organisers every year, for all their efforts in making it possible. Its also great to see old friends as well as people who have previously only been known online. Along with the work of the organisers, the summit would not have been possible without the generosity of our sponsors. And, of course, the volunteer efforts put in by the talented and diligent attendees whose ranks I somehow managed to infiltrate.

I'm already looking forward to next year.

Sponsors for the Perl Toolchain Summit 2018

NUUG Foundation, Teknologihuset,, cPanel, FastMail, Elastic, ZipRecruiter, MaxMind, MongoDB, SureVoIP, Campus Explorer, Bytemark, Infinity Interactive, OpusVL, Eligo, Perl Services, Oetiker+Partner.

Perl Toolchain Summit 2017 - Day 4

I didn't get as much done on the final day of the Perl Toolchain Summit because I needed to get home for work on Monday. But I did manage to enlist others in helping me, which is even better.

Devel::Cover has always had a problem in that top level statements (those not in a sub) in modules cannot be covered. This is because perl throws away the optree as soon as it has executed. My knowledge of the internals has never been good enough to even work out a sensible plan for how to solve this, but fortunately Aaron Crane is a good deal cleverer than me and thinks he might be able to develop a solution. It may need perl to have extra hooks, or it might involve deep magic, or both, but at least the end of the tunnel is not pitch black any more. Thanks Aaron!

Then I was discussing how to improve cpancover with Olaf and mentioned that a queue for newly uploaded modules would help to reduce latency for the coverage reports. Joel Berger was sitting right there too, and he knows something about Minion, the job queue for Mojolicious. Joel kindly offered to put together a queue system to replace my naïve shell script. Thanks Joel!

I took part in the discussion about the future of the PTS (spoiler - we like it, and have a few ideas on how to make it better still) and then it was time to head back home.

Thanks to Neil, BooK, and Laurent for organising such a great event. Thanks to Wendy to keeping us all healthily fed (or not so healthily, if desired). Thanks to Lee for giving me lifts. Thanks to everyone who attended and helped make the event such a success. And, as ever, thanks to the sponsors:, ActiveState, cPanel, FastMail, MaxMind, Perl Careers, MongoDB, SureVoIP, Campus Explorer, Bytemark, CAPSiDE, Charlie Gonzalez, Elastic, OpusVL, Perl Services, Procura, Oetiker+Partner.

Perl Toolchain Summit 2017 - Day 3

My third day at the Perl Toolchain Summit was primarily spent in trying to make the cpancover server and infrastructure into more of a production-ready system and less of a Devel::Cover playground. The first step in this direction was supposed to be easy - I made a login for the metacpan group with the idea that they could regularly rsync the coverage results for backup purposes. Unfortunately, this lead me down a yak shaving path I wasn't planning on travelling until later.

When setting up cpancover, I decided to take the easy option and chuck all the results into a single directory. I made the filesystem ext4 so I wouldn't have to worry about hitting limits. Unfortunately, the metacpan box doing the rsync is set up on ext3 and won't support more that 32k subdirectories. So I need to fix up the way that results are stored. I knew this would come sooner or later, even if only because I would surely one day get sufficiently tired of typing ls and immediately regretting it.

In order to make such a change, I need a proper development area to test it. Cpancover has basically just been running since I set it going about three years ago, when I pretty much built it in place. So before making such a large change I need to be able to control things like the docker image being used and the directories being written to. This is also important to be able to allow anyone else to work on the system. So much of the day was spent on this.

Two people separately came to me with the problem that some of their tests require extra modules to be installed and so, by default, these tests aren't run and the coverage suffers accordingly. Four years ago, at the QA Hackathon in Lancaster, a number of clever people got together, and I was there too. We thrashed out The Lancaster Consensus which, thankfully, included a solution to this problem. The environment variable $EXTENDED_TESTING can be set to indicate that the user or process running tests is willing to run optional tests that may take extra time or resources to complete. So I made cpancover set that environment variable, and Graham and Tux altered Moo and Spreadsheet::Read respectively to pay attention to it. I mention this in detail for such a simple change (for me) just to point out (again) the value of the summit, not just for the work which takes place at the time, but also for how it smooths subsequent work, even years later.

I did a few other bits and bobs, and ended up watching an Italian dancing gorilla and a horse on a ladder from Azerbaijan whilst waiting for some modules to install, which can't be bad. (Thanks Eurovision.)

Thanks again to all the sponsors who made this possible:, ActiveState, cPanel, FastMail, MaxMind, Perl Careers, MongoDB, SureVoIP, Campus Explorer, Bytemark, CAPSiDE, Charlie Gonzalez, Elastic, OpusVL, Perl Services, Procura, Oetiker+Partner.

Perl Toolchain Summit 2017 - Day 2

My first day at the Perl Toolchain Summit (PTS) was largely spent coding, or dealing with pull requests, tickets, and releasing. The second day had far less of that sort of stuff, and a lot more of the sort of stuff that it's much harder to do away from the summit.

But it started off with a new pull request from Todd Rinaldo resurrecting a closed RT ticket. There is a whole class of problems Devel::Cover has to deal with which stem from the program being exercised changing the environment as it runs. The most obvious, perhaps, is changing directory, which is also something I looked at again yesterday. But this time it has to do with dropping permissions during the run. Todd and I had a discussion about it and, with the principles in place we'll deal with the technical matters in the coming days.

Then I had a productive discussion with Leo and Olaf regarding cpancover. With a bus number of somewhat less than one, we want to make more resilient. As a first step, the coverage data will simply be backed up to the metacpan infrastructure, meaning that in the event of hardware failure, or a mistaken rm -r, we can get ourselves back up again with the historical data.

We also discussed how we might be able to link metacpan and cpancover more closely, and I look forward to making progress in that area.

Then I spent some time reviewing the way in which the cpancover server is set up, and improving and documenting the process, with the idea that it should be fairly simple for anyone to be able to reprovision the server, or set up a new one.

I should note here that the cpancover server is generously provided by Bytemark. I also have a personal server there and they provide an excellent service.

After dinner, Leo and Lee helped me through the process of setting up metacpan locally, using vagrant and virtualbox. The process was painless due to the fine work they had done in automating it.

I also managed to process a few odd tickets here and there during the day and do other bits and bobs. All in all, it was the type of day that is only really possible at an event such as this, so thanks again to all the sponsors for making it possible:, ActiveState, cPanel, FastMail, MaxMind, Perl Careers, MongoDB, SureVoIP, Campus Explorer, Bytemark, CAPSiDE, Charlie Gonzalez, Elastic, OpusVL, Perl Services, Procura, Oetiker+Partner.