Perl Toolchain Summit 2018 report

By Aaron Crane on April 26, 2018 11:04 PM

The Perl Toolchain Summit (previously known as the Perl QA Hackathon) is an annual coding workshop for people involved in the Perl toolchain, testing, quality assurance, and related issues. 2018 marked ten years of the PTS, so it seems fitting that it returned to Oslo, where it was first held. I’ve attended a few of them, and I was very pleased to take part in this one as well.

Day 0

I arrived at the hotel early afternoon on Wednesday 18th April. After dropping my bags in my room and coming down to the lobby I bumped into a few other attendees — and we got straight into a useful conversation about Perl core development. I skipped out after a while for what turned out to be an enjoyable trip to the Norwegian National Gallery. Then in the evening I joined the group in RØØR, a craft beer bar near the hotel.

Day 1

We started bright and early at the venue on Thursday morning, with a standup where we announced what we were all planning to work on. My initial order of business was to tackle some of the backlog for the core’s Configure script. First was syncing up the script in the Perl repo (which is automatically generated, but often needs to be manually modified by core hackers whose workflow and experience lie elsewhere) with its source units in the metaconfig repo. In this case, changes had been made on both sides. Tux (aka H. Merijn Brand) provided helpful assistance in figuring out how to unpick it all.

Once I’d done that, I could move on to working my way through the metaconfig pull request queue. I applied all the pending changes (including one PR of my own that had been approved by Tux, but now had conflicts with the master branch). Tux also pointed out a source of confusion in a feature I’d recently added to metaconfig, and suggested a tweak to it. After quibbling briefly, I eventually remembered I’d already (as suggested by the upstream metaconfig maintainer, Raphael Manfredi) implemented that tweak, so I merged it into our codebase too.

Day 2

I spent a chunk of Friday in group discussions. The biggest one was about managing dual-life modules in Perl, including how and when they should be synced between CPAN and blead — especially for upstream-blead modules. Though it took quite a while, I think it was productive. A smaller discussion later on was about the possibility of shipping local::lib with Perl, which has been suggested a few times. We agreed that bringing it into core wouldn't necessarily solve any problems.

Apart from that, I rifled my Perl core repo for some half-finished doc patches I'd had sitting around for a while. I cleaned them up, and got them merged to blead in time for the 5.27.11 release.

I also spent some time beginning to investigate how to handle something that Paul Johnson wants for Devel::Cover. Currently, the Perl core doesn’t provide an obvious way to produce coverage information for code that's in a module but not in a subroutine, which isn’t ideal. I've been promising Paul for the last two years that I’d look into this, so I was glad to have chance to tackle it. I managed to identify more or less what part of the Perl code was relevant before we broke off for the day.

Day 3

I returned to that problem on Saturday, but was initially sidetracked: I found that Devel::Cover was failing its tests when using a debugging build of Perl 5.27.11. This turned out to be caused by an optimisation introduced earlier in the 5.27.x cycle. In 5.26 and earlier, a foreach loop always has an internal OP_AND op to test whether the loop has finished iterating everything it needs to. The new optimisation skips the OP_AND loop, and directly embeds the branching logic into the OP_ITER op that represents the loop itself. The optimisation has internal assertions that ensure its expectations about the OP_AND are met. When code is run under Devel::Cover, those expectations don't entirely hold. Assertions are enabled only in debugging builds, which is why this was only now being discovered.

Fixing the bug involved deoptimising the OP_ITER when the expectation is incorrect. This will prevent breakage when using Devel::Cover, or any other module that uses similar implementation techniques of replace internal operations' op_ppaddr. I discovered the bug after Sawyer had made the 5.27.11 release earlier that day, so the core had gone into complete freeze; fortunately, Sawyer being present made it easy to get his explicit approval for merging despite the freeze (once Dave Miller had reviewed my patch).

With that fixed, I did manage to get much further with what Paul wanted, though again, I had to return to it the next day. That evening we went en masse to a local restaurant for the PTS 10th anniversary dinner. It was an enjoyable night, though unfortunately I was a bit too tired to make the most of it. I went back to the hotel promptly after we all left, and went to sleep pretty much immediately.

Day 4

Sunday saw me working further on the same Devel::Cover matter. Paul and I had initially characterised the issue as Perl freeing top-level module code before Devel::Cover can look for it. On that basis, we were expecting that I’d write a feature to disable the freeing, so that Devel::Cover would be able to provide better coverage data on newer Perls, even if nothing can be done on older Perls. However, after some digging, I worked out what seemed to be a way to get at the top-level module code at precisely the point between Perl compiling it and Perl running it: part of the interface for implementing a debugger involves running a hook at that point, and code written in XS can (at least in some versions of Perl) find the in-memory structures representing the code. I was happy to be able to report this to Paul. Sadly, there’s something else getting in the way: although the internal CV representing the code is present when the hook is invoked, its optree doesn't seem to be available when examining it later. I suspect I don’t quite understand what Perl does when compiling code. Hopefully I or someone else on p5p will be able to provide a solution here.

My other task on Sunday was to work on Data::Dumper. As an upstream-blead dual-life module, changes are made to it in blead, and from time to time, its maintainer pushes releases out to CPAN. However, the current version in blead has a variety of failures on Perl 5.18 and below. It has a workaround for some versions: the XS implementation of subroutine deparsing causes one obvious problem, so the XS implementation is disabled on those versions of Perl, in favour of the pure-Perl implementation. However, there were other unexplained issues too. I dug into them carefully, and made the Data::Dumper test suite pass on Perl back as far as 5.8.x.

The most interesting fix was for XS deparsing. This was a tricky one to track down, until Dagfinn Ilmari Mannsåker helpfully suggested bisecting Perl against the failing code. The explanation turned out to be that when the XS code loaded B::Deparse (to do the actual grunt work of deparsing), the Perl data stack could get reallocated and moved. An old Perl commit had fixed that in a general way for 5.20 and above, which is why there was no problem in those versions. The fix for Data::Dumper was to conditionally create a new stack when loading B::Deparse, in an older version of Perl.

I pushed the Data::Dumper fixes to a branch on the Perl repo, but they can't be merged yet, because the core is frozen. I plan to merge those changes early in the 5.29.x release cycle, and then a new non-development CPAN release can be issued based on the version in core.

Aaron Crane