Joel Berger

MetaCPAN is the community developed and maintained website and api for finding and learning about Perl modules. This year, we dedicated a long weekend to improving it and oh what a weekend it was!

MetaCPAN had been developed using an early version of Elasticsearch and though it has served the project well ES moves quickly. A year or so ago MetaCPAN developers decided they needed to address the fact that the database was well out of date and the daunting amount of breaking changes that had happened since then. This work continued at this year’s Perl QA Hackathon, when I met Olaf Alders, Mickey Nasriachi, and Leo Lapworth who were diligently working on the migration.

I hadn’t come to QAH with any specific plans; I hopped aboard with MetaCPAN when I heard that they had started using Minion. Minion is a job queue, generally used with Mojolicious but easily used elsewhere. As part of the Mojolicious project I was happy to get up to speed on the project to be ready to help them if needed.

At QAH the migration made a lot of progress but didn’t quite make it over the hump. Just afterwards, we mentioned that with a few more days of hacking we could have finished. Not only that but we enjoyed each other’s company and we thought it would be fun to do it again.

The thought took shape over the following months and this past weekend I had the extreme pleasure of attending the (first) Meta::Hack with a group of very talented developers to work on MetaCPAN. The attendees were Olaf, Mickey and Leo, as well as Thomas Sibley, Graham Knop, Brad Lhotsky and Doug Bell. Doug’s and my employer, ServerCentral hosted the event November 17-20, 2016. They also host Chicago Perl Mongers; we are very grateful to work for a company so devoted to supporting Perl and Open Source.

This primary focus of the first two days was the remaining work of the Elasticsearch migration. I’m happy to report that as a result of the hackathon the migration is complete and has been deployed!

While the lead developers were completing that work, I audited the code and wrote some additional tests for the new download_url endpoint. Once the new backend went live it was “all hands on deck” watching for errors in the log and checking for other oddities.

One of the most fun projects was trying to discover why certain search results were getting repeated. As we dug through the search, trying more and more ideas, pulling in developer after developer. Finally Graham won the day, figuring out that as a result of the new data structures, we were uniquely filtering on not the name but an array reference containing the name, defeating the uniqueness check. It may sounds obvious now, but it really was quite an adventure getting there!

As a result of that challenge, I became much more familiar with the code surrounding the search itself. Several years ago I had noticed that it was nearly impossible to get search results from the api that mimicked the website’s own results. The reason turns out to be that the website performs a very complex Elasticsearch query from the front-end server, more-or-less bypassing the api to do so. Having learned about the search from the uniqueness fiasco, I spent the next two days moving the search from the web service to the api service.

This change, once merged, will allow api consumers to get search results directly. In this way external projects could have documentation or module search directly from MetaCPAN. Additionally, the search could also be extended to expose additional search parameters, possibly allowing clients to optimize the search for their particular needs.

I was so pleased to be able to take part and contribute in my own way to the furthering of Meta::CPAN. I want to thank the attendees once again, gentlemen and scholars all, and Neil Bowers for organizational help. Further this meeting wouldn’t have been possible without the extraordinarily generous support of our sponsors:

Our platinum sponsors were Booking.com and cPanel. Our gold sponsors were Elastic, FastMail, and Perl Careers. Our silver sponsors were ActiveState, Perl Services, ServerCentral, and Advance Systems Our bronze sponsors were Vienna.pm, easyname, and the Enlightened Perl Organisation (EPO). My warm thanks to each one!