CPAN Testers at meta::hack v2

Two weeks ago, I was invited to meta::hack v2, the second annual MetaCPAN hackathon. As the primary maintainer of CPAN Testers, I went to continue improving the integration of CPAN Testers data with MetaCPAN and generally improve the performance of CPAN Testers to the benefit of the entire Perl ecosystem.

Help CPAN Testers During meta::hack v2

Would you like to help CPAN Testers build some new data dashboards and new web applications during meta::hack v2? Join us on IRC in #cpantesters-discuss on irc.perl.org, join our mailing list on lists.perl.org, or e-mail me directly at doug@preaction.me.

With meta::hack v2 only two weeks away, I've written down my todo list for the hackathon. With another brand-new machine graciously provided by ByteMark, who have been hosting CPAN Testers for years, this year's hackathon will involve more devops tasks to improve reliability and stability of the various parts of the project.

The new server will be the host for CPAN Testers backend processes, the processes that turn the raw incoming data into the various reports used by the websites and downstream systems. It will also be the new home for the CPAN and BackPAN mirrors that CPAN Testers uses for data, and provides to external users as part of CPAN's mirrors list.

Application Metrics with Yertl

A time series database is a massively useful tool for system reporting and monitoring. By storing series of simple values attached to timestamps, an ops team can see how fast their application is processing data, how much traffic they're serving, and how many resources they're consuming. From this data they can determine how well their application is working, track down issues in the system, and plan for future resource needs.

There have been a lot of new databases and tools developed to create, store, and consume time series data, and existing databases are being enhanced to better support time series data.

With the new release of ETL::Yertl, we can easily translate SQL database queries into metrics for monitoring and reporting. I've been using these new features to monitor the CPAN Testers application.

Log::Any - Now With Structured Logging

The first trial release of Log::Any with mephinet's structured and contextual logging support has been released for feedback!

These features make it easier to log structured data which will then be picked up by log parsers. Adding a single, unblessed hashref as the last argument to a log method will write that data structure as a compact entry at the end of your log string:

use Log::Any '$LOG';
use Log::Any::Adapter 'Stdout';
$LOG->info( "Hello, World", { foo => 'bar' } );

$ perl test.pl
Hello, World {foo => "bar"}

Even better, you can add values to the logging context, which will also add those values to the log string as a compact data structure. Using local will automatically remove those values at the end of the scope.

use Log::Any '$LOG';
use Log::Any::Adapter 'Stdout';
local $LOG->context->{fizz} = "buzz";
$LOG->info( "Hello, World" );

$ perl test.pl
Hello, World {fizz => "buzz"}

And you can combine the two:

use Log::Any '$LOG';
use Log::Any::Adapter 'Stdout';
local $LOG->context->{fizz} = "buzz";
$LOG->info( "Hello, World", { foo => "bar" } );

$ perl test.pl
Hello, World {foo => "bar"} {fizz => "buzz"}

Adapters get these structures directly and can decide how to most effectively use them. For example, the Log4perl Log::Any adapter would likely map the Log::Any context to the Log4perl Mapped Diagnostic Context).

This is a trial release and the features may be changed slightly for simplicity and backwards-compatibility. Please give these new features a test and let us know how you like them by commenting here, by opening a ticket on the Log::Any ticket tracker on Github, or by e-mailing me preaction@cpan.org.

CPAN Testers Has an API

[Watch this lightning talk on The Perl Conference YouTube channel]

I've been working on the CPAN Testers project since 2015. In all that time, I've been focused on maintenance (which has involved more operations/administration tasks than any actual code changes) and modernization. It's that modernization effort that has led to a new CPAN Testers API.

This new API uses the Mojolicious web framework, along with an OpenAPI schema to expose all of the most useful CPAN Testers data. OpenAPI is a specification for web APIs, and there are tools like Swagger to generate a useful documentation website from your spec, like the CPAN Testers API documentation website.