Application Metrics with Yertl

A time series database is a massively useful tool for system reporting and monitoring. By storing series of simple values attached to timestamps, an ops team can see how fast their application is processing data, how much traffic they're serving, and how many resources they're consuming. From this data they can determine how well their application is working, track down issues in the system, and plan for future resource needs.

There have been a lot of new databases and tools developed to create, store, and consume time series data, and existing databases are being enhanced to better support time series data.

With the new release of ETL::Yertl, we can easily translate SQL database queries into metrics for monitoring and reporting. I've been using these new features to monitor the CPAN Testers application.

Log::Any - Now With Structured Logging

The first trial release of Log::Any with mephinet's structured and contextual logging support has been released for feedback!

These features make it easier to log structured data which will then be picked up by log parsers. Adding a single, unblessed hashref as the last argument to a log method will write that data structure as a compact entry at the end of your log string:

use Log::Any '$LOG';
use Log::Any::Adapter 'Stdout';
$LOG->info( "Hello, World", { foo => 'bar' } );

$ perl test.pl
Hello, World {foo => "bar"}

Even better, you can add values to the logging context, which will also add those values to the log string as a compact data structure. Using local will automatically remove those values at the end of the scope.

use Log::Any '$LOG';
use Log::Any::Adapter 'Stdout';
local $LOG->context->{fizz} = "buzz";
$LOG->info( "Hello, World" );

$ perl test.pl
Hello, World {fizz => "buzz"}

And you can combine the two:

use Log::Any '$LOG';
use Log::Any::Adapter 'Stdout';
local $LOG->context->{fizz} = "buzz";
$LOG->info( "Hello, World", { foo => "bar" } );

$ perl test.pl
Hello, World {foo => "bar"} {fizz => "buzz"}

Adapters get these structures directly and can decide how to most effectively use them. For example, the Log4perl Log::Any adapter would likely map the Log::Any context to the Log4perl Mapped Diagnostic Context).

This is a trial release and the features may be changed slightly for simplicity and backwards-compatibility. Please give these new features a test and let us know how you like them by commenting here, by opening a ticket on the Log::Any ticket tracker on Github, or by e-mailing me preaction@cpan.org.

CPAN Testers Has an API

[Watch this lightning talk on The Perl Conference YouTube channel]

I've been working on the CPAN Testers project since 2015. In all that time, I've been focused on maintenance (which has involved more operations/administration tasks than any actual code changes) and modernization. It's that modernization effort that has led to a new CPAN Testers API.

This new API uses the Mojolicious web framework, along with an OpenAPI schema to expose all of the most useful CPAN Testers data. OpenAPI is a specification for web APIs, and there are tools like Swagger to generate a useful documentation website from your spec, like the CPAN Testers API documentation website.

2017 Perl Toolchain Summit

This year I had one goal for CPAN Testers: Replace the current Metabase API with a new API that did not write to Amazon SimpleDB. The current high-availability database that raw incoming test reports are written is Amazon SimpleDB behind an API called Metabase. Metabase is a highly-flexible data storage API designed to work with massive, unstructured data sets and still allow for sane organization and storage of data. Unfortunately, Amazon SimpleDB is as it says on the tin: Simple. Worse, it's expensive: Like most Amazon services, it charges for usage, so there's a huge incentive for CPAN Testers to use it as little as possible (which made some of the code quite obtuse).

So, I made a plan to excise the Metabase. Since we already cached every raw test report locally in the CPAN Testers MySQL database, I planned to write a new Metabase API that wrote directly to the cache, and then adjust the backend processing to read from the cache. I spent the better part of a month working through all the Metabase APIs, how the data was stored in the database, and how to translate between a simple JSON format and the serialized Metabase objects. However, some proper schema design prevented me from finishing this project: A single NOT NULL column could not be changed to allow nulls very easily, it being a 600GB table. The one time where a well-designed schema was a bad thing!

But then Garu, author of cpanm-reporter and CPAN::Testers::Common::Client came up with an idea to make a new test report format. These new reports would have to be stored in a new place, and I discovered that MySQL had recently started building some rich JSON tooling. Making a new JSON test report format and storing it in our new high-availability MySQL cluster seemed like a perfect solution for storing our raw test reports.

After a few weeks of discussion, I finally realized that it would be an easier task to make a backwards-compatible Metabase API write to the new test report MySQL table, even though it increased the amount of work that needed to be done:

  • Complete the new test report format schema (Garu)
  • Write the new backwards-compatibility Metabase API (Me)
  • Write a new test report processor that writes to the old Metabase cache tables (Joel Berger)
  • Write a migration script from the old Metabase cache tables to the new test report JSON object (?)

With that plan, I headed for Lyon.

Timeouts for Parallel::ForkManager

At tonight's Chicago Perl Mongers Office Hours, Ray came up with an interesting problem. While testing all of CPAN for CPAN Testers, how do you detect when a test is hanging and kill it before it takes down the entire machine? How do you simply kill a test that is taking too long? And how do you do it without having a wholly separate watchdog program?

Ray's using Parallel::ForkManager to execute testing jobs in parallel across multiple Perl installs. There are a few ways we could implement timeouts, including IPC::Run's timeout function, or the alarm Perl built-in, but these must all be implemented in the child process. It'd be nicer if we could use the parent process to watch its own children.