NeilB

This is the third article in a series on CPAN distribution metadata. The first article was a general introduction, and the second article looked at dependencies, and in particular the different phases that you can specify dependencies for (configure, build, runtime, test, and develop). In this article, we'll cover the different types of dependencies and how you combine these with the phases (described in the previous article) to specify the dependencies (or prereqs) for your CPAN distribution.

This article is brought to you by MaxMind, a gold Sponsor for this year's Toolchain Summit, being held next month (May) in Lyon, France. The summit is only possible with the support of our sponsors.

Dependency Types

Most of the time, dependencies for CPAN distributions are required. This means that your distribution won't be installed unless all required dependencies could first be installed, as it's assumed that your module(s) won't work if one of the dependencies is missing.

But sometimes you can end up with optional dependencies. This might be to provide an optional feature, such as support for a number of export formats, or it might be an optional XS implementation for extra speed, falling back on a pure-perl implementation.

There are four different dependency types: requires, recommends, suggests, and conflicts. We'll look at each of them in turn. In the discussions below, we'll refer to the target distribution: This is the distribution being installed, and that prereqs are being checked for. To keep the examples below simple, we'll assume that the target distribution has a Makefile.PL based on ExtUtils::MakeMaker.

You should also be aware that the spec refers to these as dependency relationships, rather than types.

Requires

Required dependencies must be installed before the associated phase can be run for the target distribution. So if the target distribution has a required configure module, then that module must be installed before running

perl Makefile.PL

And if the required configure dependency can't be installed, then the installation of the target distribution must fail at that point.

Our example target module has a required configure dependency on ExtUtils::MakeMaker, so you'll see the following in META.json:

"prereqs" : {
    "configure" : {
        "requires" : {
            "ExtUtils::MakeMaker" : "6.3"
        }
    },
    ...
}

The great bulk of dependencies on CPAN have type "requires", so we'll not dwell on these any further.

Recommends

The spec says:

Recommended dependencies are strongly encouraged and should be satisfied except in resource constrained environments.

The interpretation of this is left up to individual CPAN clients:

• For the CPAN module, and the cpan script front-end, it depends on whether you've set the configuration option "recommends_policy.” If it's not set, or set to a false value, then any recommended prereqs are just ignored. If set to a true value, then CPAN will treat any recommended prereqs like requires: If they can't be installed, then the target distribution won't be installed. The default used to be for recommends_policy to not be set, which is the same as a false value. In recent versions of CPAN it now defaults to true, but only if you're installing it for the first time. If you've already got CPAN installed, updating to a more recent version won't change this config setting.
• By default, cpanm will similarly ignore any recommended prereqs, unless you give the --with-recommends command-line option. If you do give the option, cpan will try to install recommended prereqs, but if they fail, installation of the target dist will continue.

So by default, recommended prerequisites are ignored. More on that below.

Let's have a look at some distributions with recommended dependencies.

JSON::MaybeXS is a Perl module which will use the best available module for processing JSON. First it will use Cpanel::JSON::XS if that's available, then it will try JSON::XS, and if neither of those are available, it will fall back on JSON::PP, a pure-perl implementation that has been a core module since Perl 5.14.0.

If you look at the runtime prereqs in META.json, you'll see:

"runtime" : {
   "recommends" : {
      "Cpanel::JSON::XS" : "2.3310"
   },
   "requires" : {
      ...
      "JSON::PP" : "2.27300",
      ...
   }
},

Cpanel::JSON::XS is a recommended, but optional, dependency, and JSON::PP is a hard requirement. JSON::XS isn't listed as any kind of dependency: Cpanel::JSON::XS is a fork of JSON::XS. So, if the former couldn't be installed, it's unlikely a CPAN client will be able to install JSON::XS (but if JSON::XS is already installed, then it will be used in preference to JSON::PP).

But this isn't the whole picture! If you look at META.json, you'll see

"dynamic_config": 1,

As we covered in the first article of this series, this tells the CPAN client that dependencies need to be generated on the target platform, so it must first run:

perl Makefile.PL

And then use the metadata generated in MYMETA.json, rather than the META.json included in the release. And if you look at Makefile.PL, you'll see that it checks to see whether the target platform can handle XS modules. If it can, and JSON::XS isn't already installed, then Cpanel::JSON::XS is changed to be a required dependency.

David Golden's CPAN::Visitor is an implementation of the visitor pattern, for iterating over a CPAN repository. In its test prereqs, we see:

"recommends" : {
    "CPAN::Meta" : "2.120900"
},

This is an optional requirement for the test t/00-report-prereqs.t, which will still run without CPAN::Meta, but not do everything. This test is generated by the Test::ReportPrereqs plugin, which is included in DAGOLDEN's Dist::Zilla plugin bundle. When you run "make test” this test runs first, and lists the distribution's prereqs: output.

Suggests

The spec says:

These dependencies are optional, but are suggested for enhanced operation of the described distribution.

Again, the interpretation is left up to CPAN clients, with the two main clients having the same behaviour as for recommends:

• By default, CPAN/cpan won't try and install these dependencies, but you can change that behaviour with the "suggests_policy" configuration option.
• cpanm similarly won't try and install these dependencies, unless you give the --with-suggests command-line option.

Let's look at some suggested dependencies.

The Time::Stamp module provides functions for generating and parsing timestamps. If you have Time::HiRes installed, then you'll get fractional seconds in timestamps, but integer seconds otherwise. So Time::HiRes is specified as a suggested runtime prerequisite. It's been a core module since Perl 5.8.0, so for most people the suggested dependency is irrelevant (though an appropriate use of "suggests").

Module::Pluggable provides a mechanism for a module to easily support plugins. This works with App::FatPacker (a module which lets you roll up all the module dependencies for a script into a "packed" version of the script), and has a test that confirms this. So you can see in the metadata that App::FatPacker is a suggested test dependency.

Conflicts

This one is different. All the previous relationships are for expressing a module that will, or may, be used by the target distribution. A positive assertion, if you like. But a conflicts prerequisite says that the module must not be present. Here's what the spec says:

These libraries cannot be installed when the phase is in operation. This is a very rare situation, and the conflicts relationship should be used with great caution, or not at all.

With the regular type of prerequisites, it's common to see no version specific (a "0" given instead of a version). But with conflicts, you will almost always see it with a version. It is generally used to identify a version of a dependency that had a critical bug.

A good example is DBI's metadata, where you'll see the following:

"runtime" : {
    "conflicts" : {
        "DBD::Amazon" : "0.10",
        "DBD::AnyData" : "0.110",
        "DBD::CSV" : "0.36",
        "DBD::Google" : "0.51",
        "DBD::PO" : "2.10",
        "DBD::RAM" : "0.072",
        "SQL::Statement" : "1.33"
    },
}

Normally when you see a version number in prereqs, it means "this version or later", but here it means "not this version, or earlier".

The spec doesn't mention it, but you can also use expressions, rather than just a simple version number. For example, in META.json for Catmandu-PICA you'll see:

"runtime" : {
    "conflicts" : {
        "Catmandu::SRU" : "< 0.032"
    },
    ...
}

Which says that 0.032 and later are ok.

And in MooX-ClassAttribute's META.json you'll see:

"runtime" : {
    "conflicts" : {
        "Moo" : "== 1.001000",
        "MooseX::ClassAttribute" : "<= 0.26"
    }
},

This identifies a conflict with one specific release of Moo, and all releases of MooseX::ClassAttribute up to and including 0.26 conflict with MooX-ClassAttribute.

There are a number of problems with the "conflicts" type, though:

• In a prereqs statement, a version number normally means "this version or later", but here it's assumed to mean "this version or earlier".
• When DBI is installed, the user might not have DBD::Google installed. If the user later installs DBD::Google 0.51, will the CPAN client let the user know about the conflict? I'm pretty sure the answer is currently "no".
• I don't know if all CPAN clients handle expressions instead of version numbers.

The "conflicts" mechanism is now seen as an experiment that didn't work out. It was discussed at the Lancaster QA Hackathon, and again at the Berlin QA Hackathon. The currently proposed solution is an x_breaks key in distribution metadata. No CPAN client implements this as yet, though. No CPAN client implements this yet, but the [Test::CheckBreaks] plugin for Dist::Zilla lets a distribution enforce it, and the Moose distribution includes a moose-outdated script that checks for conflicts.

Conclusion

At the time of writing, there are just over 35,450 distributions on CPAN. Of those, only 2,994 (8.4%) have a recommended dependency, and just 436 (1.2%) have a suggested dependency. And a mere 24 distributions (0.07%) have a "conflicts" dependency.

Given the definition of "recommends" in CPAN::Meta::Spec ("strongly encouraged and should be satisfied except in resource constrained environments"), I think the behaviour of the two most popular CPAN clients (CPAN and cpanm) could be improved. I think a better default behaviour would be:

• Try to install recommended dependencies, but don't abort the phase if they can't be installed.
• Don't try to install suggested dependencies, unless the user has explicitly requested that.

I wondered if there is some reason for the current behaviour, or is it just that when it was spec'd out, no-one was quite sure exactly how they'd be used? I asked Tatsuhiko Miyagawa, the author of cpanm, and he said that circular dependencies were a problem, which he solved with this behaviour.

One approach to avoiding circular dependencies is to install recommended dependencies after the target distribution has been installed. This was also discussed at the Berlin QA Hackathon.

Writing this article has been a real journey. I discovered things I didn't know about, or fully understand. And some of those where I thought I had then found out the full picture, it turned out I hadn't! And no doubt still haven't. I'll try and improve some of my remaining gaps at the toolchain summit, and possibly submit some pull requests for documentation updates!

My thanks to everyone who helped with this article, particularly David Golden, Andreas König, Tatsuhiko Miyagawa, and Karen Etheridge.

Specifying dependencies for a CPAN distribution

In the next article in this series, we'll look at the different ways you can specify dependencies for a distribution, depending on which builder you're using.

About MaxMind

Founded in 2002, MaxMind is an industry-leading provider of IP intelligence and online fraud detection services. Thousands of companies of all sizes rely on MaxMind's GeoIP geolocation and end-user data for fraud prevention, ad targeting, content customization, digital rights management, and more. MaxMind's minFraud Network leverages this data along with IP and email reputations established from over 100 million transactions per month to help merchants prevent fraudulent online transactions and account registrations.

A number of CPAN authors work at MaxMind, including Olaf Alders (OALDERS), Florian Ragwitz (FLORA), Mark Fowler (MARKF), Chris Weyl (RSRCHBOY), TJ Mather (TJMATHER), Mateu Hunter (MATEU), Greg Oschwald (OSCHWALD), Andy Jack (ANDYJACK), Pat Cronin (PCRONIN), and Ruben Navarro (RUBEN).

Olaf Alders, who was the founder of the MetaCPAN project, will be attending the summit.

Our thanks again to MaxMind for supporting the Toolchain Summit.