Buddy Burden

You may recall that my mentioning that my favorite talk at this year’s YAPC was Sawyer X’s “The Joy in What We Do”.  If you remember (or click one of these links I keep throwing at you), long about 26:28 Sawyer X makes a radical suggestion: if you want to release an application which uses Perl, maybe you shouldn’t be releasing it via CPAN.  I mean, CPAN is awesome for modules, and it’s not too shoddy for Perl apps either.  It makes it very easy to install for people who already have Perl, and know how to operate the CPAN shell, or cpanm, and either have root access on their machine or are already using perlbrew or plenv, and ...  In other words, other Perl programmers.  And, if that’s your audience, then lovely.  Although, even then ... what if you need a particular version of Perl, and particular versions of certain modules?  It’s all doable, certainly, and even moderately easy for Perl’ers of a certain experience level.  But why should we limit ourselves unnecessarily?

What Sawyer X points out is that we have the technology: we have perlbrew, and local::lib, and cpanm, and so on and so forth.  We have ways to bundle everything you need to run a given Perl app, from the precise version of Perl to the preceise version of every dependency module, all in one place, without giving a royal shit what version of system Perl someone has, or what version of this or that module they have, or whether they can install modules or not (either because of access issues or inexperience), or any of that.  We have all the pieces.  We just need to put them together.

He also points out that GUGOD even made a first cut at doing just that.  It’s called Seacan, and it’s listed as a “proof of concept.” I took a look at it, and it has some great ideas (which I happily stole), but I couldn’t figure out exactly how it was supposed to work.  I’ll freely admit, though, that I didn’t spend too much time trying to.  Because it wants to create a binary package for distribution, and I don’t think that’s the right way to go about doing it.

Imagine you were writing a program in, let’s say, C++.  You would create yourself a tarball with all the source code, and you’d include a Makefile, and you’d put in some instructions on how to build it, and that’s what you’d distribute ... right?  You wouldn’t try to create a binary distribution.  Because it’s not practical: binary packages have to be compiled for a certain OS and architecture and all that happiness.  And you don’t want to mess with all that.  If people want binary versions of your software, someone will eventually make an RPM and a .deb out of it, and there you go: binary distribution issues solved.  Totally separate from you.  Which is, I think, how it should be.

So I think that, instead of trying to come up with a way for the author to make a binary package, what should we come up with is a standard way to create a build process ... our version of the Makefile.  I mean, it could be an actual Makefile, I suppose ... except that I utterly despise Makefile syntax, and honestly if your language is not compiled it’s rank overkill anyway.  Anything a Makefile can do, Perl can do better anyhow ... hell, even bash can do better.  And of course if we use bash for most of the logic, we don’t have to worry about what version of system Perl people are running, which was one of our concerns in the first place.  This is how you install perlbrew, right?

curl -L http://install.perlbrew.pl | bash

There’s a simple little bash script, and you just download and feed it directly to bash, and then it just ... works.

No reason another bash script couldn’t download your app, install perlbrew, brew up the proper Perl, and install all the exact versions of all the modules you need ... right?  No reason at all, I say.  So, here you go:

#! /bin/bash
set -e
set -o pipefail
 
 
# AUTHOR-CONFIGURABLE OPTIONS
# Author of the package should set these to appropriate values.
# End-user should not touch, unless you really know what you're doing.
 
PERL_VERSION=5.14.4
 
INSTALL_METHOD=download-tar
INSTALL_URL=https://github.com/you/yourproj/archive/latest_release.tar.gz
# --or--
#INSTALL_METHOD=git-clone
#INSTALL_URL=https://github.com/you/yourproj.git
#INSTALL_BRANCH=test
 
# Author should set a default value here:
: ${INSTALL_PACKAGE_TO:=/opt/yourproj}
 
# End user can override like so:
#   curl -kL https://raw.githubusercontent.com/you/yourproj/master/build/install | INSTALL_PACKAGE_TO=~/opt/foo bash
# Or, not:
#   curl -kL https://raw.githubusercontent.com/you/yourproj/master/build/install | bash
 
 
#############
# FUNCTIONS #
#############
 
function die
{
    echo "$@"
    exit 1
}
 
function verify_program
{
    program=$1
    if which $program 2>/dev/null
    then
        return 0
    elif type $program 2>/dev/null
    then
        return 0
    else
        die "cannot locate program $program (or else can't figure how to locate things)"
    fi
}
 
function count_files
(                                                   # parends used advisedly
    dir="$1"
 
    shopt -s nullglob
    local c=("$dir"/*)
    echo ${#c[@]}
)
 
 
#####################
# VERIFY/CREATE DIR #
#####################
 
if [[ $INSTALL_PACKAGE_TO ]]
then
    install_dir=$INSTALL_PACKAGE_TO
else
    install_dir=$(dirname $0)/..
    if [[ ! -d $install_dir/perl5 ]]
    then
        die 'you must set $INSTALL_PACKAGE_TO before calling this script'
    fi
fi
 
if [[ -e $install_dir ]]
then
    if [[ ! -d $install_dir ]]
    then
        die "requested directory $install_dir is not a directory"
    fi
 
    if [[ ! -w $install_dir ]]
    then
        die "requested directory $install_dir is not writeable (re-run with sudo?)"
    fi
else
    parent_dir=$(dirname $install_dir)
    if [[ ! -e $parent_dir ]]
    then
        die "can't make $install_dir because $parent_dir doesn't exist"
    fi
 
    if [[ ! -d $parent_dir ]]
    then
        die "can't make $install_dir because $parent_dir isn't a directory"
    fi
 
    if [[ ! -w $parent_dir ]]
    then
        die "can't make $install_dir because $parent_dir is not writeable (re-run with sudo?)"
    fi
 
    mkdir $install_dir || die "can't make install dir (maybe disk error?)"
    if [[ ! -w $install_dir ]]
    then
        die "made directory $install_dir but now can't write to it (hunh?)"
    fi
fi
 
# need an absolute path
install_dir=$(perl -MCwd=realpath -e 'print realpath(shift)' $install_dir)
 
 
##############################
# INSTALL FILES IF NECESSARY #
##############################
 
if [[ $(count_files $install_dir) == 0 ]]
then
    # install directory appears to be empty
    # let's fill it up
    case $INSTALL_METHOD in
        git-clone)      verify_program git
                        git clone ${INSTALL_BRANCH:+ --branch $INSTALL_BRANCH} $INSTALL_URL $install_dir
                        ;;
        download-tar)   verify_program curl
                        verify_program tar
                        curl -kL $INSTALL_URL | tar xpzf - -C $install_dir
                        # tricky bit: if our directory contains nothing but another directory,
                        # move all the inner dir's files up to the top level and ditch the dir
                        if [[ $(count_files $install_dir) == 1 && -d $(echo $install_dir/*) ]]
                        then
                            (                       # subshell to localize the `shopt`
                                shopt -s dotglob
                                tar_dir=$(echo $install_dir/*)
                                mv $tar_dir/* $install_dir
                                rmdir $tar_dir
                            )
                        fi
                        ;;
        *)              die "unknown install method: $INSTALL_METHOD"
    esac
fi
 
 
#############################
# INSTALL PERL IF NECESSARY #
#############################
 
perlbrew_dir=$install_dir/perl5
unset PERL_CPANM_OPT
unset PERL_LOCAL_LIB_ROOT
unset PERL_MB_OPT
unset PERL_MM_OPT
unset PERL5LIB
export PERLBREW_ROOT=$perlbrew_dir
export PERLBREW_HOME=$PERLBREW_ROOT
unset PERLBREW_PERL
unset PERLBREW_VERSION
unset PERLBREW_CSHRC_VERSION
unset PERLBREW_PATH
unset PERLBREW_MANPATH
 
perlbrew=$perlbrew_dir/bin/perlbrew
if [[ ! -x $perlbrew ]]
then
    curl -kL http://install.perlbrew.pl | bash
fi
source $perlbrew_dir/etc/bashrc
 
# let's just always do this
# that way, we're sure to have the latest version of cpanm
$perlbrew install-cpanm --force
 
if [[ ! -x $perlbrew_dir/perls/perl-$PERL_VERSION/bin/perl ]]
then
    perlbrew install -n $PERL_VERSION
fi
 
 
###########################
# INSTALL/UPGRADE MODULES #
###########################
 
$perlbrew_dir/bin/cpanm -n --cpanfile $install_dir/build/cpanfile --installdeps $install_dir/build/

You would put this into a build/ directory, along with a cpanfile.  Regular readers of The Perl Weekly¹ may recall a blog post on cpanfile from over a year ago.  In the comments, Reini Urban asks “Why should anyone use a 4th format to define dependencies?” To which the proper answer is, “they shouldn’t.” But cpanfile isn’t really for defining dependencies for modules in the way that the other 3 methods Reini cites are.  What cpanfile is good for is this exact situation: a Perl app that will be installed via some method other than the standard CPAN module toolchain.  Plus cpanm supports exact versions, which CPAN module installation doesn’t (yet).  Here’s a simple/stupid example of a cpanfile:

requires 'IPC::System::Simple', '== 1.25';
requires 'Moose', '== 2.1403';
requires 'Path::Class', '== 0.32';
requires 'Const::Fast', '== 0.014';

Your main program, in the bin/ directory, might look something like this:

#! /bin/bash
 
basedir=$(dirname $0)/..
 
export SHELL=/bin/bash                              # have to fool perlbrew a bit
export PERLBREW_ROOT=$basedir/perl5
export PERLBREW_HOME=$PERLBREW_ROOT
source $basedir/perl5/etc/bashrc
 
perl -I$basedir/lib -e '
    use 5.14.0;
    use warnings;
    use autodie ":all";                             # requires adding IPC::System::Simple to cpanfile
 
    use App::YourProj;
 
    App::YourProj->run;
' "$@"

And that’s pretty much all there is to it.  Now, this isn’t perfect, but it almost completely works.²  It could be made even more expansible, of course—the idea of build/ shouldn’t be hardcoded in there, and maybe it should be smart enough to deal with curl not being present and fallback to wget—but this moderately simplistic script already supports two install methods, and doubles as an upgrade script if run from the directory it builds after initial install.  It will only work for Unix-oid OSes, true, but I think it ought to work on all flavors of Linux, BSD, even OSX (probably).³  The only bits of Perl it uses are backcompatible enough to work on anything going back to 5.6 ... or even beyond.  And, once installed, everything just runs self-contained, completely oblivious to any Perl verisons or Perl modules that may or may not be installed in other places.

It still wants more testing, of course, but it seems to work pretty damn well so far.  It can download a tarball from GitHub (or anywhere, really), and you can get GitHub to give you a tarball of any tag you make.  Or it can just clone a GitHub repo and let your user run the app right out of that.  In the latter case, you can also specify which branch you want.⁴  And of course you can specify any Perl version you like.

I wonder if it might be worthwhile to turn this into a CPAN module ... or, better yet, a little Perl app.  I could even use this script to deliver it to Perl application programmers.  Sort of like how a compiler is generally used to compile itself.

Comments certainly welcome. :-)