Configure at the 2017 Perl 5 Core Hackathon

By kid51 on

Configure at the Perl 5 Core Hackathon

One major focus of discussion at the Perl 5 Core Hackathon in Amsterdam last month was the status of the program Configure. In this post, we provide a brief introduction to Configure and then discuss the work done on it at the Hackathon and in the subsequent weeks.

What Does Configure Do?

In order to build and install an executable program on a machine, the programmer first has to identify characteristics of that machine: the CPU; the operating system; the libraries; and so forth. Where the machine offers the programmer choices -- e.g., which C-compiler do you want to use -- those choices have to be recorded in a way they can be used by the build and installation procedures.

A configuration program is one which performs these tasks. When building a program from source code, the programmer will typically start with something like this:

    $> ./configure --config-option-1 --config-option-2=foo

The configuration program will write one or more output files to disk. In almost all cases one of those output files will be a Makefile. The programmer will then issue a sequence of commands like this:

    $> make
    $> make test
    $> make install

Perl 5 follows this pattern with only one exception: Our configuration program is spelled upper-case-first: Configure. As Perl 5 has evolved over the years, the outputs from configuration have grown to more than a dozen files, not just a Makefile. Here is a list of files typically created by running Configure:

    bitcount.h
    cflags
    config.h
    config.sh
    generate_uudmap
    generate_uudmap.o
    makedepend
    makefile
    Makefile
    makefile.old
    mg_data.h
    myconfig
    opmini.c
    perlmini.c
    Policy.sh
    runtests
    uudmap.h

Invocations of Configure can take many different command-line switches. When working on blead -- the "master" branch in the Perl 5 repository -- you could simply call:

    $> ./Configure -des -Dusedevel

But suppose you wanted a threaded perl? Then you might call:

    $> ./Configure -des -Dusedevel -Dusethreads

Or suppose you wanted to build a perl on FreeBSD which closely matched the characteristics of the "system" perl on that platform. In that case you might call:

    $> ./Configure -des -Dusedevel \
            -Duseithreads \
            -Doptimize="-O2 -pipe -fstack-protector -fno-strict-aliasing"

Perl 5's configuration system provides users with many different options, so fine-tuning of a perl executable to a user's needs is quite possible.

Perl 5 Configure's History

Suppose you wanted to study Configure's history. You would start by calling:

    git log --reverse -- Configure

... and you would find that Configure entered the Perl source code here:

    commit 8d063cd8450e59ea1c611a2f4f5a21059a2804f1
    Author: Larry Wall <lwall@jpl-devvax.jpl.nasa.gov>
    Date:   Fri Dec 18 00:00:00 1987 +0000


        a "replacement" for awk and sed

In other words, Configure came into the world with Perl itself. As a matter of fact, if we examine that commit more closely:

    $> git show 8d063cd8450e59ea1c611a2f4f5a21059a2804f1

... we see that Configure is the very first file whose "diff" is displayed with the above command.

When we examine that commit further, we learn about some aspects of Configure which hold true to this day:

  • Configure is a shell script, optimized for Unix-like platforms, which needs modification to run on non-Unix systems. In the original commit, Larry Wall commented:

        # If you are trying to port this package to a machine without sh, I would
    # suggest you cut out the prototypical config.h from the end of Configure
    # and edit it to reflect your system.


  • There is no human who is the real author of Configure. Configure had 1279 lines of code in its original version. Today, in Perl 5 blead, it has nearly 25,000 lines of code. No human could do this. Instead, Configure, from its inception, has been generated as the output of an entirely separate program called metaconfig. Larry comments:

        # (Note: this Configure script was generated automatically.  Rather than
    # working with this copy of Configure, you may wish to get metaconfig.)


(We'll have much more to say about metaconfig in a moment.)

Over the years more than 130 individuals have contributed to Configure's evolution in more than 1300 individual commits. Here are the people whom git log says have contributed at least 10 commits:

     38 Aaron Crane 
     11 Andy Broad 
     78 Andy Dougherty 
     38 Brian Fraser 
     34 Gurusamy Sarathy 
    179 H.Merijn Brand 
    579 Jarkko Hietaniemi 
     11 Jess Robinson 
     31 Larry Wall 
     65 Nicholas Clark 
     31 Perl 5 Porters 
     16 Yitzchak Scott-Thoennes

However, if we exclude from the list above people who have served as Perl 5 pumpkings, we might surmise that the number of people with intimate knowledge of Configure who are currently active in Perl 5 development is very small. Indeed, one of the objectives of work on Configure at the Perl 5 Core Hackathon was to raise the low bus number of people with expertise in that program.

What Is metaconfig?

As mentioned above, Configure is too long a shell script to have been written by a single human. Where, then, did it come from?

There is a class of computer programs whose purpose is to compose configuration programs. The best known and most widely used such program these days is GNU autoconf (https://www.gnu.org/software/autoconf/autoconf.html; https://en.wikipedia.org/wiki/Autoconf). autoconf dates to 1991, so Perl pre-dates it. Perl uses an older configuration program generator known as metaconfig which was written by Larry Wall as he was first developing Perl circa 1987. However, Larry turned over maintenance and development of metaconfig to others soon thereafter. Since 1991 it has been maintained by Raphael Manfredi and is currently available on github (https://github.com/rmanfredi/dist). (We should, however, distinguish between that repository and the repository in which Perl's flavor of metaconfig is kept: https://github.com/perl5-metaconfig/.)

metaconfig is, in essence, a collection of programs written in Perl 4 which launch a series of probes into the hardware and software available on a given machine and then use the results of those probes to write the Configure shell script itself. As Configure maintainer H.Merijn Brand ("Tux") has put it, Configure is built out of over 900 (meta)units, that automatically get selected based on what the current Perl source tree looks like; it just selects what it needs. The process of picking the units and ordering them in the correct sequence is what the metaconfig process does.

metaconfig and Configure at Start of Perl 5 Core Hackathon

Going into the October 2017 Perl 5 Core Hackathon, Perl's metaconfig repository was showing signs of neglect. That repository was found alongside Perl 5's repository at perl.org, but it is likely that most Perl developers never even noticed its presence. In the twelve months preceding the hackathon, only one person -- Tux -- had made commits to that repository. The layout of files in the repository was, to put it mildly, not self-evident. Why, one could have asked, were we bundling a copy of Manfredi's dist distribution within ours? The instructions in the README were confusing, if not outright misleading.

But it was not just the metaconfig repository that was little understood; Configure itself was also little understood. Over an eighteen year period this author has attended dozens of Perl conferences and workshops and well over a hundred Perl usergroup meetings -- and never once did he hear a presentation on Configure. Truly, the bus factor for Configure was hovering just above 1.

Fortunately, the Core Hackathon attracted participants who, in various ways, wanted something better out of Perl's configuration system and were ready, willing and able to put in the work to make such improvements happen. For example, the Perl maintainers for Debian Linux, Niko Tyni and Dominic Hargreaves, were present to describe the headaches which Configure poses for them as packagers. Karl Williamson, who has spearheaded much of Perl 5's work on Unicode and locales in recent years, identified specific libraries and system calls for Perl's use in those areas for which Configure was not yet probing. Tux, having dealt so much with Configure over many years, wanted to increase the bus factor and prepare the way for long-term change.

metaconfig and Configure at During and After Perl 5 Core Hackathon

The biggest accomplishment with respect to Configure at the hackathon was the transfer of Perl's metaconfig repository from perl.org to github.com. It can now be found at https://github.com/perl5-metaconfig/metaconfig. Tux moved the repository from perl.org to github, then Dennis Kaarsemaker did the blocking work on the old repository and moved it to new hardware. The old repository will not be used for pushing; Dennis transformed it into a read-only mirror of the github repo.

This enabled members of the new perl5-metaconfig github organization to get to work immediately. Over the four-day course of the hackathon there were 35 commits made to the repository. In the fourteen subsequent days 52 additional commits were made, for a total of 87 commits by eight different authors -- seven more authors than this repository had had in the previous twelve months. In the same period of time there have been 37 commits to Configure in the Perl 5 repository.

So what specifically has been accomplished? In the metaconfig repository:

  • The README and subsidiary READMEs were updated -- though more remains to be done.


  • 51 revisions to configuration units or introduction of new configuration units:

        dist/U/filexp_path.U
    dist/U/filexp.U
    U/compline/d_closedir.U
    U/compline/d_gconvert.U
    U/compline/i_time.U
    U/compline/nblock_io.U
    U/modified/cf_who.U
    U/modified/Cppsym.U
    U/modified/d_dbl_dig.U
    U/modified/d_longdbl.U
    U/modified/d_msg.U
    U/modified/d_sem.U
    U/modified/d_strerror.U
    U/modified/d_volatile.U
    U/modified/Head.U
    U/modified/i_stdlib.U
    U/modified/Protochk.U
    U/modified/prototype.U
    U/modified/vaproto.U
    U/perl/d_builtin_overflow.U
    U/perl/d_c99_variadic.U
    U/perl/d_dirfd.U
    U/perl/d_fpclassify.U
    U/perl/d_isfinite.U
    U/perl/d_isinf.U
    U/perl/d_isless.U
    U/perl/d_isnan.U
    U/perl/d_isnormal.U
    U/perl/d_ldbl_dig.U
    U/perl/d_libm_lib_version.U
    U/perl/d_memrchr.U
    U/perl/d_modfl.U
    U/perl/d_signbit.U
    U/perl/d_sprintf_len.U
    U/perl/d_strnlen.U
    U/perl/infnan.U
    U/perl/issymlink.U
    U/perl/i_wchar.U
    U/perl/longdblfio.U
    U/perl/mantbits.U
    U/perl/need_va_copy.U
    U/perl/perlxv.U
    U/perl/time_size.U
    U/threads/d_localeconv_l.U
    U/threads/d_localtime_r.U
    U/threads/d_mbrlen.U
    U/threads/d_mbrtowc.U
    U/threads/d_nl_langinfo_l.U
    U/threads/d_strerror_r.U
    U/threads/d_strtod_l.U
    U/threads/d_strtold_l.U


  • Revisions to 15 executable programs within the distribution

        U/ln-all.pl
    U/mkgloss.pl
    bin/diff2unit.pl
    bin/dual.pl
    bin/find_config_h_deps.pl
    bin/jmake
    bin/mconfig
    bin/metaconfig
    bin/metagrep
    bin/metalint
    bin/metaxref
    bin/mlint
    bin/patch2p4
    ls-diff.pl
    make-dist.pl


  • Revisions to glossaries and other reference files


Sponsors

The Perl 5 Core Hackathon could not have happened without generous contributions from the following companies:

Thanks also to Tux for help in the preparation of this post.

Leave a comment

About kid51

user-pic kid51 is the IRC handle for James E Keenan, New York City-based Perl aficionado, CPAN contributor (id: JKEENAN), user group organizer and frequent conference speaker

More info »