kid51

Until the past year I never truly appreciated how much faster make will build an executable when you are running on some heavy iron and use the -j option. As a p5p committer I have a shell account on dromedary, a server donated by booking.com++ and maintained by Dennis Kaarsemaker and friends (more ++). On that box I set TEST_JOBS=8, which enables me to configure, build and test perl so fast that I have never bothered to time it (probably less than 5 minutes).

Today, however, was the first time that I learned that running make -j${TEST_JOBS} can obscure the results of make.

I had created a branch where, as an experiment, I moved Pod-Html from ext/ to dist/. I grepped the source code for all instances of ext/Pod-Html and changed them to dist/Pod-Html. I then ran a shell function I use all the time to test a branch:

sh ./Configure -des -Dusedevel && make -j${TEST_JOBS} && \
    TEST_JOBS=${TEST_JOBS} make -j${TEST_JOBS} test_harness

I noticed something peculiar: The make part of this command was terminating prematurely, but with no error output at the end where I would have expected it.

Processing BidiBrackets.txt
Finishing processing Unicode properties
Compiling Perl properties
Creating Perl synonyms
Writing tables
Making pod file
Making test script
Updating 'mktables.lst'

I didn't think much of this at first. I typed make and make resumed and proceeded to its normal completion, where it instructs you to run make test. And, after debugging some unrelated issues, everything in make test PASSed.

But I was puzzled about this and repeated the process one step at a time, only running make under script so that I could capture the build output for easier study. I grepped the transcript for Pod-Html and, to my surprise, found one place where there was still a mention of ext/Pod-Html. This occurred about 90% of the way through `make':

Extracting pod2html (with variable substitutions)
pod2html.PL: cannot find '../ext/Pod-Html/bin/pod2html'
make[1]: *** [pod2html] Error 2
make[1]: Leaving directory '/home/jkeenan/perl/utils'
make: *** [utilities] Error 2
make: *** Waiting for unfinished jobs....

Those "unfinished jobs" ran, but when they completed, make exited with non-zero status, which meant that make test did not automatically start. But, because of -j${TEST_JOBS}, the error output was not to be found at the end of make's output.

When, however, I re-ran make with no -j option, make failed as soon as it tried to build pod2html, so the error output appeared at the end of the transcript:

../miniperl -I../lib pod2html.PL
Extracting pod2html (with variable substitutions)
pod2html.PL: cannot find '../ext/Pod-Html/bin/pod2html'
make[1]: *** [pod2html] Error 2
make[1]: Leaving directory /home/jkeenan/perl/utils'
make: *** [utilities] Error 2

It turned out that pod2html.PL creates the string ../ext/Pod-Html/bin/pod2html in part by joining the elements of the list qw(ext Pod-Html bin) with OS-appropriate path separators. Once I changed ext to dist in this location, make -j${TEST_JOBS} ran as expected and all tests PASSed.

Moral of the story: If you customarily build an executable with make -j and make ends prematurely, you either have to search earlier in the make log for the error or you have to re-try make without a -j value.