sirhc

Recently I had the pleasure of spending three hours debugging an obscure bug. An obscure bug I caused by introducing a newline. That little punk, 0x0A.

I released a new version of a command line program. For me, it’s an elegant piece of work, combining a marvelously complex-but-intuitive configuration for system administrators with an absolutely simple interface for users. To use the command, the user runs it with a couple of arguments and it prints out a single line of useful text derived from that marvelously complex configuration.

But, it doesn’t print a newline.

It’s never printed a newline. The original author didn’t include one for some reason. Anyone who has ever encountered a command like this knows well my irritation.

my awesome prompt> some_lame_command
my awesome prompt>e answer

Argh!

The workaround I’ve seen used, after seeing the above is to face-palm, then run the command again, only differently.

my awesome prompt> echo `some_lame_command`
42 is obviously the answer
my awesome prompt>

I’m embarrassed when users see behavior like this from a program I wrote. Being the arrogant bastard programmer that I am, I decided to fix this. Since all commands print newlines, everyone should already be assuming that this one does too and should already be handling it in the proper manner. Right? When writing a shell script, the distinction between newline-printing and non-newline-printing commands is irrelevant. In either Bourne shell:

FROBBED=`frobnosticate`

Or C shell:

setenv FROBBED `frobnosticate`

The shell is benevolent enough to remove the newline, if it exists. After all, this is the most commonly desired behavior when assigning command output to a variable. However, things are a bit different when switching to a so-called real programming language, like Perl:

$ENV{FROBBED} = `frobnosticate`;  # Caution, newline ahead!

Sure, it looks more or less the same, but veteran Perl programmers will immediately grimace when reading the above. Unlike the shell, Perl, like other programming languages, will preserve the output of the command verbatim. In this case, preserving data and letting the programmer decide how to use it is the most commonly desired behavior. Since everything coming from an external command ends with a newline, the environment variable being set in this case will have a newline. This will almost always cause a problem. One that, as I’ve learned, is not always easy to find. Since stripping input of newlines is just as common as the desire to preserve data, Perl makes this easy and most Perl programmers will habitually write it.

chomp( $ENV{FROBBED} = `frobnosticate` );

Now it doesn’t matter if the command prints a newline or not, the chomp builtin has your back. It’s just like being in the warm embrace of the shell, only with a little extra syntax.

So it turns out that one of the engineering groups I support was using a Perl script that set an environment variable as in the first example. The value of this environment variable was then being passed off to the batch system and used by an engineering program as a network address to connect to. Of course, the program made the fatal mistake of trusting user input and, in a spectacular fashion, failed to connect to the server whose name just happened to contain a newline.

After chasing down a couple of red herrings which left me flummoxed, one of the affected users shared with me an error log and the script that generated it. There, in all its syntax highlighted, monospaced glory was the environment variable being set without any attempt made to trim off the newline. I immediately swallowed my pride and released an updated version of the command reverting the newline behavior and the problem went away. My engineers—at least, the subset using this particular script in this particular way—could once again get their work done.

By far, this isn’t the worst thing I’ve done to our batch system. One time I caused all jobs executing on Solaris hosts to immediately fail. Whoops.

Anyway, what’s the lesson to be learned from today’s experience?

Never—and I’ll repeat that, never—assume everyone will be doing the right thing (remember what they say when you assume something). Inevitably, someone won’t be.

There’s a corollary to today’s lesson. When coming across something that could be improved with a small change, don’t. Seriously, just don’t. Inevitably, someone will be depending on the current behavior, no matter how right or wrong it may seem. This is why the phrase “bug compatible” exists in the lexicon.