Ovid

If you've ever used git bisect, you know what an incredibly useful tool this is. It allows you to do a binary search through commits to find out which commit caused a particular error. Many people seem unaware of git bisect run ... which automates this even further, but it has a limitation: it won't let you find a particular error, it detects success or failure, that's all. So I decided to do something about that.

But first, let's have a quick review of git bisect for those who don't know about it.

I use it a lot with testing. I'll fetch my changes from the master branch and run the test suite. If a test fails, I then do this from the root directory of my repository (that's required by git bisect):

git bisect start 
git bisect bad  
git checkout HEAD~20 
prove ...
git bisect good

Those translate to:

• Start bisecting
• Tell git this is a bad commit
• Go far enough back in history to find a commit where the test passes
• Rerun the test to make sure the test passes (very important)
• Tell git that this is a good commit

At this point, git will automatically check out a commit halfway between the two commits. You rerun your test and if it passes, you type git bisect good. If it fails, you type git bisect bad. (There's a whole lot more that I'm not covering here). Even if you don't have a test to run, you can use git bisect and, for example, refresh your browser to look for that display bug you're trying to track down and then return to the command line and mark it good or bad as appropriate.

Repeat this process a few times and eventually git bisect will tell you which commit first introduced the failure. Then you type git bisect reset to stop bisecting.

However, I hate effectively typing this:

git bisect good
prove t/test/that/fails
git bisect bad
prove t/test/that/fails
git bisect bad
prove t/test/that/fails
git bisect bad
prove t/test/that/fails
git bisect bad
prove t/test/that/fails
git bisect bad
prove t/test/that/fails
git bisect good
prove t/test/that/fails
git bisect good
prove t/test/that/fails
git bisect bad

No matter how wonderful git bisect is, I hate the boring repetition. However, you can automate this away, too. After you start your bisect and mark your starting and ending commits as good and bad, you can then do this:

git bisect run prove t/test/that/fails

And git bisect will happily run the test for you and mark commits good or bad as appropriate, using the exit code of the program. You just sit back and wait for it to finish. Much nicer.

There are a couple of cases where this doesn't work, though. One case, obviously, is where you must manually verify good/bad and can't write a program to do this. Another case is where sometimes the test fails in different ways. When you do this manually, you can use git bisect skip to skip a commit (or various commits). However, this means falling back to the tedious manual usage of git bisect. Thus, I hacked together the following proof of concept:

#!/usr/bin/env perl

use strict;
use warnings;
use Getopt::Long;
use IPC::Open3;
use Symbol qw(gensym);

GetOptions(
    'contains=s' => \my $contains,
    'matches=s'  => \my $matches,
) or usage();

usage() unless $contains xor $matches;
usage() unless @ARGV;

my $command = join ' ' => @ARGV;

my $stderr = gensym;
my $pid = open3( gensym, ">&STDERR", $stderr, $command );

my $output = '';
while ( my $err = <$stderr> ) {
    print STDERR $err;
    $output .= $err;
}

my $failed = $matches
  ? ( $output =~ qr/$matches/ )
  : ( index( $output, $contains ) > -1 );

waitpid( $pid, 0 );

exit $failed;

sub usage {
    die <<"END";
Usage:

$0 --contains 'exact text to fail on' command to run
$0 --matches  'pattern to fail on'    command to run
END
}

I dropped that into my path and named it fail_if. You pass it a string of text to look for (--contains for an exact match or --matches for a regex match). If that text is found in the programs STDERR, the fail_if exits with a non-zero exit status. Otherwise, it exits with 0 (success). You use it like this:

git bisect run fail_if --contains "Field 'user' doesn't have a default value" \
    prove t/path/to/test/that/sometimes/fails/in/different/ways.t

And now, git bisect will consider a commit a failure if that particular string is found in the output. So far it's allowed me to track down some fairly messy problems.

It probably can use a whole bunch of TLC (such as automatically skipping commits that fail in other ways). Suggestions welcome.