Reini Urban

After too many failing cpanreports for perl-compiler releases I'll finally unify my failing test results into manable code of todo logic. Automatically.

Test::TAP::Unify at http://gist.github.com/506341

NAME

Test::TAP::Unify - Create a minimal set of TODO code for a set of test results

SYNOPSIS

rm log.test-*
os=$(uname -o)
for p in 5.10.1 5.10.1-nt 5.12.1 5.12.1d 5.12.1d-nt 5.13.3 \
            5.13.3d 5.13.3-nt
do
  perl$p Makefile.PL
  make test >log.test-$os-$p
done

use Test::TAP::Unify;
# take features from test filenames or test comments
my $features = {debugging => qr/\dd/,
      threads   => qr/-nt/,
                os        => qr/log.test-(\w+)-/,
                version   => qr/(5\.[\d\.]+/)};
my $t = Test::TAP::Unify->new(filename => $features,
                              comment  => $name);
my $u = $t->parse(<log.test-*>);

# prints the complete test matrix
print $u->matrix;

# prints @TODO lists as test.pl code snippet
print $u->todo->code('PUSH', 'index'); # as PUSH'ed lists of test indices

my $m = Test::TAP::Unify->new
  ( filenames => {debugging => qr/\dd/,
                  threads   => qr/-nt/,
                  os        => qr/log.modules-(\w+)-/,
                  version   => qr/(5\.[\d\.]+/)},
   comment  => {module => qr/(pass|fail) (\S+)/})
  ->parse(<log.modules*>);
print $m->matrix;             # all results
print $m->todo->code('IF', 'module');  # as IF checked lists for module lists

DESCRIPTION

You hate failing tests, you get a set of test results per perl version by yourself, and also gathered by cpantesters. So you can mark all failing tests as TODO check tickets, and release a new version, where all tests pass. Great! Error free software.

Really, for more complicated XS modules with some randomly failing tests you need to keep TODO lists uptodate. Just to see regressions.

This modules was written to unify test results of the perl compiler B::C, you will get tired to mark TODO tests manually after each feature change or added platform or perl version you are testing against. It will give you insights which feature might cause the damage. You can e.g. mark tests with any feature name and check them with this module.

Test::TAP::Unify reads the output of make test (TAP results), detects features by various methods, e.g. the filename where make test is logged to, or the test comments and tries to find the minmal set of failing tests per feature. The shortest set of feature-fail combination. Since you have an arbitrary long list of features - typically: version, threads, os, debugging, cc - producing this set is not trivial.

The sets can be printed as matrix, or as code, lists of @TODO with added feature logic. Each test is indexed by number 1..plan.

Sample output for todo->code('PUSH', 'index')

sub is_todo {
  my @TODO;
  @TODO = (15,39,44);
  @TODO = (39)           if !$ITHREADS;
  @TODO = (15,41..45)    if $version < 5.007;
  @TODO = (39,41,44)     if $version >= 5.010;
  @TODO = (15,29,39,44)  if $version >= 5.010 and !$ITHREADS;
  push @TODO, (27)       if $version >= 5.012 and $ITHREADS;
  push @TODO, (6,8..10,16,21,23,24,26,30,31,35) if $version >= 5.013002;
  push @TODO, (15,25,42..43) if $version >= 5.013 and $ITHREADS;
  return @TODO;
}

Sample output for todo->code('IF', 'module')

sub is_todo {
  my ($module, $version, $name, $threads) = @_;
  foreach (qw(ExtUtils::MakeMaker LWP Attribute::Handlers MooseX::Types)) {
    return 'generally' if $_ eq $module;
  }
  if ($] < 5.007) {
    foreach(qw( ExtUtils::CBuilder Sub::Name)) {
      return '< 5.007' if $_ eq $module;
    }
  }
  if ($] < 5.010) {
    foreach(qw(B::Hooks::EndOfScope)) {
      return '< 5.010' if $_ eq $module;
    }
  }
  if ($threads) {
    foreach(qw(
               File::Temp ExtUtils::Install
               Test::Tester Attribute::Handlers
               Test::Deep FCGI B::Hooks::EndOfScope Digest::SHA1
               namespace::clean DateTime::Locale DateTime
               Template::Stash
              )) {
      return 'with threads' if $_ eq $module;
    }
    if ($] >= 5.010 and $] < 5.012) {
      foreach(qw(
                 Class::Accessor Class::MOP
                )) {
        return '5.10 with threads' if $_ eq $module;
      }
    }
  } else {
    if ($] >= 5.010 and $] < 5.012) {
      foreach(qw(
                 IO ExtUtils::Install Test::Tester Test::Deep Path::Class
         Scalar::Util
                )) {
        return '5.10 without threads' if $_ eq $module;
      }
    }
  }
}

SEE ALSO

• TAP::Harness
• Test::Harness::Straps
• B::C

UPDATE

I've got a basic version of this idea running. Just the simple optimization to find the shortest combination not yet.

Code: http://code.google.com/p/perl-compiler/source/detail?r=587

Sample

$ ./status_upd -q t/reports/1.28/r581

$common_fail = {
                 't/issue39.t' => '1',
                 't/issue29.t' => '2',
                 't/z_portfs.t' => '1',
                 't/issue37.t' => '1'
               };
$v5.10.1 fail = {
                  't/modules.t' => '37, 39, 149, 151, 373, 375'
                };
$v5.12.1 fail = {
                  't/c_o3.t' => '12, 28, 35',
                  't/c_o4.t' => '28, 35',
                  't/cc_o2.t' => '35',
                  't/c_o1.t' => '35',
                  't/cc_o1.t' => '35',
                  't/cc.t' => '35',
                  't/c_o2.t' => '12, 14, 17-20, 22-23, 28, 34-38, 40',
                  't/c.t' => '35'
                };
$v5.13.4 fail = {
                  't/c_o3.t' => '29',
                  't/c_o4.t' => '29',
                  't/cc_o2.t' => '12, 28',
                  't/c_o1.t' => '28',
                  't/cc_o1.t' => '12, 28',
                  't/cc.t' => '12, 28',
                  't/c_o2.t' => '29',
                  't/c.t' => '28'
                };
$v5.6.2 fail = {
                 't/bytecode.t' => '27, 46',
                 't/c_o3.t' => '35, 46',
                 't/c_o4.t' => '35, 46',
                 't/cc_o2.t' => '27, 35, 46',
                 't/c_o1.t' => '35, 46',
                 't/cc_o1.t' => '35, 46',
                 't/cc.t' => '35, 46',
                 't/modules.t' => '13, 15, 21, 23, 45, 47, 61, 63, 73, 75',
                 't/c_o2.t' => '35, 46',
                 't/c.t' => '35, 46'
               };
$feature d fail = {
                    't/bytecode.t' => '3-4, 10, 28-29, 31, 41-43, 45-46',
                    't/c_o3.t' => '29',
                    't/c_o4.t' => '29',
                    't/cc_o2.t' => '12, 28',
                    't/c_o1.t' => '28',
                    't/cc_o1.t' => '12, 28',
                    't/cc.t' => '12, 28',
                    't/c_o2.t' => '29',
                    't/c.t' => '28'
                  };
$feature d-nt fail = {
                       't/bytecode.t' => '3-4, 10, 28-29, 31, 45-46',
                       't/cc_o2.t' => '12, 28',
                       't/c_o1.t' => '28',
                       't/cc_o1.t' => '12, 28',
                       't/cc.t' => '12, 28',
                       't/c.t' => '28'
                     };
$feature nt fail = {
                     't/bytecode.t' => '3-4, 28-29, 31, 45-46',
                     't/c_o3.t' => '12, 28, 35',
                     't/c_o4.t' => '28, 35',
                     't/c_o1.t' => '35',
                     't/c_o2.t' => '12, 14, 17-20, 22-23, 28, 34-38, 40',
                     't/c.t' => '35'
                   };
$common_todo = {
                 't/c_o3.t' => '11, 23',
                 't/issue34.t' => '2',
                 't/cc_last.t' => '1, 3',
                 't/c_o4.t' => '11, 23, 27',
                 't/issue39.t' => '1',
                 't/issue27.t' => '1',
                 't/c_o2.t' => '11, 23'
               };
$v5.10.1 todo = {
                  't/modules.t' => '13-14, 16, 25-32, 41-43, 45-47, 73-74, 77-79'
                };
$v5.12.1 todo = {
                  't/modules.t' => '13-14, 16, 41-43, 77-79, 93-95, 133-135'
                };
$feature  todo = {
                   't/c_o1.t' => '23, 25, 27, 42-43',
                   't/modules.t' => '13-14, 16, 21-24, 33-36, 38, 42, 46, 54',
                   't/c.t' => '23, 25, 27, 42-43'
                 };
$feature d todo = {
                    't/modules.t' => '13-14, 16, 21-24, 33-36, 38, 42, 46, 54'
                  };
$feature d-nt todo = {
                       't/modules.t' => '13-14, 16, 21-24, 33-47, 53-56, 61-62, 64-68'
                     };
$feature nt todo = {
                     't/c_o1.t' => '23, 29',
                     't/c.t' => '23, 29'
                   };