Test::Builder2 vs CPAN and How You Can Help
Test::Builder2 is now down to two issues.
- The problem of using Mouse
- The problem of backwards compatibility
The former is complicated, but suffice it to say TB2 cannot rely on Mouse or Moose or Moo. It's being solved by writing an OO compiler, something which will generate accessor methods and roles at build time rather than relying on a runtime compiler. This should also solve TB2's less than ideal startup time. It might be Mite or it might be Moo, but the problem is being taken care of.
The second is harder and is what I call "Test::Builder2 vs CPAN". Because Test::Builder has been around or so long, and so much depends on it indirectly, there's a lot of not entirely documented behavior being relied on. We've been using CPAN modules as a broad test suite right along for this reason. TB2 has the potential to seriously break a lot of module's test suites, so it's best to get it as right as possible before stable release.
Andreas, using the incredible analysis.cpantesters.org has provided a list of the top 100 modules which are failing because of TB2. A lot more than I thought at this point honestly. Worse, I was hoping they were mostly failing for a handful of reasons. Unfortunately it seems they're unique.
I'm asking for two ways people can help.
First, if you're the author of a module which is failing because of TB2 I ask you to please report it to TB2 even if the problem turns out to be your own (like if you violated encapsulation) and even if you can fix it on your own. We want to know how Test::Builder is being used.
More importantly, if your CPAN module is failing then probably other software we don't know about will fail. We want to know about these problems and be able to decide what to do about them. So please, report any trouble you have with TB2. Also report any successes you have with TB2. Really just talk to us about how TB2 is effecting you.
Second, I'd like to crowd source some bug analysis. Each failure in the list needs to be examined to see what is failing and then why it is failing before anything can be done about it. It would be a great help if this could be tackled by people taking a handful of them.
For each failing release...
- Make sure nobody's already done it.
- Create a new issue in the TB2 queue
- Put the release in the title, ex: "XML-LibXML-SAX-ChunkParser-0.00004"
- Add the "CPAN Regression" label
- Add a link to one illustrative failing test report
- Mention issue #249 in the description
If all you do is that, that would be a great help! I can't stress this enough.
If you'd like to go further...
- Run the release's tests with Test::More 0.98 and 1.5 to verify the failure.
- Post the relevant bits of the failing test in the issue's comments.
This provides a list of pre-digested issues for people to examine and determine how to fix. If you'd like to go even further you can do some of that analysis yourself.
- Provide an analysis of what caused the failure.
If you really want to be a champ...
- Write a test for TB2 to test the regression.
And if you're bucking for my job...
- Write a patch for TB2 to fix it.
There's 100 releases to go through, a lot for me, but not much for a few dozen people who read this blog and do a few. Here's a good example of what the resulting issue looks like.
Thanks crowd!
Ya know, the guy who tells you to write tests and not use MakeMaker.
I recently had to update two of my modules to deal with TB2. I was alerted to this by a CPAN smoker using it, which was helpful...
You can see what I was doing in TB1, compared to what I do now. The module sniffs the TB version:
https://github.com/sheriff/test-more-prefix-perl/commit/b521356294a862e0c1fd3d2d59765beaae5c555c
It's hard to isolate what I changed in Test::BDD::Cucumber, and I can't easily find it, although I'm sure I had to stop violating TB's encapsulation...
(Github for some reason isn't accepting comments on that change)
_print_commentis gone from TB2, that was likely the TB1 vs TB2 change which broke things.Work at
done_testingtime is better done by adding an EventHandler for the TestEnd event. This will guarantee your module will work even with non Test::Builder based modules.As for adding a message to the log role, I'm not sure why you're doing that. You may simply want to make, issue and handle your own event.
OIC what's going on now with the Log event. That will work, but the better way to handle that is to put in an early_handler which modifies Log events as they go by.