Test-Simple, Test-More, and Test-Builder. What is breaking? Should I be scared?
Whats breaking? Much less than you might expect, but more than I had hoped.
Should I be scared? If you do not test your things now, and file reports for me to fix bugs, then yes, you should be VERY scared.
That about sums it up! (Just kidding.) But in all seriousness, PLEASE TEST YOUR CODE!. Please install the trial releases of Test-Simple, try building your dependency chains against it! Please also try your non-cpan code against it. There has been plenty of warning, and you still have almost 3 months to file bug reports.
What breaks?
Code that does any of the following is no longer supported:
- Replacing the $Test::Builder::Test singleton
- Monkey-patching Test::Builder (Note: Exceptions for some commonly monkey-patched subs)
- Directly modifying values in the Test::Builder instance hash
There are a few monkey-patching exceptions, if your code wrapped 'ok', 'diag', 'done_testing' or a couple other common ones, some fancy hackery has been done to allow them to keep working.
Backstory:
Early last year I adopted Test-Simple from Schwern. This includes Test::Simple, Test::More, and Test::Builder. Essentially this is the foundation that nearly all testing in perl and cpan is built upon. Initially my plan was to simply run maintenance and not shake things up. My plan failed.
Shortly after adopting the module I was asked by David Golden to apply a seemingly innocent patch. This patch changed the indentation of a comment intended for humans reading test results. Sounds simple enough right? WRONG. This single change caused a cascade of failures that ultimately would prevent most Moose extensions from installing. The root cause of this was the horrible state of testing modules intended to make testing easier.
This issue caused me to engage in a deep study of Test::Builder and tools that used it. Test::Builder was of course an amazing bit of engineering, it is no wonder the entire ecosystem is built on it. Schwern and the other contributors deserve a giant ++. That said, I reached the same conclusion most of them had a year or 2 earlier... Due to cruft and some unfortunate design decisions, most of the internals would need some form of rewrite to really solve the big problems.
My decision to undergo a major change was the second such decision. The first one was the Test::Builder/More 2.0 project. The 2.0 project was a complete rewrite from scratch, with the intentions of then updating the old tools later to use the new stuff. This project saw a lot of good development, but ultimately petered out. The way it was explained to me is this, the 2.0 project blocked Test::Builder development. Everyone devoted attention to the new stuff, which ultimately didn't happen, and nobody gave any love to what we already had.
I decided to take a very different approach. I was not going to write things completely new, or from scratch. I was going to refactor things, preserving compatibility wherever possible. Some design decisions needed to be altered, and that has resulted in some incompatibility, but these are minimal, and have effected relatively few modules, many of which have been fixed already.
Cool story bro, but what else do I REALLY need to know?
- The Test::Builder singleton is no longer the central hub
- Subtests were the root of all evil, they have been re-implemented
- The ability to test your testing tools was the biggest focus
- Most things that previously required monkey-patching or other hacks can be done via an API now
Here are some docs for anyone who actually wants to see what is new:
I blog about Perl.
Thanks for your work on Test::Simple, Chad. A somewhat unrelated question from a Test::More user: now after the refactoring, is it easier to fail early (after the first failure) from a test file? From what I remember, the old design of Test::Builder makes it hard to support this feature.
Of course you can do something like this [1]:
But I was thinking of a flag or attribute to enable this generally to an existing unmodified test script.
[1] http://stackoverflow.com/questions/16674839/perl-testing-how-to-stop-testing-in-a-single-file-after-first-failure
perlancar. It is in fact easier, but still not 100%. If you are talking purely Test::More, or Test::Simple then yes, it is easy. However extra tools that still use Test::Builder, instead of the new internals (Test::Stream) the behavior is lack a key feature that makes this possible.
With the Test::Stream internals everything is an event. Further Diagnostics can be attached to 'Ok' events. You also get a listener system that lets you provide callbacks. Using this system you can watch for a failed Ok event and end the test by whatever means you want. For Test::More and updated tools this is fine. However for tools that still produce the OK and the Diag separately you have the problem that bailing will prevent the diagnostics from being seen.
So the answer is yes, the new internals solve this problem, but you have to wait for the rest of the ecosystem to catch up :-/