Chad 'Exodist' Granum

*** UPDATE ***

After this was written I received feedback from several respected members of the community alerting me to the problems that could be caused by is() *guessing* if it should be comparing numbers or strings. After hearing this feedback I agreed that the behavior constituted a bug, and one serious enough to alter the behavior post release. Test::Stream was marked *stable* recently enough that the change should not impact very many people, if in fact any.

The latest version of Test::Stream on CPAN no longer guesses if it is given a number vs a string.

In the very near future cmp_ok, and a 'Classic' bundle which provides just the functionality of Test::More, including classic 'is' and 'is_deeply' will be released, and will be the recommended bundle for people moving from Test::More. Branches and pull requests for both of these have been written, I am giving them some time for review before proceeding with their release.

'use Test::Stream;' without any import arguments throws an exception. There are no defaults. You, the user of the tool, must make a choice. This is achieved through the loader/plugin/bundle system, partly influenced by Dist::Zilla.

• The Test::Stream module is just a loader.
• The loader is sugar, you can load plugins directly if you don't like it.
• Test::Stream ultimately loads plugins.
• The Test::Stream::Plugin::Core plugin contains the basic tools that are unlikely to cause dispute (ok, done_testing, etc)
• The loader can pass configuration options into the plugins.
• Well made plugins let you rename subs that you import.
• You can write bundles for the loader that encapsulate lists of plugins along with custom configurations.
• Test::Stream has a 'recommended' bundle (the V1 bundle). This bundle is sugar intended for newcomers, not experienced users.
• I expect some very prominent community members to write and advertise their own bundles.
• If the community produces bundles that stand out above others I am not opposed to including them into the Test::Stream dist.
• If the recommended bundle is found to be harmful it can be replaced (-V2) without breaking any back-compat, we just change the recommendation.

The recommended bundle has these ideas in mind:

1. If people keep tripping over it, that is a design flaw.
2. Things should be as intuitive as possible to new people.
3. 'Test::More' did it that way is not a good argument for anyone new.
4. It should have what is needed to provide examples that teach people the right way to do something people usually do wrong when they first try.

Very few people will complain about the first 2 ideas. The third idea is likely sending chills down some spines. The fourth idea will be explained later. One of the first things I did was look at what I saw as common pitfalls. These are things that often get questions on irc, reported as bugs, etc. Here are the ones that have caused the most commentary:

• is() stringifying things is non-obvious. Why does 1 != 1.0, oh, because these are strings. Long time test people respond "Of course, that is how 'is' works, that is how it has always worked.
• cmp_ok has resulted in tests that incorrectly pass on several occasions, including one from a bug that went undetected for close to 10 years. People often cite "Only perl can parse perl" nearly every issues with cmp_ok also boils down to this.

I was surprised at the backlash against not having 'cmp_ok'. The arguments I have received here have caused me to reconsider, I will likely be adding cmp_ok to Test::Stream. If someone beats me to it, and their implementation is sound, I will happily incorporate it.

I knew that changing is() would ruffle some feathers. However I think that new people, put two things into 'is()', and expect it to say if they are the same. Thinking about the fact that they are comparing numbers, strings, hashes, etc. is an unexpected burden on new test writers. Once they are experienced they will learn WHY thinking about it is important, at which point they can use the Compare plugins advanced usage to be clear what they want.

Ultimately this line of thought lead me to create the Test::Stream::Plugin::Compare plugin. Note, the implementation of 'is()' is it's own plugin, it is not part of the 'Core' plugin. This is important, I knew that some people would disagree with my implementation (my third to be precise). I expect alternatives to appear, and I hope they are better than mine. I felt that putting it in its own plugin made it seem an open invitation to alternative implementations, as opposed to calling it 'Core'.

I made the Compare plugin work in what I felt was the most useful way for an average person. This library compares whatever you give it, if you give it string it will compare them as string, numbers are checked as numbers, if you give it references it will compare the references, structures like hashes are compared deeply.

There are a couple things in people would take exception with here:

• That is not how Test::More did it, I expect a string compare, deep checks require a separate function
• In perl you can never be sure if a scalar is a number or not.

Following the ideas outlined further above I put more weight on the second issue here. You can never be sure, but you can make very accurate guesses. looks_like_number from Scalar::Util is pretty good for this. I further added 2 more features to help:

1. string compare is used unless BOTH sides look like numbers.
2. If a comparison fails it will tell you which operator the algorithm picked.

Here is an example on using this implementation of 'is' as 'is_deeply', and loading your own 'is':

use Test::Stream::Plugin::Core;
use Test::Stream::Plugin::Compare is => { -as => 'is_deeply' };
use Test::Stream::Plugin::MyIsPlugin;

Here is the same thing, but using the loader:

use Test::Stream 'Core', Compare => [is => { -as => 'is_deeply' }], 'MyIsPlugin';

Of course that is still a lot to type at the start of all your tests, so you would create the MyWay bundle that you use in all your tests:

use Test::Stream -MyWay

Back to Idea 4: "It should have what is needed to provide examples that teach people the right way to do something people usually do wrong when they first try."

I included 'dies { ... }' and a similar tool for warnings in the recommended bundle. This was done so that I could include examples of their use. History shows that people have hit the Test::Exception problem multiple times, with things like Sub::Uplevel produced to solve them. Then something glorious happened, Test::Fatal, which showed us a better way. By including these I can provide examples early on to new people so they do not fall into the same trap.

That was my justification for including them, though I have since relented. I expect that the next recommended plugin (-V2 if/when I add it) will likely not include them. Instead it will recommend Test::Fatal directly.

Notes:

Some may wonder why I am including my Warnings and Exception plugins in the main dist at all. There are probably other plugins that you might ask about. Ultimately my decision for which plugins were included in the main dist boiled down to which ones were essential to testing Test::Stream itself. If a plugin is in the main dist it is because I felt it was essential to testing Test::Stream. I need to test my exceptions, I need to test my warnings. Test::Fatal is not core, and I don't want to duplicate exception checking code over and over again.

This blog post reflects the entirely new module 'Test::Stream'. This document does nto address Test::More in any way. The Test::More API will not change at all. Test::More will not be altered to use Test::Stream plugins. The planned changes for Test-Simple involve changing Test::Builder so that it sends events to the Test::Stream infrastructure instead of directly writing TAP. The actual guts of the tools will change very little.

Why not include an 'is()' implementation like Test::More's in the 'Core' plugin? Because I felt its stringification behavior, while expected by people who have used it a long time, is a tripping hazard for new testers. I would rather leave Core unopinionated on 'is', and let that battle be fought in the plugins and bundles people create.

I fully expect some people to disagree about decisions made for the recommended bundle. This is fine, it is probably not intended for you.