Ultimately, I believe I would like to leave the commit as it is, and remove support for Perl versions below 5.8.1. My reasons are as follows:
1. I understand the argument for not removing the old code but I would not feel happy leaving the pre-5.8.1 code in place even with the explicit declaration that it is not supported. I can support it now, even though it is not particularly easy to do so. I could probably continue to do so for some time. But it is not easy for me to do so and if the code is there I would want it to work.
2. I do not have a large team helping me to support Devel::Cover. Or a small team. Any time I spend dealing with 5.6 is time I am not spending on something that could be of more use to more people. Or being with my family.
3. I still have a hard time believing that there is even one person using 5.6 or 5.8.0 who would be inconvenienced by not using the most recent version of Devel::Cover. I can just about believe that there might be people still using Devel::Cover with 5.6, but if they are unable to update their version of Perl would they really be able to update their version of Devel::Cover? And even if they could, the toolchain now requires 5.8.1, so they wouldn't be able to update that.
4. If someone really does want to support 5.6 or 5.8.0 on an ongoing basis and in a way which does not require a lot of effort on my part then I am quite happy to revert the commits and accept pull requets to that end. The door is not permanently shut.
I believe this covers all of the points people have raised.
However, I will hold off on making a release which increases the minimum supported version until the QAH in Rugby. It is possible that discussions there may change my mind.
]]>We've already spent an extra year moving progressively and smoking out issues. We definitely needed that extra year, but it's already happened. Further attempts to do so are likely, sadly, to result in more delay than gain, I think.
> And that feels like it covers all the bases for "real people who want to use TB based code and T2 based code together".
Dev releases have been useful but at this stage having a proper TB-on-T2 release on CPAN is going, I suspect, to be necessary to shake out the remainder of the problems.
> The only residual that that proposal omitted, is the "force everyone to use T2" stage, which we can /still do/ under that proposal, we just don't have to be in such a rush.
We're not in a rush. We've been so much not in a rush people are starting to make perl6 jokes about the damn thing. But over the course of the process Exodist has learned at least a decent percentage of the correct paranoia level of a decent toolchainer, and it wasn't ready last year, and it's hopefully ready now.
So we're going to try it and find out, and if everything catches fire we'll re-upload the pair of independent tarballs that were already prepared specifically in case it turned out to be the wrong thing to do, look at what happened, and then figure out what needs to happen before we consider trying again.
]]>--
chansen
1) It is broke
2) The 'explicit opt-in' approach works only until people start porting existing extensions on CPAN, which a number of Test:: authors have expressed an interest in, and then users would see far, far weirder breakages than what we're currently doing is likely to cause
3) The Test::Builder API continues to work, so I fail to see how we're 'punishing' anybody.
4) You could've got involved in this process ages ago, coming along after the fact and shouting at people is not, actually, going to change or improve anything.
5) If you want to keep ranting, there's always /r/cperl, but please don't do it here.
I work on two test modules that will greatly benefit from Test2. These are Test::Class::Moose (TCM) and TAP::Formatter::TeamCity (TeamCity).
Neither of these modules would see much (if any) benefit from an opt-in approach.
When using TCM, you need to use other test modules like Test::More, Test::Fatal, etc. to actually do any testing. TCM is essentially a harness, but it doesn't provide the low-level "is this the expected value" type testing pieces.
For TCM to work with Test2, every single module you use to output tests must also use Test2. This means that a TCM built on top of Test2 would only be useful to people who were willing to write their entire test suite with other modules that had opted into Test2.
This would be viable for green field code bases that could choose to use Test2::Suite (which is pretty damn great), but presumably most existing TCM users are using Test::More and friends.
So to use Test2 in TCM (which has allowed me to _hugely_ improve the internals and fix many bugs with parallel testing), I'd either have to make a new distro (T2CM?) or tell everyone using TCM that the next release will break their entire test suite. Neither of these options seem very appealing, though I suppose the new T2CM distro is a better bad option. That said, I don't really want to maintain two separate distros!
The TeamCity formatter is in a similar situation. Unless the entirety of a test suite is emitting T2 events, I cannot write a decent TeamCity formatter. At $WORK, where we use this formatter, we already have a huge test suite built on top of TCM, Test::More, etc. We are really not likely to rewrite it all to use Test2::Suite any time soon (though I hope to at least use it for _new_ code going forward).
So that leaves us back with the two options that mst brought up. Do nothing or merge it and see what happens (with the option of unmerging in the face of catastrophe). T2 brings many huge improvements for test module authors. It's a huge step forward for the Perl testing ecosystem. I think the benefits it offers are worth some risk.
The fundamental question is "it is better to have a united or a split ecosystem, and for whom".
For the end-users, a split is clearly the less risky option, and there doesn't seem to be any direct advantage to them in keeping the ecosystems unified. On the short term I only see negatives, though on the long term there may be positives (if Test2 manages to give better reporting, and harnesses catch up to it).
For module authors, a similar thing applies: No short term advantages for current usages, though it does seem to make new usages possible.
And then there are the testing library authors. For most of them, a split is nothing less than a pain in the ass. While some modules may be obsoleted by Test2::Suite (and hence not necessarily need a port), many will not be.
In the grand scheme, I still think a split is the best path forward. Quite frankly I don't quite understand what's so bad about a new framework after 15 years. Yet the discussion did make me realize a new dimension that I hadn't fully grasped yet before: a split will need support and diligence from the testing library authors and module authors. That level of support is a second unknown variable in this equation (the first being the amount of stuff that breaks now), and we have even less data on this.
This second unknown left me with less certainty than I started that discussion with, though I still think the merge isn't the best we can do; but I think I utterly failed to clearly express this nuance of what I was meaning to say.
]]>For the end-users, a split is clearly the less risky option, and there doesn't seem to be any direct advantage to them in keeping the ecosystems unified.
I think the advantage for end users is that they can use things that test module authors produce. For example, with the TeamCity formatter I referenced in my earlier comment, it is nearly impossible to make a better version in the current ecosystem. Parsing TAP is incredibly painful, because TAP sucks (I need to write a blog post on this) and simply doesn't have enough information in it.
With Test2, I can create a much less buggy, much more useful TeamCity formatter. This will directly benefit anyone who wants to use existing test tools like Test::More under TeamCity.
This logic applies to many other testing tools that will benefit from Test2. Being able to write more reliable, more flexible, better tested test tools with less work is a win for everyone who wants to use those tools.
]]>Just so this statement is not left unchallenged, let me be on record that in verbatim form this argument is nonsense; although it may be meant as a shorthand summary of a longer argument which does make sense.
]]>This, however, I will call unqualified nonsense. Let me demonstrate to you why:
Thus the claim that Test modules would have to be written specifically against Test2, in order to be able to work with other Test2-based modules, is simply factually incorrect.
There are many test modules that easily work against Test::Builder in both its old incarnation and its Test2-based form, with no code changes.
There are very few test modules that truly need the new infrastructure provided by Test2.
So technically it would easily be possible to let the .t pick whether it wants the old or new form of Test::Builder, based on what modules it intends to load. Most test modules would require no changes of any kind to be able to work in either environment.
Only a handful of modules would need to be restricted to run on just the traditional Test::Builder or just its Test2-based reincarnation.
(And so as I just belatedly realise, the proposal shouldn’t even have been summarised with “bifurcating the ecosystem”-type phrasing. That is in fact an incorrect, misrepresentative framing. At the table, Rik asked me if that was a fair phrasing and I acceded; my bad. I should have objected.)
]]>Thus the claim that Test modules would have to be written specifically against Test2, in order to be able to work with other Test2-based modules, is simply factually incorrect.
That wasn't what I was trying to claim. Maybe I don't understand what you and Leon wanted to happen.
My point was simply that Test2-using tools will not cooperate with code that uses the existing pre-Test2 Test::Builder. Of course, if Test::Builder is using Test2 under the hood, then tools using Test::Builder do not also have to change (but AFAICT no one is saying they should).
]]>Yes, sorry, I’ll be publishing a long-form, detailed version of the proposal very soon. I’ve realised the proposal is (either largely or entirely) non-understood – the abridged version I presented on the fly at that table was insufficient. (I had hoped but failed to have it out there sufficiently early before the QAH that people could digest it in detail.)
My point was simply that Test2-using tools will not cooperate with code that uses the existing pre-Test2 Test::Builder.
Yes, I realise that now upon re-reading with fresh eyes. I see that “every single module you use to output tests” can be understood to mean “any module such as Test::Builder”.
It was the ambiguity of “output tests” combined with the “every single” quantifier that threw me off. “Every TAP emitter” or something would have been clearer to me.
Anyway, I think we’re on the same page now.
Of course, if Test::Builder is using Test2 under the hood, then tools using Test::Builder do not also have to change (but AFAICT no one is saying they should).
Right. The core point of the proposal is that every test library remains built on top of Test::Builder (unless it has more specific needs), and .t files can then individually pick whether Test::Builder’s guts are the old ones (default) or the Test2 wrapper shim. There are more details about how to make this two-pronged approach work at the ecosystem level (and I’ll be publishing them ASAP), but that is the gist: that test files get to pick.
]]>Test2 and Test::Builder remain separate. Tools pick one a build with it. To use both together you load a shim such as Test2::Legacy, which replaces Test::Builder with the alternative guts.
This was the alternative being discussed in the Test2 meeting on day 1 of the 2016 hackathon. We decided it was not the right plan. My main reason for opposing this plan is that nothing stops people from updating old modules to use Test2, and also auto-loading the shim to insure they do not break things that depend on them.
In this scenario you can never be sure that updating your test modules will not make the choice for you. If you use Test::Moose::More for instance it is currently on Test::Builder. Lets say the author updates Test::Moose::More to use Test2. If he does so he will break a lot of tests that also use other Test::Builder modules. So in order to avoid the problem he loads Test2::Legacy automatically. Now when you update Test::Moose::More, suddenly all your .t files load Test2::Legacy even though you did not directly ask for it.
In this scenario you have no idea when or if the switch will happen unless you watch all your test deps like a hawk every time you update from cpan. You have no idea which pebble will fall causing many of your test files to avalanche into Test2.
The proposal we went with instead was to make new Test::Builder use Test2. This means that there is only 1 point for the switch to happen. If you update Test-Simple then you get new guts. If you do not update you get old guts. If you pin your Test::Builder version than any downstream deps that require Test2 will fail to install and you do not need to worry about accidental updates.
The proposal we adopted means there is a single point of concern, which Test-Simple version you have installed. With the alternate that I myself mentioned back in 2015 in Berlin, and was still on the table table last week, you have no idea when or if your deps will change and suddenly switch you unwillingly.
This goes further. If we update Test::Builder to use Test2 (the current plan) than there is one thing for Test tool authors to target. If we keep them separate they need to insure their modules work with both the old and the compatibility shim, or they can just choose to break without the shim. This would lead to any number of possible combinations of test modules that can just explode one day due to simple changes.
Making Test::Builder use Test2 means we have 1 breakage event. Using an optional shim means we have any number of breakage events in the future as people maintain the test tools. The only way to prevent that would be to not have a shim, and force people to maintain both namespaces. But if I don't release the shim I am certain someone else eventually will, and then we have the same problem, possibly with even more things breaking if the shim is not as good.
]]>Thanks for the feedback. The software by default lists only one's own dependencies. Out of ~30 of my CPAN distributions, I believe only two of them aren't 5.8 compliant (due to external dependencies).
Also, I have automation software that allows me to test all reverse dependencies after any change to one of my distributions, so:
cd ~/repos/mock-sub
git pull
# make changes, perhaps bumping one of my
# own prereq versions
My [Test::BrewBuild](https://metacpan.org/release/Test-BrewBuild) software does a lot of things, but a basic run is started out of the current working directory, and runs unit tests of itself and optionally as in the case below, runs all the unit tests of all reverse dependencies against the changed version of the distribution I'm planning on uploading to the CPAN against any and/or all installed versions of Perlbrew perls (you can also remove them all and install any number of them from scratch to have a clean slate for every test if you want).
Here's an example after a random change that clearly broke things for some of the revdeps:
brewbuild -R
reverse dependencies:
RPi::DigiPot::MCP4XXXX
Devel::Examine::Subs
PSGI::Hector
App::RPi::EnvUI
File::Edit::Portable
Devel::Trace::Subs
RPi::DigiPot::MCP4XXXX
5.10.1 :: PASS
5.18.4 :: PASS
5.20.3 :: PASS
5.24.1 :: PASS
5.26.0 :: PASS
Devel::Examine::Subs
5.10.1 :: PASS
5.18.4 :: PASS
5.20.3 :: PASS
5.24.1 :: PASS
5.26.0 :: FAIL
PSGI::Hector
5.10.1 :: PASS
5.18.4 :: PASS
5.20.3 :: PASS
5.24.1 :: PASS
5.26.0 :: FAIL
App::RPi::EnvUI
5.10.1 :: FAIL
5.18.4 :: FAIL
5.20.3 :: FAIL
5.24.1 :: FAIL
5.26.0 :: FAIL
File::Edit::Portable
5.10.1 :: PASS
5.18.4 :: PASS
5.20.3 :: PASS
5.24.1 :: PASS
5.26.0 :: PASS
Devel::Trace::Subs
5.10.1 :: PASS
5.18.4 :: PASS
5.20.3 :: PASS
5.24.1 :: PASS
5.26.0 :: FAIL
Once all tests pass for all reverse dependencies, it's safe for me to upload the new version to CPAN.
]]>