On the eve of CPAN Testers
Have a look at the CPAN Testers reports for two TRIAL releases of the same module, one from 2 days ago, the other a little over 3 years ago:
Matrix for Text-Tabs+Wrap 2018.0419-TRIAL
Note: the current version of Perl in late April 2018 was 5.26.2, so all reports on more recent perls came in (long) after the initial testing which I’m interested here. The relevant part of the page is therefore the part from 5.26.2 on down.
Last time, reports started coming in within hours of the release; over 60% of the picture was there within a day; some 85% after 2 days; and the first wave of reports lasted a week.
This time, it took almost a day to even start getting reports, and the diversity has been much lower. 3 days in, reports are still absent for many platforms:
- None for NetBSD, almost none for OpenBSD.
- Only a handful for Solaris (but I’ll admit to small surprise about that one).
- Windows is sparse – although coverage is not notably down relative to last time.
- Cygwin is entirely absent this time – but not a big difference from the one lone report it had last time.
- FreeBSD and Linux are at least healthy – but no longer anywhere near comprehensively covered. (How’s that for something I’d never have expected to see?)
- Who’da thunk the best-covered platform would someday be… Darwin?
(Also of note is that the only 5.8 tested at all so far is 5.8.9, a maintenance release from long after 5.10.0 with little real-world relevance. Last time 5.8 had good coverage, including the most important releases (5.8.1; 5.8.5; 5.8.8). (Of course, any 5.8 is better than no 5.8 at all.))
Within just a few years, we are severely down on CPAN Testers resources.
And that’s compared to 2018, which already was a long way from any kind of golden age for CPAN Testers.
CPAN Testers is on its way out.
I never bothered with CI for CPAN modules before but now it seems an unavoidable necessity.
(And common CI products have never made broad platform coverage easy…)
Waxing philosophical
I've found CPAN Testers to be extremely helpful lately in troubleshooting a new distribution that I've been working on.
Many thanks to everyone involved in the CPAN Testers ecosystem for continuing to keep this valuable resource online and useful.
Within just a few years, we are severely down on CPAN Testers resources.
So, let's get concrete. What steps are you proposing or personally taking to improve the situation?
I always have, as well. Like I said, I never bothered with CI before – CI has always been better at some things (primarily turnaround times) but CPAN Testers used to be good enough at all of those things, and even at the diminished level of today, CPAN Testers offers a breadth of platform coverage that would take a lot of work to achieve with CI.
Aye, but my gratefulness is not limited to the ones who are continuing to keep it online. Many who contributed in the past have chosen to stop doing so (as evident from my observations). But their past contributions were not compelled nor compensated by anyone, so their choice to walk away diminishes not at all the generosity in contributing in the first place.
Well, the problem is twofold – people who used to contribute have walked away, and new people have not replaced them. Why have they walked away? Why did new people not step up? I’m not so cocksure I have the answers on either count as to propose solutions.
VPSes are dirt cheap in money and setup time these days so watching the paucity of reports over the last few days has had me thinking about spending some of my own resources to set up… something. But I’m not actually a user of the more uncommon platforms (even NetBSD is out of my range), so it’s not something I could easily make a substantial dent in. (If it happens. I have plenty more things to get to than I really can, already. Tending to b.p.o is already getting less of my time than I meant to devote to it…)
In any case, me running a few smokers myself isn’t going to address a many-years-long decline that has recently accelerated further. Unless maybe a post like this prompts the same “hmm, maybe I could spend a few bucks and hours of my time to help out” thought in people besides myself? I don’t know. Has it prompted you? You asked what I am proposing to fix this. Well, do you come into the picture anywhere? More importantly, how does a “we” get into the picture here rather than a “you”?
Aristotle wrote: "Has it prompted you? You asked what I am proposing to fix this. Well, do you come into the picture anywhere?
Yes, it has. When I set up a new virtual machine (which I do via vagrant, I now include Task::CPAN::Reporter among the CPAN modules to be installed at the outset. That way, any CPAN distro I subsequently install gets a test report without my having to think about it.
Earlier this year I assumed co-maintenance of Devel-NYTProf. I made extensive use of CPANtesters reports in determining whether my revisions were sufficiently acceptable or not.
So where can I RTFM on how to set up a smoker?
I’m not sure, Matthew. I haven’t kept up with the topic in a couple of years so I only have vague memories to go off that I’d have to refresh. One thing I found just now is CPAN::Reporter::Smoker. I seem to remember there being multiple turnkey solutions on CPAN by different people (basically a few of the most prolific testers codifying their own setups), but I can’t find more than that one right now.
(There are, of course, CPAN::Reporter itself for CPAN.pm and cpanm-reporter for cpanminus, whereby you can send reports for the day-to-day module installing that you do manually. But those are missing the endless loop of a dedicated CPAN testing rig, which would poll PAUSE uploads and download and test them as they come in. Which needs some extra code to deal with hung installers, downtimes at CPAN Testers, etc etc – all the robustness stuff you’d want in an automated setup.)
cpan Task::CPAN::ReporterThanks, I've set up a reporter on my local CPAN client. That's the least I can do, but maybe I'll set up something automated when I have more time