CPAN Testers & pre-requisite reporting
Seeing as b.p.o won't let me comment on others' blog entries, I'm posting here.
This is in reply to brian's recent post about a trial version of Test::More.
Secondly, this is a conversation that has cropped up before, and I'm still in two minds about it. Short answer: I tend to side with brian. The tester platform shouldn't be doing any testing with the trial/development releases of pre-requisities, unless the tester is going to manually filter the results and send to the pre-requisite author if appropriate. I do understand Ether's perspective too, and there is merit in having these reports, but they more often target the wrong author.
Longer answer: The difficulty we have once the reports hits the Metabase, is that all the parsing into the CPAN Testers databases cannot know whether the test failures are because of the pre-requisite or the distribution being tested. The fault could be in the pre-requisite, but equally as Ether notes, the pre-requisite could be the next release and the distribution being tested makes assumptions that are no longer true. The analysis site, run by Andreas, could possibly highlight this with a suitable corpus of reports, but it too wouldn't necessarily know whether the pre-requisite or the distribution being tested is at fault. It takes a human to determine that.
I want CPAN Testers to be helpful to authors, and in this situation it isn't. The wrong author is being alerted, and in order for the right author to be notified, it requires the wrong author to take the time to feedback to the pre-requisite author. It's a step I don't think we should be asking authors to do. In the short term, the solution is to ask testers not to test with trial/development pre-requisites. This could be to ensure smokers test in a clean environment, or to exclude any trial/development releases if you test in batches.
One longer term solution would be to add more logic in the smoker client to re-attribute the report to the pre-requisite. This would require quite a bit of work, and I'm not sure how easy that would be, but it might be worth someone investigating one of the clients to see what could be done. Another solution would be to add logic to the backend parser, which I plan to abstract out to make it easier to use in more applications. However, both of these solutions could still be attributing reports to the wrong author, and we would be back in the same position, just from a different perspective.
The Admin site can now allow authors/testers to mark reports to be ignored from the system. But I will look to see what can be done so that the system can re-attribute reports rather than ignore them. It's still fixing the problem in the wrong place in my opinion, and we're still relying on authors to do the work, but at least we won't lose the reports.
In Summary, I think testers have a responsibility to direct reports to the right author, and if they don't then stop using trial/development releases as pre-requisites.