Promoting CPAN Testers
Recently Gabor Szabo has been tweeting about his climb up the CPAN Testers Leaderboard. It was something that Damian Learns Perl also picked up on his recent CPAN tester post. It was also exactly the kind of healthy competition I had in mind to encourage people to become CPAN Testers. However, is there anything more we can do to not only attract testers in the first place, but also keep them submitting reports? Do we need to raise the profile of CPAN Testers?
On the CPAN Testers Discussion mailing list, Gabor has been suggesting some ways to promote CPAN Testers, and get more regular testers involved. While I don't have any problems with encouraging anyone to get more involved with CPAN Testers, I still want to keep those occasional and even one-time testers still willing to submit as many reports as they feel comfortable with, which some have seen as a problem. Here is why I believe it's not a problem.
Firstly, occasional and one-time testers often only submit reports when they install modules for their currently working or development environment, which may well be significantly infrequent. This does not mean that their contributions are any less significant though. They are testing on real machines in real life environments, and can often run into to problems that automated testers may not experience. This is particularly true for distributions which rely on libraries not installed on automated smoker environments.
Secondly, the statistics only give a picture based on the user profile. The leaderboard doesn't make any distinction between individual testers and automated smoker networks. As a consequence, someone like Chris Williams, who has several dedicated automated smoker environments, distorts the weighting. The trend would likely be a little more even if we were able to calculate the number of reports from each unique smoker environment. Many of the testers who appear to have submitted only one report, in many cases have submitted others. Although I have put in a lot of effort to consolidate multiple email addresses into a single profile, there are many email addresses which are no longer used, and for which I am unable to pair up to an existing profile.
Thirdly the leaderboard as it stands is for the whole of CPAN Testers history. It took nearly 10 years to reach 1 million reports, while now we have over 500,000 reports submitted each month. The distribution of tester contributions is very different over the last few years than it was in the early years.
However, while it's not a problem, it would be much nicer to be able to encourage more of these low volume testers to submit more reports more frequently, as their contributions are valuable. This primarly means trying to understand why some testers have only posted a few reports and then have choosen not to continue. Is there some reason for this that we can resolve? Has the change to the HTTP submissions confused people and they've not continued?
I'd also like to make it as easy as possible to install the required software, both for infrequent testing as well as for more automated environments. To a large degree we already have this covered, with various instructions on the Wiki, although I'd be delighted if we could encourage contributions to explain more scenarios.
Gabor also asked how we can encourage companies to run smoke testing on their machines, particular during any out-of-hours downtime. I'm not sure of the best way to do this, as many companies have usage policies that prevent the use of machines for this kind of purpose. For those that do want to contribute and want to support CPAN Testers, what is the best way we can help them?
There has also been the suggestion to create logos and banners to promote CPAN Testers, which can be used both by companies and individuals on their websites and blogs. As a starting point we have the smoking onion, which I would like to keep as a brand for CPAN Testers, but if you have graphic skills, I would be very interested for anyone to create some logos and banners that build on this. I'd like to be able to create a page with a variety of images which people can then use to link back to us and show their support for CPAN Testers.
So if you have any ideas for promoting CPAN Testers, or have 133t graphics skills to create some images, please get in touch via the CPAN Testers Dicussion mailing list and share your ideas with the CPAN Testers community.
Cross-posted from the CPAN Testers Blog
This is the new account for incidental and summary updates to what's happening with the CPAN Testers. For all the latest news and views please see our blog.
I think that if CPAN Testers wants to increase the number of people submitting tests, it needs to get more interesting.
Everything that I've seen posted by CPAN Testers is aimed at the CPAN Testers community itself. There are Barbie's monthly updates that say "We've passed N million updates, and so-and-so is working on the Foo subsystem," but that's only interesting to the CPAN Testers group. They're not interesting to the Perl community in the large.
You're telling what, but not why.
I suggest that telling some interesting stories about the value that CPAN Testers brings to CPAN authors and to Perl itself would help raise public perception of the good work that you guys do.
* Why is it good to have cross-platform checks on distributions?
* What sorts of problems can arise on different platforms? Can you give examples?
* How are CPAN authors helped by having the CPAN Testers squad check these things?
* Why would someone not yet part of CPAN Testers want to join up and help?
These questions are all obvious to those of you in the CPAN Testers echo chamber. They are not obvious to the outside.
Can you tell a story about an author being helped by CPAN Testers? Can you tell at least one a month in your monthly status reports? This will help attract people, and also encourage the testers themselves to keep things going.
Actually my main hope is to get more people send in test reports from their real world environments just as they install modules.
For this the current "competition" is probably not a good way to encourage people.
Though I have setup a few smokers now so I can move ahead in on the leader board but I also configured my regular development perl to send out reports when I install or upgrade a module.
I also included CPAN::Reporter in the recent Padre on Strawberry Perl release. I just need to provide instructions on how to turn those on.
BTW the link to my tweets is here.
Andy, the summaries, etc are indeed primarily meant for the CT community, but I post them here too as others are also interested in what's happening. I think your story ideas are worth following up and will look into writing something for future posts. I once did a talk about the experiences of cross-platform testing, so will have to dig that out again :)
Gabor, sorry I missed out your link to twitter, I've put it into both posts now. The competition element of the leaderboard was only ever intended for the hardcore testers, it isn't something that will ever really motivate casual testers. However, getting more casual testers would definitely be a good thing, and I think some of Andy's story ideas may work well.
Thanks for adding CPAN::Reporter to that Padre release, it will go some way to giving new users a good introduction to CPAN Testers.
The core question when you want to motivate people in some way is always the same: “Why would I want this / want to do this?” Therefore the story you want to tell for CPAN Testers is this:
“This month, Joe R. Tester found a bug with X in the Y module, which John Seepanauthor fixed. Peter Othertester found a bug with Z in the Foo module, fixed by Nick Otherauthor. […Etc…] Overall, $n distributions had new releases this month fixing bugs found by CPAN Testers. $n1 of these were found by smokers, $n2 came from casual testers.”
That answers the core question for a potential casual tester viscerally: code they rely on (or would like to use) gets better if they pitch in, and doing so takes near-zero effort on their part.
It even indirectly demonstrates to authors what value they get out of participating (since, as we know, not all of them care to hear from CPAN Testers).
I think for now you will have to lobby CPAN authors to send you such stories. But you could gear the infrastructure towards making these stories visible, by providing hooks for turning test reports into new bug reports (and for attaching them to existing tickets) (ideally one that doesn’t work with just RT). Then these trails can followed and tabulated by crawling that data.