CPAN Testers

30 Million Reports

2013-04-18T06:28:54Z

Congratulations to Nigel Horne for submitting the 30 millionth CPAN Testers Report. The report itself was a PASS for DBI.

CPAN Testers Summary - January 2013 - Up

2013-01-11T20:58:06Z

Quite a few updates for this month's summary. New CPAN distributions, plenty of fixes, some updates, lots of discussion ... and 27 over million reports!

]]> Firstly, there was a new CPAN distribution relating to CPAN Testers, cpantimes. This is a smoker tester for cpanminus. If your preference for an installer is cpanminus it might be worth taking a look. Toby Inkster also wrote a blog post about its release.

Gabor Szabo asked about how to find modules that weren't being tested on Windows. After some investigation, I updated CPAN-Testers-WWW-Statistics and revised the No reports pages. You can select the Operating System of your choice, and the resulting table lists all the latest distribution releases that have no reports for them. There is a slight caveat in that releases within the last 4 weeks are not included, to give testers a chance to test them. If you spot distributions that shouldn't be included, please me know. Gabor has now followed up with a post detailing 13% of CPAN distributions don't have a test report on Windows.

Our catch-up of reports has gone very well. We're mostly about 2 days behind at the worst, and less than a day for the most popular pages. Diab Jerius's post on the mailing list, asking why there were no reports for MooX::Attributes::Shadow, gave me a chance to post detailing why reports often take time to show, and why the side panel bar graphs can be more up to date than the reports listed on the page. Once I have the latest catch-up processes automated, I will be looking to improve the performance of the page builder.

Also on the mailing list, Nat Goodman posted a question about missing content for some PASS reports. On further investigation, Chris Williams discovered it was a bug in one of the CPANPLUS components. On the other side of the fence, Kirk Kimmel alerted us to what appeared to be CPAN::Reporter is disappearing. Again it took a bit of investigation, but between Kirk and David Cantrell, they discovered Acme::Bleach inside Task::Cpanel::Internal was at the heart of the problem. David Golden is looking add suitable skips to handle the issue. Martin J. Evans started a conversation regarding the use of META files, which led to how the automated tools use them. Andreas König then clarified how configure_requires and dynamic_config are used by CPAN.pm. Reini Urban advertised his distroprefs, and finally Mario Roy thanked the all CPAN Testers for helping with testing MCE (Many-core Engine for Perl).

Although the discussion started in January, I did want to include a topic raised by Ricardo Signes regarding testing of old development versions of Perl. While all of us can see the merit of testing stable releases of Perl, even prior to 5.6, but there really isn't a lot of use in testing anything on CPAN against a development snapshot of 5.11, 5.13 or 5.15. As such, if you're looking to test against development releases, please only use the current development release. At the current time this is 5.17. During the development and testing life-cycle for the next stable release, testing against CPAN can be useful. However, once the the next stable version is released, the old development snapshots are no longer relevant. We'll try and advertise this a little more as stable versions are released, but please bear it in mind if you're testing on a variety of versions of Perl.

On a final note, congrats to Nigel Horne who submitted the 27 millionth test report to CPAN Testers. It was a PASS report for Devel-StackTrace-AsHTML-0.11.

Cross-posted from the CPAN Testers Blog.

CPAN Testers Summary - December 2012 - Shabooh Shoobah

2012-12-10T22:31:44Z

November turned out to be a very eventful and productive month. Aside from various code updates with some CPAN-Testers distributions, including porting many of the tests to Test::Database, and discovering the usefulness of Test::Trap for testing some of the scripts, we also got a handle on the missing reports. For the past few months, questions about missing reports has increased. Back in August I started to look at a more thorough catch-up. After some suggestions and ideas from David and Andreas, I also added some to code to collect data in a similar way to the tail log. As a consequence of the tail parsing, the improved catch-up code and the rewritten generate code, it now means that not only have we caught up, but we now have a much more robust mechanism in place to ensure we're not missing any reports.

]]> November turned out to be a very eventful and productive month. Aside from various code updates with some CPAN-Testers distributions, including porting many of the tests to Test::Database, and discovering the usefulness of Test::Trap for testing some of the scripts, we also got a handle on the missing reports. For the past few months, questions about missing reports has increased. Back in August I started to look at a more thorough catch-up. After some suggestions and ideas from David and Andreas, I also added some to code to collect data in a similar way to the tail log. As a consequence of the tail parsing, the improved catch-up code and the rewritten generate code, it now means that not only have we caught up, but we now have a much more robust mechanism in place to ensure we're not missing any reports.

During November, the builder was under heavy load to compile all the pages with new reports. With 3 processes running simultaneously, a high volume of request were being made, but thankfully it held its own, and is managing requests very well. The oldest request is less than 2 days old at the moment, with the latest report lag being about 15 hours behind. As such, if you don't see one of your reports or your distribution doesn't seem to be quite up to date, please wait a couple of days to make sure it's not just waiting to be processed.

Last month also saw some significant limits being pushed a little higher. First and foremost 1033056 reports were processed in a single month. Although we've had more than a million reports submitted in a single month, its never been this high before. Chris Williams became the first tester to pass 10 million report submissions, not that we doubted it was coming soon. We also now have 6 testers who have submitted over 1 million reports each, and last month saw an notable increase in the number of testers currently producing reports. Hopefully this is a trend that will continue. November also saw the biggest variety of platforms we've ever had tested too, and with 102 different platforms its the first time we've managed to cover more than 100 in a month. With all these new highs, it perhaps isn't too surprising to hear that at 67, we've also had the highest number of perl versions being used in testing too, with 63 so far this month already. And finally the number of reports ... last month we passed the 25 million (although last week we passed 26 million) reports submitted. As always many thanks to everyone who has submitted reports over the past 13 years.

On the mailing list Shmuel Fomberg began a lengthy discussion about reports grades that would highlight distributions with dependencies that failed to install. While I understand why many users might find this useful, for authors it would be a blight on their distributions, which may not be truly representative of the state of the distribution. For example, a distribution may fail with one version of a dependency, which then gets fixed, but the report remains tag against the latest distribution, even though the new latest version of the dependency works. It gives a false impression of the distribution. Without extensive metadata analysis the CPAN Testers systems wouldn't know that first report is now is not necessarily relevant. Or is it? What if the user can't install the latest version? Either way an author is tarred with a fault beyond their control, and that isn't what CPAN Testers is about. This isn't the first this discussion has come up, and I doubt it'll be the last.

JT Smith asked about the best practice for test network access. As yet there isn't a one size fits all, LWP has an is_online method, some distributions use ping, although this available on all platforms, and others test whether they get a HTTP 200 response from a known URL. If you know of a more suitable method, please add it to the wiki. Kirk Kimmel asked how long it took test the whole of CPAN, which didn't get a definitive answer, but was guess to be several days, but less than a week. Nathan Goodman questioned why he saw no output in a PASS report for hist distribution, and after some investigation, Chris Williams discovered a bug in CPANPLUS (specifically CPANPLUS::Dist::Build), proving yet again how useful CPAN Testers can be ... even with PASSing reports.

With all the improvements and great support from the community, I'm very pleased to see the CPAN Testers project is in a very healthy state again. I'd also like to thank everyone who has contributed to the CPAN Testers Fund, and particularly those who attended the London Perl Workshop at the end of last month and poured their spare cash into the fund buckets, including the CPAN Testers Fund. Projects like CPAN Testers can only survive with the volunteer contributions, both in time and cash.

Although we still can't say too much just yet, we do have some great news that the Metabase will be moving to a new home in the new year, and we'll be moving off SimpleDB, meaning we might have even better response times in future. My personal thanks to David Golden for following this up, as well as initial introductions and negotiations by Karen Pauley and Ricardo Signes. 2013 is looking very bright for CPAN Testers and long may it continue.

Cross-posted from the CPAN Testers Blog.

CPAN Testers Summary - November 2012 - Kick

2012-11-06T08:14:52Z

If I could kick the Amazon SimpleDB we have, I'd take great delight in doing so. Thankfully, our days of relying on it are numbered ... in a good way. More news on this will be forthcoming soon.

However, over the last month the most popular email regarding CPAN Testers I have received, was along te lines of "is CPAN Testers broken?" The answer is definitely no, but Amazon's SimpleDB most definitely is. It's not clear why its broken, but the results it returns often mean we don't get the full result set requested, and the cpanstats database appears to miss reports. September appears to have been particularly bad with 496784 reports missing from the initial result set. As such, I have been running a catch-up process for the past few weeks, and it is picking up all the previously missed reports. I'm now running this for October, and over the next week, possibly two, I aim to be completely up to date. Thanks also to some help and ideas from David and Andreas, I now have a process ready to deploy that will allow the system to run a catch-up daily, and avoid missing reports entirely.

According to SimpleDB the following is an ordered result set by date:

2012-10-10T11:02:17Z
2012-10-10T11:02:20Z
2012-10-10T08:09:10Z
2012-10-10T08:07:59Z
2012-10-10T11:02:12Z
2012-10-10T11:02:30Z
2012-10-10T11:02:23Z
2012-10-10T11:02:34Z
2012-10-10T11:02:38Z
2012-10-10T14:14:54Z
2012-10-10T11:02:46Z
2012-10-10T11:02:41Z
2012-10-10T14:14:55Z
2012-10-10T08:09:33Z
2012-10-10T11:02:47Z

Is it any wonder that we miss so many reports!

Part of the solution has been to look at a slightly different query. The tail log that David currently creates unfortunately only lists a reference to the underlying report summary fact, and not the main report fact. As such, I have now adjusted the catch-up to use the same query as the tail log. Once in place I should be able to have a much more reliable feed.

On the mailing list, Shlomi Fish asked about Getting rid of the problem of an overloaded PERL5LIB on Windows, Kjetil Kjernsmo asked about Scalar::Util 1.23 openhandle changes and several people asked about missing reports.

Rest assured I am working on the missing reports and hope to have everything back on track within the next week or two.

Cross-posted from the CPAN Testers Blog.

CPAN Testers Summary - September 2012 - Listen Like Thieves

2012-10-13T10:53:36Z

We had a bit of downtime on the CPAN Testers last month. Did you notice? I doubt it, as the guys at Bytemark did a wonderful job helping to get us back online. We had a disk failure and with minimal fuss they replaced the disk and had us back up and running within a few days of spotting the problem. Such a far cry from the fiasco of our previous hosting company. Many thanks to all the guys at Bytemark.

A major catch-up of reports was undertaken last month, and we managed to retrieve several thousand reports missing from the results that Amazon send us via their broken API to SimpleDB. I'll be running another catch-up for the last month this weekend too. Thankfully Andreas König has created a script that will help me combat the problem in the future, and potentially reduce the impact of requests on the Metabase too. In the short-term it should mean we are less prone to missing reports, although in the longer-term moving to a new Metabase will eradicate the problem completely. Discussions are on-going at the moment with regards to the longer-term plans of the Metabase, and we hope to have more news on that for you soon.

CPAN Testers helped out Neil Bowers last month, when he added some test scripts to his distribution, Module::Path. His blog post details the steps he took in creating the tests and how he was able to refine them with each release. The story uncovers some interesting asides that emphasise why testing is important, as well as some unusual edge cases.

Last month I mentioned Brian Cassidy's CPAN::Changes Kwalitee Service, and how his next target was to reach 10,000 releases passing his tests. I'm pleased to say that as of the time of writing, there are now 10145 distributions passing his tests. I'm not sure what his next target is, but it would be nice to see another 405 releases passing to reach 40% :)

On the mailing list, Leo Susanto had trouble with including files in his gcc compilations, David Golden asked for test feedback for HTTP-Tiny, and Shlomi Fish asked for the best way to test for the existence of header files from external libraries. Finally Neil Bowers asked about having a smoker that can specifically tests development releases, as a lot of the test reports don't include them. The main for not including them, is that several smokers are set up for installation, and typically most people don't install development releases. However, many of the hardcore CPAN Testers are quite happy to test, but for other reasons their smokers may not pick them up. Several testers responded with example scripts and ideas, and at some point I'll collate these and add them to the Wiki, unless someone else gets there before me.

Unfortunately the summary was rather delayed this month, due to my current workload, but I aim to be back on track again for next month.

Cross-posted from the CPAN Testers Blog.

CPAN Testers Summary - August 2012 - Wish

2012-09-10T18:49:54Z

August was quite an exhausting month, with several news worthy items. The YAPC::Europe 2012 conference, as mentioned last month, featured a few talks relating to testing, including my own looking specifically at the main components of CPAN Testers. I have often been asked about how the infrastructure of CPAN Testers works, and hopefully my talk gave some insight into that. The process diagram featured in the talk will be added to the CPAN-Testers distribution soon, and will be expanded at a later date to try and capture all the parts of the current CPAN Testers environments. CPAN Testers has grown substantially from its early beginnings, and its much easier to get involved. With so many parts to the whole system, it would be great to get more and more new contributors to the code base. See the Development site, which also needs updating now, for further links. It also intrigued me to see Test::Reporter::Transport::Metabase in the (xvii) MetaCPAN favourites weekly report post. Hopefully whoever up-voted it, has been looking at contributing too :)

Kenichi Ishigaki was also at YAPC::Europe, and he previewed some of the forthcoming changes to CPANTS. He's done a lot of work behind the scenes getting the site back online. Working alongside his CPANAuthors site, the CPANTS site now looks a lot more modern, and hopefully will get used a little more again. While watching a twitter feed during the conference, I also happened to notice that Brian Cassidy posted about reaching the next goal for his CPAN::Changes Kwalitee Service. As of writing this, he needs another 12 distributions to pass the tests to reach 10,000 passes. While it's not a necessity to have your Changes files in a machine readable format, it is useful to have a convention that works for other uses. It highlights Kwalitee rather than Quality :)

In the last few weeks I've been cleaning up some of the code behind the CPAN Testers Reports site, including some requests posted in RT. As such, the distribution CPAN-Testers-WWW-Reports got a major update and now has better crawler detection, now displays the report counter headers for each distribution release and fixes the summary displays. The backend code has also had a bit of a clean up to improve some of the performance. In the coming weeks as promised some of the crawlers will be let back into the site again. Another big change has been to match the convention used elsewhere so that TRIAL distributions are now marked as development releases. I have also worked through old reports and changed the settings to reflect this too.

Proof, if proof were needed, why we need to move away from Amazon's SimpleDB, is the recent catch-up I started running recently, to collect all the missing reports from the last 3 months, that SimpleDB failed to extract from the Metabase. It's a constant source of confusion from testers as to why their reports are not appearing. The catch-up added 245989 reports to cpanstats database. That's over 80,000 reports that are currently being lost each month. For the money they charge every month, that's exceptionally poor service as far as I'm concerned.

Special thanks this month are reserved for Steffen Schwigon, aka renormalist, who has made a significant personal contribution to the CPAN Testers Fund. Steffen was one of the people who pushed us to get a CPAN Testers Fund set-up a few years ago, so we are delighted to see him come good on his promise to donate. If you'd like to make a personal contribution, please visit the donation page and contribute.

The CPAN Testers Fund is growing, and while we are grateful for every contribution, big or small, we'd love to see more, so we can help pay for all the running costs every year. Over the last month, Mark Keating had been throwing some ideas my way of how he could promote CPAN Testers, and at YAPC::Europe the light-bulb was most definitely on. Mark launched a corporate donation programme for CPAN Testers, whereby companies could contribute a regular monthly donation, rather than a big lump sum. Mark's idea is that if we could get 20 companies all contributing monthly about £20, that would be enough to cover our current expenses for a year. Taking the first step, ShadowCat are the first company to now make a regular donation to CPAN Testers. You and your company can too by simply visiting the CPAN Testers Fund page and setting up a regular recurring payment.

On a final note, I'd like to thank chromatic for mentioning CPAN Testers in a post about why he uses Perl testing. As chromatic notes, the Perl community's attitude to testing, and they have produced to support testing, has made Perl the success it is.

Cross-posted from the CPAN Testers Blog.

CPAN Testers Summary - July 2012 - Head On The Door

2012-08-12T13:56:56Z

July was a relatively quiet month for CPAN Testers. Although reports have been flowing, our attentions have largely been elsewhere. Development work behind the scenes is still continuing, but nothing major to report just yet.

Ben Bullock asked on the mailing list, whether he could search other people's test reports? The problem currently with this, is that we don't really expose the reports themselves, except via the CPAN Testers Reports website, when you specifically ask for the report. The reasons for this have largely been because the search of the Metabase still needs to be written. The demand on the current Metabase is expensive, and until we are able to move to the new backend system, we can't afford to expose the results. For the time being the Analysis site covers some of the demands, but Ben's specific needs aren't covered.

Ben also noted that because several web crawlers are blocked from the reports site, this restricts the ability to search the reports for common error strings via other sources. The blocks were implemented due to some crawlers being rather heavy-handed. I specifically don't like the way Google allow website owners to restrict their crawlers for a set period, and then blast their way through the site when the period ends. It's counter-productive as they get banned from sites. This wasn't the only reason Google were blocked, but the tactics aren't appreciated. Now that some of the code has been reworked, it is a little easier to restrict the side effects the crawlers can now have on the site. As such, I'm looking at unblocking some of the crawlers to see how they and the site performs. If all goes well, more crawlers will be allowed to access the site. I will be monitoring the crawlers quite closely, and any abuse will result in those crawlers being blocked again. The end result will mean that searching for specific strings, will be possible via search engines.

The next major event in the Perl conference calendar will be YAPC::Europe 2012 in Frankfurt am Main, Germany. I'll be presenting "The Eco-System of CPAN Testers", which aims to show you the process flows of CPAN Testers, highlighting the code used, and where you can help contribute code. Other talks also featuring testing are Eric Johnson with "Selenium testing with Perl" and Kenichi Ishigaki (charsbar) with "CPANTS: Kwalitative website and its tools". There are several talks that look at ways to investigate broken code and improve your approach to code too. The current schedule looks very worthwhile, so if you're still undecided take a look at what you could take away from the conference, and book your ticket now. Hope to see you there.

If you're a CPAN Tester, and will be at YAPC::Europe, please come find me and say hello during the conference. It's always great to meet testers who feature in the leaderboard, but I only know by name. If you have any suggestions for future posts, questions about CPAN Testers, or ideas for collaborations with CPAN Testers, as I'll not be arranging a BOF in Frankfurt, feel free to let me know during the conference or during the evening sessions. If you're not attending YAPC::Europe, please join the mailing list, and post your suggestions, questions and idea there. Happy testing.

Cross-posted from the CPAN Testers Blog

CPAN Testers Summary - June 2012 - Seventeen Seconds

2012-07-10T13:08:58Z

MDK is the man! A big thank you to Mark Keating for a great post about how to donate to the CPAN Testers Fund. I have received some feedback about how to make the fund and the donation process a little more prominent on the websites, and I plan to address that in the coming months. I have also been very encouraged by some of the feedback, and hopefully we shall see more donations and sponsorship in the coming years too. Aside from asking your company if they can donate, or writing on your blog about how CPAN Testers have helped you, if you're so inclined, you might want to add a note to your README or POD to tell users how they can donate. There are plenty of other funds you might want to advertise too, so don't feel restricted to the CPAN Testers Fund. CPAN Testers can also benefit indirectly from the other funds, so its all good.

A post I forgot to mention last month was the news that CPAN Dependencies now accommodates META.json files in distributions. With the move to use META.json files for version 2 of the Meta Specification, a number of distributions have started adding the file. Some have included both a META.json and a META.yml, but some are now only releasing distributions with a META.json. This change to CPAN Dependencies now means that these distributions can once again be included in the calculations. Thanks to Dave Cantrell for the update.

Another post I should have mentioned last month, but only became aware of it after the post, was from Vyacheslav Matjukhin. Although his post wasn't particularly related to CPAN Testers, it does make an interesting discussion point regarding how automated testing isn't the only avenue for testing. Automated tests provided in your distribution are only part of the whole picture. Documentation is important and we can test for it being correctly formatted, but we can't test whether its understandable. We can't also test whether the usage and results of the code make sense to another user. You can write tests that make sure you understand what you expect, but if a user of your distribution struggles to understand the arguments and results, it doesn't help them. Without a doubt Validation::Class has benefited from the feedback, and while the author and reviewer might not agree on all points, it was a very worthwhile exercise. Not always easy to get such feedback, but if you can it is well worth taking advantage of it.

On the mailing list Nigel Horne highlighted to me an issue with the Leaderboard. His numbers didn't appear to be adding up correctly. This was in part due to the trawl through the Metabase to find the missing reports Amazon had failed to send through, but also due to an issue with the way the leaderboard is generated. The code has now been rewritten, and a new database table created, to better manage all the counts. As of now, the leaderboard numbers should be much more accurate, and be incremented as expected. During this process I took the opportunity to map some addresses, and got through 105 mappings, of which 44 were brand new names to the system.

Next month sees YAPC::Europe 2012 taking place in Frankfurt. There are a few testing related talks planned, but I'll cover those in next month's summary. The deadline for talk submissions is 15th July, so there are still a few days to complete your submissions before the deadline. Hope to see you there.

New CPAN Testers Sponsor: Webfusion

2012-06-10T15:24:08Z

It is with great pleasure that we officially announce a new sponsor for the CPAN Testers Project. Webfusion have provided us with Managed Hosting, for us to use with some of the supporting websites. As such, we will be using it for the Analytics and Matrix websites, as well as a secondary failover site for the Static Reports site.

Webfusion are the latest corporate sponsor to support CPAN Testers. If your company would like to support CPAN Testers, please get in touch. You can also donate to the project via the CPAN Testers Fund, managed by the Enlightened Perl Organisation.

For further details, please see the CPAN Testers Blog.

CPAN Testers Summary - May 2012 - Black Moon

2012-06-08T11:37:29Z

May proved quite an interesting month. Firstly, I got several confused emails relating to the Status page on The CPAN Testers Reports site. Secondly, we ran out of slots in the namespace for Amazon. And then thirdly a very involved discussion on versions on the mailing list.

During the begin of the month, I started running a regeneration script to find missing reports from the past year. It seems there were several thousand missing, due to the way Amazon's Simple DB works. I will be running the script again soon to catch any missing reports from May. On the Status page, it lists the last stored report, which some found rather confusing. The Status page itself is just a simple dashboard to help highlight whether there are any problems that need investigating, but also help to identify how far behind the feed is. In this case it didn't give a true picture, until the normal feed was restarted. If you spot any listings for older reports, it is likely I am running the regeneration script again to find missing reports. It is perhaps also worth repeating what the status on the page means. The status (Excellent, Good, Manageable, Struggling and Overloaded) simply refers to the number of requests in the queue to be processed by the Reports Builder, though not necessarily the number of reports waiting. I may look at adding an extra variable to determine the status, based on the age of the youngest report, which might give an additional indication of something being wrong.

Relating to the feed, the namespace we were using within the SimpleDB instance ran out of slots. This meant David Golden had to quickly update the Metabase to a new namespace. Thankfully he was available and online, having got up early, even though he was on holiday. So big thanks to David for fixing so soon. The current namespace should keep us going for some time, and hopefully will not be a problem as we're looking to move to a new system soon.

Phillip Moore has recently taken over the NetApp suite of modules, and discovered the problems encountered with versioning. He began a thread asking for suggestions on the best way to rework the modules, some of which he wanted to split into their own distributions. Version numbering has been covered many times, and although new authors and modules follow more reasonable version numbering, before the discussions and recommendations, CPAN had all manner of version numbers. As such, trying to change the varying version numbers in the NetApp suite has proved interesting! It has proved how important consistency in numbering can be, especially when subsequently abstracting modules into their own distributions.

We passed the 22 million reports mark last month, which highlights a reduction in the reports being submitted, as well as the number of testers we have at the moment. We still have over 100 voluteers submitting reports, but the report submissions have fallen to just over 500,000. This is still a fantastic amount, but with the numbers we saw towards the end of last year, I wondered whether we would see further increases. I guess a recruitment drive might be worthwhile at some of the forthcoming conferences. Speaking of which...

This month sees YAPC::NA taking place in Madision, Wisconsin. The conference itself is now sold out, but for those attending, there is an great selection of talks to choose from. Anyone interested in testing, might like to check out James E Keenan talking about "Perl Testing 101: 82% of What You Need to Be a Competent Perl Tester", and/or Belden Lyman presenting "A Test-Driven Developer's Cookbook". August will have YAPC::Europe to look forward to, so hopefully there'll be a few testing related talks there too. For any one presenting a testing talk, whether at a YAPC, Workshop or user group technical event, please get in touch, and I'll happily advertise it for you.

Finally this month, David Cantrell posted that he was having trouble with Ukraine spiders. It seems that some just cannot learn from the mistakes of others. Now they've been blocked. Possibly CPAN Testers are a victim of their own success, but it doesn't bode well for search engines in general if they ignore robots.txt and/or launch unnecessary (even accidental) DOS attacks on sites. Maybe if some of them wanted to donate to the CPAN Testers Fund, then we'd be more willing to let them back in.

Cross-posted from the CPAN Testers Blog.

CPAN Testers Summary - April 2012 - Pictures At An Exhibition

2012-05-08T12:27:52Z

Firstly for this summary, I would like to extend a big thank to lestrrat, Mark Allen and Ron Savage for the first individual donations to the CPAN Testers Fund, as managed via the Enlightened Perl Organisation. The fund has been a long time coming, and we are really greatful to all the donations. If you'd like to contribute something to help CPAN testers, whether as a one-off or a regular contribution, please see the EPO CPAN Testers Fund page for further details. We plan to list all donators on the CPAN Testers Sponsors website, as a further thank you to you all.

Continuing the thank yous, a very special thank you goes to Bytemark Hosting, who host the main database and webserver for us. Back in March we were running out of space running the backups, which occasionally took out the websites. As such we approached the guys at Bytemark to see whether we could increase the disk space. Having re-evaluated their server offerings recently, they now provide the bigger hard drives for our package, and so we scheduled a day of time after Easter to allow us to upgrade. Unsurprisingly the databases took the longest to copy across, but once new disks were mirrored, we were up and running very quickly, with no issues at all. Very many thanks go specifically to Chris and David for making the upgrade so seemless and smooth.

And finally, but no means least, a further thank you has to go to Webfusion. They are donating a server for CPAN Testers use, which we will be using to host a 2nd tier BACKPAN, CPAN and MiniCPAN, as well as some of the CPAN Testers family of websites. expect more details very soon.

Last month saw some blog posts, particularly from Jonathon Swartz and Martin Evans, which looked at why the CPAN installer tools shouldn't run tests. While I can appreciate their perspective, the argument is slightly flawed. Their reasoning is based on the fact that DEB and RPM packages can be installed without testing, except the problem with this is that you're not comparing like for like. The Debian and RedHat package maintainers actually run the tests on your behalf before packaging and releasing the distributions. As such there isn't a need to test on install when using the DEB and RPM installers. The CPAN installers (cpan, cpanp and cpanm) are all retrieving the source distributions, not pre-prepared packages for a given platform. In this case you have no idea whether the distribution will install on your platform.

However, as a way to avoid the testing stage for those who really want to skip it, it would be interesting to see whether the CPAN installers could make use of CPAN Testers to attempt to verify the platform, and see whether reports have been submitted and what success rate it achieved. Tests could then be skipped if the number of reports and success rate is above a (user) pre-determined value. CPANPLUS has alreay previously looked at this, but I don't believe it currently makes use of the new data structures. I would be interested to see if the maintainers of the installers wanted to pursue this. The JSON data is already there and the CPAN-Testers-WWW-Reports-Query-AJAX distribution aims to provide the statistics for a specific platform/perl.

For several months I have been getting email, both privately and via the mailing list, highlighting missing reports. Although we are aware of the deficiencies of Amazon's SimpleDB, and David Golden is closer to moving to a new infrastructure, it is often difficult to find the specific reports we need to reimport. Recently Andreas sent me some very specific details, and as such I have now started running a regeneration script that can look for holes in the lists SimpleDB returns and make further requests for the reports within a shorter time limit. So far it has picked up several thousand missing reports from the last month. As such I will now start periodically running this agains the whole database to try and discover missing reports from the last 18 months. Until the new system is in place, I will now look at running the regeneration script at least on a weekly basis to catch any reports that slip through SimpleDB's very dubious net.

Once again, I would also like to make you aware of the problems with the cpanstats SQLite database. We still have not been able to resolve the issues, and the lifetime for the SQLite download is going to be limited now. I haven't decided on a date to decommision the file, but if you are using it, please look into CPAN-Testers-WWW-Reports-Query-Reports and see how you can make use of it. The new API reduces the need to download Gigabytes of data, and reduces the overhead for producing the database considerably. If you have any problems with the API, please let me know.

More news coming soon...

Cross-posted from the CPAN Testers Blog

CPAN Testers Summary - March 2012 - Brain Salad Surgery

2012-04-08T09:16:24Z

So March ended on quite a high, following the 2012 QA Hackathon. With so many key people in one room, it was impressive to see how much got done. You can read reports from myself (parts 1 & 2), David Golden, Ricardo Signes, Miyagawa, Paul Johnson, Ovid and Dominique Dumont, and there were several tweets too, during and after the event, and the wiki also has a Results page. There was a significant number of uploads to PAUSE during and after the event too. And CPAN Testers has benefited hugely from the event.

Arguably the two biggest significant developments during the event were thanks to Breno G. de Oliveira, who not only added support for CPAN Testers within cpanminus, but also began the work on the long desired CPAN::Testers::Common::Client. While working on the former, Breno noticed that there was a lot of common reporting tasks (and differences) performed within CPAN::Reporter and CPANPLUS::YACSmoke. As he wanted to replicate this within his client for cpanminus, he asked whether it would make more sense to wrap this into a separate library, to which David and I were full of encouragement for. Breno set about setting up a GitHub repository and has been doing some fantastic work bring all the reporting together. You can follow his efforts of GitHub, as garu, and hopefully we shall start to see this distribution on CPAN and in Smoker clients soon.

While that may have been the most significant output for CPAN Testers, other parts of the toolchain also made some giant leaps. Having Andreas, David, Ricardo, Schwern and Miyagawa all together meant David's idea of changing the way we look at indexing CPAN, allowed kinks and ideas to be ironed out in minutes rather than weeks. David's idea is to distance the indexing system from the repository. This will allow other repositories, such as the many DarkPAN repositories out there, to use the same system and enable toolchain installers to point to different repositories as needed. Nick Perez is now working on CPAN::Common::Index, which will form the basis for the new system. You can read the details in David's write-up of the event. Hopefully this will be a major step forward to enable CPAN Testers to be used for independent repositories, which has been a request for many years.

In other news we finally announced the sponsorship website and CPAN Testers Fund. Over the past 10 years the CPAN Testers has been funded largely by the developers. In the last 5 years hosting, network and bandwidth has been increasing, with the developers and Birmingham.pm being the principal donors. While this is great, and we really do appreciate this, the bigger CPAN Testers becomes, the more support we need. As such we are hoping to encourage sponsorship from businesses and corporations, especially if they use Perl. If you would like to know more, please get in touch. Many thanks to the Enlightened Perl Organisation for managing the CPAN Testers Fund for us.

On the mailing list there was a discussion about the reports not showing the differences in tests where strings look identical, but where one may contain control characters. Personally I feel trying to fix this in the browser is too late. As we treat the test report as a complete piece of text, we cannot currently isolate the test output to know what control characters to highlight. It could end up confusing the viewer. Andreas also thought that the author should include more appropriate tests in their test suite, and suggested the use of Test::LongString, Test::Differences or Test::HexDifferences.

As we've mentioned a few times, the SQLite database download is no longer as reliable as it once was. Andreas believes that we may have a unusual text pattern that is causing the problem, but whatever it is, it's not something we know how to solve. As the database file is high maintenance, I would like to abandon it sooner rather than later. If you currently consume the SQLite database, please take a look at the new CPAN::Testers::WWW::Reports::Query::Reports release. This now the preferred way to request records from the master cpanstats database. For the summaries, you now also have CPAN::Testers::WWW::Reports::Query::AJAX. If you have any problems or suggestions for improvements, please let me know.

On a final note, please be aware that CPAN Testers will be down for maintenance on Thursday 12th April. We'll try get everything back online as soon as possible. Thanks to Bytemark Hosting for helping us with the disk upgrade.

Cross-posted from the CPAN Testers Blog.

Sponsoring CPAN Testers

2012-04-02T07:11:15Z

CPAN Testers has now been running for nearly 13 years. In that time we have been supported mostly by the community and some very thoughtful individuals, to whom we owe a massive thank you. Ten years ago we were submitting less than 1,000 reports (March 2002) each month, 5 years ago we reached a new high with just under 25,000 reports (March 2007) submitted. In the last year it is no longer unusual to see 1 million reports (August 2011) submissions in a single month. Regular readers of this blog will be well aware of how successful CPAN Testers has become, and how much data we store, as we sped past 20 million reports in February 2012.

This awesome success is now reaching the limits of community generosity. As we are capturing ever larger amounts of data and providing increasing amounts of report analysis, we are starting to look further afield for the help and support we would like. The increasing volume of report data, and the more sites we deploy, means we need to look at more powerful servers, more disk spaces as well as several other things (such as SSL certificates). As a consequence our costs for maintaining the CPAN Testers project have also been increasing.

Although we can sustain this growth in the short term, we really need to look towards more corporate involvement to ensure we have a long term future.

For the past 4 years most financial contributions have either been from the admins themselves, or via Birmingham Perl Mongers (being a registered non-profit company), but we wanted a more formal sponsorship and donation programme. We needed a way to encourage donations via a public Perl entity, that businesses could recognise and acknowledge as a legitimate representative of the Perl community.

Following several long private discussions between Mark Keating, David Golden and myself during 2011, Mark put forward the proposal to the Enlightened Perl Organisation to manage a fund on our behalf. Very kindly the members agreed, and we now have a CPAN Testers Fund you can donate to. But this is just a first step. Having the fund in place now means that not only do we need to promote it, we also have to actively promote the sponsors who have helped CPAN Testers get to where they are now, to encourage further sponsorship. And so....

Please welcome The CPAN Testers Sponsors site.

This site will contain all those sponsors who we would like to thank, including individuals, and provides further information about how and what to sponsor. If you have any questions regarding sponsorship, please free to contact us and discuss what we can do to help make sponsorship from your company as easy as possible.

In addition the site provides a feed that allows the corporate sponsors to be promoted across the CPAN Testers family of sites. The feed hasn't been activated just yet, but I hope to implement this across most of the sites during the coming weeks.

If your company has a sponsorship programme, or you think there will be a benefit to them sponsoring the CPAN Testers project, please point them at this new site and ask them to get in touch with us.

But that isn't all. There are many individuals within the Perl community who would like to donate to help CPAN Testers cover their overall costs. This new CPAN Testers Fund will now make this possible. While we would welcome large donations from the business community, we are also just as pleased to have members of the Perl community continuing to support us. As such, the new site aims to make a point of thanking those individual sponsors too. No matter how big or small the donation, it all counts.

CPAN Testers has provided a lasting and valuable service to the Perl community, even if indirectly for some. We have helped to make the CPAN repository one of the most respected and revered amongst the OpenSource community. The new fund and site will help us keep that reputation for many years to come.

Cross-posted from the CPAN Testers Blog

CPAN Testers Summary - February 2012 - Highway 61 Revisited

2012-03-11T22:53:10Z

With 20 million reports, CPAN Testers is very definitely one of the biggest online repositories of test reports for both programming languages and software applications. While other languages and applications may have larger communities than CPAN Testers, the Perl community's commitment to testing has uniquely enabled CPAN Testers to build on the benefits from many areas of the test community. The TAP protocol has now been incorporated into several other language and application testing frameworks, and stand-alone applications, such as Smolder, have been able to harness the output to present results in a way which best highlights problem areas. CPAN Testers too has been striving to improve the way we present and provide analysis of reports.

With regards to the analysis CPAN Testers provides, a notable example has been the work by Andreas König with the CPAN Testers Analysis site. By taking a selection of test reports, his tools have the ability to find common areas which help authors to pin-point unusual problems. One aspect this has recently helped with, is identifying a problem with development releases, which in themselves don't necessarily fail their tests (possibly because the don't include tests which test all scenarios), but when used within other modules may highlight faults in the prerequisite. The problem with this is that the report is associated with the calling distribution, not the prerequisite which has the fault. Andreas' tools recently were able to identify a prerequisite that was causing concern for a number of distributions, even though the fault lay elsewhere. This in turn led to a lengthy discussion on the CPAN Testers Discussion List, of how we could best approach this. The difficulty though is writing parsers that would automatically determine what was a fault. With many of the test reports it requires a human to determine this for single reports, while Andreas' tools can only highlight where further human investigation is needed, and needs many reports to determine if there might be a cause that may need investigating.

In the short term it is possible to re-associate reports with other distributions, by requesting to me, but in the longer term it would be better to enable a automated process. Which is where the Admin site comes in again. Although that's been on hold for a while, I think this is another feature which could simplify the process of correcting mis-filed reports. As such, I hope to continue on that site soon.

The discussion on the mailing list asked whether we could more appropriately filter these types of reports, but didn't lead a conclusive answer. However, it will be a discussion topic for the forthcoming QA Hackathon. At the end of the month, a number of testing hackers will be assembling in Paris to further Perl's testing infrastructure, processes and code. It will be a good opportunity to have a number of different views discussed in one room with all those likely to be actioned with writing the solutions. This QA Hackathon looks to be the biggest so far with roughly 40 people attending. There are a diverse set of projects being planned for the 3 days, so expect lots of output afterwards.

Gabor Szabo highlighted an issue running CPAN-Reporter with Net-HTTPS-6.02 last month on the mailing list. Thanks to Manoj Kumar, if you experience a "fact submission failed" error, try rolling back to Net-HTTPS-6.01. Tim Bunce also asked about testing on Windows. Although Windows is still a regularly tested platform, the volumes are considerably less than for other more Unix based OSes. David Golden put in a lot working getting CPAN-Reporter and Strawberry Perl working on Windows, so if you have the spare capacity and want to get involved with CPAN Testers, Windows is certainly an ideal platform to make a difference. And if you're running Windows, Cygwin would welcome some more regular reporting.

Lastly, for now, the problems with the SQLite Database have surfaced again. Although I can't definitely say why there is a problem, it would seem to be related to the sheer size of the data CPAN Testers stores. The uncompressed version is now over 8GB. To reduce the bandwidth, and better provide data in a way that consumers can store and use it, an API is being written to supply a record set, rather than the full data set. Expect further work on this during the QA Hackathon.

Cross-posted from the CPAN Testers Blog.

20 Million Test Reports

2012-03-07T22:10:17Z

I am rather delighted to be able to announce that on 22nd February 2012, CPAN Testers reached a very significant milestone in our history. We reached and have now surpassed 20 million test reports, as can be seen via the Interesting Stats page. That's approximately 14 million via the HTTP API.

For the past few months this has been nearly 1 million reports each month. Expectations are now that we could reach 30 million by the end of the year. This is a phenomenal number of reports, and I did notice a little while ago, that a certain other test report repository have now removed their claim of having the biggest database of test reports. It's going to be a long while before any other code repository has anything like the number of test reports available for authors and users to analyse. In part that is also thanks to the 24588 distributions now on CPAN and the 5576 active CPAN authors, but is also thanks to the 1,000+ testers (even those who've only submitted one report), who keep making CPAN Testers a worthwhile project to be involved with.

Congratulations to Andreas J. König for posting the 20 millionth report. It was an UNKNOWN report for Makerelease-0.1.

Cross-potsed from the CPAN Testers Blog.