CPAN Testers

Last month saw a number of changes, both major and minor, to the workings of CPAN Testers behind the scenes. As mentioned in the last post, some of these have had a striking effect on the ability to process more reports at a time. Last weekend actually saw us less than 24 hours behind a report being posted and appearing on the site. However, there is a caveat to that which I will come to in a moment. Having done so well to close the gap between reports being submitted and appearing, last weekend I took the opportunity to perform a major reindexing, which prepares the data storage for the next round of changes. Hence the spike you might have seen on the status page.

The biggest change at the moment is that the RSS feeds are now generated dynamically. This has reduced the amount of disk access, as less files are being written to disk for any particular request. Instead the data is now written to the JSON data store, and reformatted as required. Initially this is just for RSS feeds, however I plan to do the same for the YAML feeds too, which will be part of a further post soon. All this does mean there is slightly less for the builder to do, and has been a part of what has helped us to catch up.

As I mentioned, there is a caveat to the improvements. Due to the way the Metabase performs searching, there are some descrepencies between the reports we have stored in the Metabase and what the cpanstats database has. We are addressing the issue and there will be some improvements behind the scenes, however initially we have identified several thousand missing reports. Over the next week these will be injected into the cpanstats database, so we have a more complete picture. If you have spotted missing reports, and they haven't appeared by next week, please let me know.

The Metabase search issue is down to problems with SimpleDB not returning the search results correctly. As a consequence David Golden has been investigating the NoSQL databases, to see how we can improve this. There are several options available, and David is currently evaluating the pros and cons of a few to see which offers the most benefits to us, and can provide us with the best migration path and data replication methods to make any transition transparent to the testers and users.

Saywer X this month picked up on a benefit of CPAN Testers in his "Blessed are the CPAN Testers" post. Although this case involved a little more investigation to understand the problem, the fact that CPAN Testers cover such a variety of platforms does mean that we do have a better chance of exposing potential cross-platform issues.

After several weeks of uncertainty, it seems that GoDaddy has decided that we are still an Open Source project, and have granted us another SSL Certificate for a year. As such the Preferences site will shortly be restarted with a new certificate.

That completes the round-up for this month, though there are a few more posts I hope to get out over the coming weeks, so keep an eye out for further updates. In the meantime, happy testing :)

Cross-posted from the CPAN Testers Blog