CPAN Testers

All was going well last month, we had a few problems balancing the feed and database inserts, but with some help from Devel::NYTProf, we had improved performance and were getting back on top of things. We also had two presentations at YAPC::Europe by myself and Léon, and were heading for 1 million test reports submitted in a single month ... when.....

The CPAN Testers server hard drive developed a fault. Tthis wouldn't have been a problem had the mirrored drive, which had failed earlier in the year been correctly replaced. Alas the first failed drive was still absent, with the second drive failing and switching to read-only mode almost immediately on reboot. As such I had just started to backup in case we lost the drive completely, and then it did fail completely :( Our hosting company required us to accept complete data loss before replacing the two drives.

Before I continue, please note that this hasn't affected the Metabase server, and all reports currently being submitted are being safely stored. Once the CPAN Testers server is ready the feed will be switched back on and the the server will start updating the websites again.

Starting from scratch has meant reinstalling all packages and modules needed, as well as reloading the database with over 10 years worth of data. The cpanstats tables I took from a backup a week before the drive failed, as it appears some faults had affected the more recent backups. These have now all been reloaded. The metabase tables are currently being uploaded for 7-8 million records (and associated indices). The articles tables unfortunately hit a problem. The original database backup has a fault in the tar file and with it being so huge, tar couldn't cope with trying to fix it. Thankfully, Robert and Ask gave David and I an archive of all the original NNTP posts back last year, and these are being reparsed and inserted into the database. Once all these metabase and NNTP article records have all been uploaded, I'll then be running some database checks to ensure we have everything all sync'ed to the last known point.

The first websites and processes to be put back into the action, have been the archive accesses to the CPAN and BACKPAN directories. You can once again get full access via the HTTP, FTP and rsync protocols, to the CPAN Testers' Tier 1 (fast mirror) services.

It seems there were 2 sets of files that I hadn't backed up for quite sometime, which I really should have done. The first is the apache configuration files, which have been tweaked every so often, so while I have some backups for the entries, they are mostly out of date. As such please be aware that some sites may come back quicker than others. The second set are the cronjob files, which again have been tweaked since the backups I have. While its nothing I can't fix, it just may take a little longer to get the server back on form.

Interestingly Leon had asked me in Riga about the backups, and whether we needed somewhere else to store them. Thankfully the offsite backups are in London and Birmingham, but these only cover the database and source code files. The resulting website files will need to be recreated. As such, I'll now be looking to make a periodic snapshop of these so we can rebuild from a known point rather than rebuilding from scratch.

Our biggest problem is time. With approximately 250GB of database data to insert and index, and then having to rebuild the website data files, I'm not too sure how long it will take to get us back on our feet again. I hope that some of the sites will be back online by the end of the week, but the main Reports site may not be fully operational until next week. Please bear with us, and we'll get things back on track as soon as possible.

One last thing I had meant to include in last month's summary. I was interested to see the graphs the Carey Tilden had done using the CPAN Testers Statistics data. I may well include similar graphs on the stats site in the future. So thanks Carey for the idea.

Hopefully, I'll have a more uplifting summary next month, and will be able to confirm our report submission rate. I'll also have some news of other things happening within the CPAN Testers world.

Cross-posted from the CPAN Testers Blog