CPAN Testers

November turned into a rather more traumatic month than I would have liked. It didn't start well with the old hosting company issuing us a invoice for 1 year's hosting, which we had previously cancelled due to their appalling service regards our loss of data. They now claim we still owe them a month, regardless of the month outage we suffered due to their incompetence and failure to replace our mirrored HDD before the second one failed. I'm now seeking compensation, but I'm not hopefully it'll get very far.

The SQLite issues are still surfacing. Following a complete rebuild of the SQLite database, Andreas is still seeing errors for some searches. Though not all, so those of you who use the SQLite download, may see intermittent faults. I have no idea why this happens and if there are any SQLite experts out there, feel free to get in touch.

However, these issues highlight a need to move away from SQLite as our means to port the data around the net. It compresses to just over 1Gig, but the bzip and gzip tasks, as well as the requests via Apache, are all taking their hit on the server I/O. A more managable solution is needed. As a result I hope to have more time to investigate this next year, and will possibly be concentrating on it for the 2012 QA Hackathon. If anyone has some good suggestions for replicating nearly 20 million records, I'd be interested to hear from you.

The I/O issues are still causing us a bit of a headache, as the although the CPU processing is fast, the disk is still taking quite a hit. As a consequence the report feed, parser and builder are all vying with each to keep on top of things. We're roughly 36 hours at most behind, although I've been trying manually ease the pressure occasionally to try and get best usage out of the disk. Again I shall be looking closer at this in the new year.

I've been approached by a few people with regards to sponsorship. At the moment we're not in a position to take monetary donations, but all being well in the new year a special fund will be available to anyone wishing to contribute. We've also had offers of server space, mirrors and backups, which I hope to take advantage of too. If you're emailed regarding anything like this in the last few months, if I haven't got back to you yet, please be assured I will do :)

We're back to nearly 1 million reports a month again, and this is one of the reasons why we're a little behind after the server rebuild. We are catching up, but slower than I would have liked. However, it has to be said that I'd rather have too many reports to process than not enough ;)

Cross-posted from the CPAN Testers Blog.