MSNBOT must die!
If you've suffered any problems accessing any of the sites, the databases, the CPAN mirror, etc from the CPAN Testers server last night, please direct your wrath at Microsoft. Last night the msnbot took out the CPAN Testers server with a dedicated denial of service attack. As a consequence measures are now being put in place to completely ban the msnbot from accessing at least the Reports site, and probably all the sites on the server.
Microsoft in their incompetent wisdom decided to unleash 20-30 bots every few seconds. I know this because I can see the IP addresses in the logs. The ones spotted within a few minutes of rebooting the server this morning to clear the processes were:
It seems their bots completely ignore the rules specified in the robots.txt, despite me setting it up as per their own guidelines on their site, and worst of all they don't talk to each other to see they are accessing the same domain. Most sensible bots, such as those of Google or Majestic 12, will only let one bot at a time crawl a site, as most sensible companies acknowledge that a DOS attack is not good policy. As a consequence I'll now be denying access to anything with the IP matching /^65\.55\.(106|107|207)/. If you discover you fall into that pattern, and are a real person, please let me know.
If anyone from Microsoft ends up reading this, though likely you'll have to do it in person and not via a bot, I now consider you to be no better than a script kiddie trying to bring down a government computer. DOS attacks usually get people charged and arrested. If CPAN Testers was a legal entity, then I might have been able to follow this through. Instead I'm locking the doors, and no longer letting you through.