Microsoft Attack CPAN Testers (Again)
Back In January, I reported how Microsoft had launched what amounted to a denial of service attack on the CPAN Testers server. It seems that 4 months later, we have yet again been targeted for attack from Microsoft. After the last attack, any IP address matching '65.55.*.*', hitting the main CPAN Testers website, was blocked (returning a 403 code). Every few weeks I check to see whether Microsoft have actually learnt and calmed down their attack on the server. So far, disappointingly, despite an alleged Microsoft developer saying they would look into it, the attack on the server has continue with little alteration to their frequency and numbers. Had they changed and been considerably less aggressive I would have lifted the ban.
Yesterday, Microsoft launch a further attack on the server using a complete new set of IP addresses. Now, just to clarify, this wasn't just a complete new set of IP addresses, but a completely new set PLUS the original set, thus effective doubling the attack on the server. Now you could claim stupidity or ignorance on behalf of the msnbot/Bing developers, but after being warned last time, and receiving 403s from their existing bots, by adding in a whole new set of IPs, I consider this latest attack nothing short of malicious.
These new IP address have now been added to the blocklist, and I'm now writing a script to alert me should any new IP address from Microsoft be added to their attack formation. Thankfully, I happened to be on the server at the time as both attacks hit, and managed to catch the IPs before they took out the server completely.
With my last post about this, I was accused of doing a disservice to Perl. Had I not been furious at the time, and written about the incident, I wouldn't have learnt that this was a Microsoft tactic that had infuriate a lot of people, and discovered that I wasn't the only sysadmin or website administrator around the world that had chosen to block Microsoft from their websites and servers. If Microsoft think thuggery is the way to improve their search content, then they are very sadly mistaken.
Cross-posted from the CPAN Testers blog
Update: Microsoft have now been in touch, and again apologised. We'll have to wait and see whether this can be resolved.
This is the new account for incidental and summary updates to what's happening with the CPAN Testers. For all the latest news and views please see our blog.
Apparently you need to add a robots.txt with Disallow / for all sub domains. Also the CrawlDelay 10 is maximum for msnbot.
http://www.reddit.com/r/programming/comments/c3l8f/microsoft_attacks_cpan_testers_again/c0pxk9w
@kthakore, the msnbot ignores robots.txt, as mentioned in my previous post. I had previously set all values as specified on Microsoft's own site, and it blatantly disregarded them. And yes it was a valid robots.txt too :)
Also note that this is just for www.cpantesters.org. All other subdomains are not hit in the same way thankfully. As such I haven't blocked them from those sites, but will if they try something similar on those too.
That's one additional reason to boycott Bing.
Even though Google is apparently losing its focus on search (with that hideous sidebar) they're still the most ethical ones around.
A couple of questions:
1) Do you have the facilities to block these with a firewall? (or better yet tarpit them?)
2) If you really think this is an attack, why don't you contact the appropriate authorities to deal with his? (http://www.justice.gov/criminal/cybercrime/reporting.htm)
@Phred, the IPs are blocked using Apache directives. Blocking at the firewall would deny them access to all the sites, currently they are only hitting the main www.cpantesters.org at an unacceptable rate.
Regards your second point, the server is in Germany, I'm in the UK and have neither the time, money or resources to fight the might of Microsoft. If the US government failed, what chance do I have?
I posted this on hackernews too.
@CPAN Testers,
I'd suggest blocking the perps by ip at the iptables level, that should take care of the immediate problem.
If you need mirrors, please ask for help. Help is out there :)
- Phred