boakes.org

Something’s broken on the database that runs this site, specifically there’s b0rkage in the comment table, and the ssh daemon is not reachable. Perhaps it’s to do with the larger than normal amount of spam that’s been arriving today (for “larger than normal” read: one metric truckload). More info later after I’ve had a poke around. Film at 11.

Update1: Comments recovered, the DB had become corrupt somehow, possibly an overloading issue as the trackback comments were recorded.

Update 2: I’m not sure what caused the see-saw to tip over but when rebooted the load average was starting at 1 and heading skywards through 6 before the ssh session died. This all happened after I upgraded to WP2.5 (though I’ve been on the bleeding edge builds for ages, so it’s unlikely to be related… For a while I thought it could be file permissions. Thankfully the folks at bytemark provide a VM admin shell that enabled me to reboot the machine whilst it was under siege, gaining a couple of minutes after each reboot during which I could invoke some countermeasures.

Update 3: I’ve installed Donncha’s WPSuperCache plugin which has reduced the load average down enough that the server is at least usable again, so I can do more investigation.

Update 4:Looking at my helpfully rotated log files I notice that today’s log is approximately 25x the size of a normal log. a grep of the logfile suggests that we’ve been hit with approximately 25000 trackback spam messages between 4am and 2pm … hmmm.

labs:/home/www/boakes.org/logs# grep -c "/trackback HTTP" access.log.1
24447

Update 5: I’ve also installed Mike Hampton’s Bad Behaviour and that’s helped reduce the load too…

The sooner I can get automatic htaccess level banning working again on WP2.5 the better! Today the server’s fielded fewer trackback requests, just 7000 between 4am and 11am, a mere 1000 per hour.

Update 6 (48 hours on): top is now showing a 15 minute load average of 0.15, something way better than the previous normality has been resumed. What’s particularly interesting is that Google analytics for the day shows a lower than normal page view count and the number of advert clicks was also proportionally down. This suggests that spammers do little or no automatic clicking on adverts, which will be reassuring to customers of Google Adsense!

Update 7 4 days later: After more tweaking of super cache (it seems it was only enabling the default cache due to a mod_rewrite configuration problem) the load average on top now looks like this…

top - 19:05:01 up 3 days, 20:51, 1 user, load average: 0.07, 0.00, 0.00
Tasks: 62 total, 1 running, 61 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.7%us, 0.0%sy, 0.0%ni, 99.3%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st


That’s a fifteen minute load average of 0.00 … very low!