To see that happen is quite strange, akin to having your water supply cut off when the reservoir up the valley is full to the brim. There’s no easy way to ask why (and if you find your way through the webmaster tools to request reconsideration (in the stairless-basement; locked filing cabinet, “Beware of the leopard” sign etc), the likelihood is they won’t respond in a timely manner, and when they do respond the content will be terse and useless, at least, it was for me).

After much banging of my head on the wall of Google, eventually, an anonymous feedback engineer offered up that this site was “manually blocked” because someone else (let’s call them an operative, because it sounds dystopian) decided this is a “spam blog”. Google provided no more information than that, but eventually, after about 6 months, decided to unblock the pipes and people started to find the site again.

The Moral

The moral? It’s not safe to rely on only Google traffic for a business. If someone at Google makes an honest mistake then your whole livelihood can be removed overnight, with no warning, no explanation and no hope of a quick fix.

Hopefully, Google+ will go some way to alleviating this problem. The massive verified-human-crowd-sourced database of opinion on sites should go some way to removing the mis-flagging of normal small sites as spam blogs.

If you’d be so kind as to +1 this site, it might go some way to avoiding any future unplanned absence from the web (as google describes it).

• Comment on Google+
• Comment on twitter

For example: using search.twitter.com I get the folowing results for samoa

1. Hypeflash: Alao Siva-American samoa flag day 2008 http://tinyurl.com/[REMOVED]
2. Hypeflash: Bizarre Foods SAMOA 4 – http://tinyurl.com/[REMOVED]
3. lisa1248: Get free medicines and sample for free http://bit.ly/[REMOVED] #everlastingsong Goodnight Google Wave #HealPhilippines Kraft Life Motto Samoa
4. dixiefs51: after Manila, Typhoon Ketsana ( ondoy) has killed people in Vietnam then a tsunami hit Samoa.
5. jessecardol3e: American Samoa | got a cam? wanna go one on one? http://wowurl.com/[REMOVED]

Only one of the most recent five tweets is of any relevance, and even that is just passing on the story, not adding any detail. The rest all link off to sites selling medicines or other vices. I’ve got to look harder to find more information, but wading through an 80:20 spam ratio is not convenient. When looking for timely information, this kind of noise can only lead to the source being dropped from the search.

Live news from witnesses, moments after a major event, is Twitter’s most compelling virtue. The problem is, ne’er-do-wells looking to make a quick buck are now all over it like a bad rash.

Now, it might be possible to utilise an n-degrees of separation model for searches. I might do a search for samoa, and restrict results to those written by people withing 4 degrees of separation from me: however, such a system is almost certain to be useless, because there’s no guaranteeing that my expended network is affected by and recording the event.

A more fine grained model of trust might be a solution here. I don’t necessarily know everyone I follow on twitter, but I do know who my friends are and who I trust. If there was a way to put a figure on that trust then rather than the binary degrees-of-separation model, a probabilistic model could be used where I ask for search results within a trust threshold.

Update1: Comments recovered, the DB had become corrupt somehow, possibly an overloading issue as the trackback comments were recorded.

Update 2: I’m not sure what caused the see-saw to tip over but when rebooted the load average was starting at 1 and heading skywards through 6 before the ssh session died. This all happened after I upgraded to WP2.5 (though I’ve been on the bleeding edge builds for ages, so it’s unlikely to be related… For a while I thought it could be file permissions. Thankfully the folks at bytemark provide a VM admin shell that enabled me to reboot the machine whilst it was under siege, gaining a couple of minutes after each reboot during which I could invoke some countermeasures.

Update 3: I’ve installed Donncha’s WPSuperCache plugin which has reduced the load average down enough that the server is at least usable again, so I can do more investigation.

Update 4:Looking at my helpfully rotated log files I notice that today’s log is approximately 25x the size of a normal log. a grep of the logfile suggests that we’ve been hit with approximately 25000 trackback spam messages between 4am and 2pm … hmmm.

labs:/home/www/boakes.org/logs# grep -c "/trackback HTTP" access.log.1
24447

Update 5: I’ve also installed Mike Hampton’s Bad Behaviour and that’s helped reduce the load too…

The sooner I can get automatic htaccess level banning working again on WP2.5 the better! Today the server’s fielded fewer trackback requests, just 7000 between 4am and 11am, a mere 1000 per hour.

Update 6 (48 hours on): top is now showing a 15 minute load average of 0.15, something way better than the previous normality has been resumed. What’s particularly interesting is that Google analytics for the day shows a lower than normal page view count and the number of advert clicks was also proportionally down. This suggests that spammers do little or no automatic clicking on adverts, which will be reassuring to customers of Google Adsense!

Update 7 4 days later: After more tweaking of super cache (it seems it was only enabling the default cache due to a mod_rewrite configuration problem) the load average on top now looks like this…

top - 19:05:01 up 3 days, 20:51, 1 user, load average: 0.07, 0.00, 0.00
Tasks: 62 total, 1 running, 61 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.7%us, 0.0%sy, 0.0%ni, 99.3%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st


That’s a fifteen minute load average of 0.00 … very low!

As before Worst Offenders works cooperatively with other anti-spam plugins: its primary purpose is identifying and deleting the comments that are 100% definitely spam (sent by the very worst offenders) so that any “false positives” (sent by real genuine humans) can be rescued from the spam bin!

Download it from
WordPress.org

I’ve got it working on this site already, where it’s proved faster than the previous versions – it also has a nicer user interface. There are a few minor operational features that need to be finalised, but it’s basically capable of doing what it’s supposed to.

This version has a pluggable interface, so different “litmus tests” can be applied to spam at the same time, and third parties can easily write tests without having to write a whole interface.

I’m keen to hear from people who:

• Know their way around WordPress/PHP already.
• Can take a look at the litmus test API and comment on ways to improve it.
• Suffer from very high spam loads (hundreds or thousands per day) who’ll be able to give the existing litmus tests a bit of a workout to check if their SQL is as efficient as I hope.

Development SVN is being kindly hosted by Automattic and releases will be available here.

There’s been a bit of a ramp, however:

1. Nine months ago Akismet caught it’s 100 millionth spam.
2. Sometime this week Akismet will kill spam number one billion.

Check out the stats, watch it happen live.

A virtual high-5 to everyone at Automattic. Great job guys, a great service, and the scaling has been perfectly handled.

Perhaps we should start a sweep-stake for how long it will take to hit one trillion?

Worst Offenders

I wrote the other day about a small Akismet extension that I’d been playing with that helps remove the worst offenders from the list of caught spam – this in turn makes false positives easier to recognize. One of the things that the extension notices is which IP addresses are particularly prolific, it’s this information that helped me to a 95% (and rising) reduction in the spam that I send to Akismet to be checked.

htaccess

When writing about the extension on the Akismet mailing list, I suggested that hooking the worst offenders IP list apache htaccess file should provide a simple dynamic means of rejecting spams before the http request has been processed by the server (and before the akismet plugin has had to check it against it’s server).

So I tried it, and it works, apparently flawlessly so far. In my access log I can see that over 100 requests have been rejected so far today. That’s 100 requests that are not sat in my “spambox” waiting for me to check for false positives.

Todays Akismet spam count is in the single figures, instead of the triple figures.

Fantastic.

Ongoing Thoughts

So what does this mean for this site?

1. If a spammer does manage to leave a message here, then it’s caught by Akismet and marked as such, it never gets through.
2. When I remove those messages, I can optionally ban the IP address from whence they came.
3. Bandwidth usage is reduced becuse the server does not accept connections from spammers, and fewer two way chats with the Aksimet server are necessary.
4. The comment database does not get needlessly filled with temporary comments that are removed once their spammyness has been identified, so the DB and DB indexer has less work to do.

So what does this mean for the Akismet project?

1. There will be fewer hits from this site, so more time to concentrate on others.
2. If the changes can be used by others, the net effect will be a more scalable and responsive service.
3. If large scale uptake was achieved, the spam zeitgeist might start to look different because the number of spams being checked daily should significantly reduce.

There are several obstacles to global spam nirvana, including:

1. The installation proecess will require a tiny bit of “hand cranking” to ensure the htaccess file is in a suitable state for automatic updating.
2. The system should probably exist as a separate plugin that hooks into the Akismet plugin at appropriate points, but those points haven’t been defined yet.
3. The system should probably exist as a separate plugin that hooks into the Akismet plugin at appropriate points, but those points haven’t been defined yet.
4. Not everyone uses Apache, so not everyone has an htaccess file.
5. Not everyone uses WordPress, so Akismet service users on other platforms will have to re-implement the idea.

Download

Note: this is an experimental extension to Akismet. It appears to work for me, and I will be delighted if others try it an can give useful feedback for the experiment.

i.e. Please don’t ask linux/htaccess questions – I’d love to help, but I don’t currently have time to hand-hold on a non-production experiment.

Still keen? Great.

If you’re really brave and want to try it out, you need to carefully follow these steps.

1. Precondition: Follow the installation instructions for Akismet and ensure it’s working correctly.
2. Download the Extended Akismet Plugin.
3. Replace the akismet plugin with the one you just downloaded – it should pick up all the configuration from the “official” version of the plugin.
4. In the WP admin interface Open the “Manage | Files” tab, and check that your htaccess file is writable.
5. The automatic IP banning is written between two markers that are “# BEGIN worst-offenders” and “# END worst-offenders”. If you already have a deny list in your htaccess file, just add the markers to the list. Mine looks like this:

   Order Allow,Deny
   # BEGIN worst-offenders
   Deny from 202.75.49.130
   Deny from 202.75.49.131
   Deny from 202.75.49.133
   Deny from 202.75.49.134
   # END worst-offenders
   Allow from all

6. From now, when you look at your “Worst Offenders” list in Akismet, you should see the option to ban the spamming IP addresses when you delete the messages.
7. Feedback and ask questions below

FAQ

1. Is this a polished and buffed plugin that’s ready for the prime time?No! Absolutely not! For many reasons. This is an experiment to see if the idea works and to generate some discussion around what’s needed for dynamic spam blocking systems to work. It’s public so that those with the right skills can try it, or examine it, and perhaps learn from or contribute to it. If you’re an armchair amateur blogger, this plugin is not for you; yet.
2. Why are some items in the list of offenders not ticked?Items with fewer than 4 spam messages are not ticked – this is flexible within the plugin, but not yet configurable.
3. Why do I only see 10 “Worst Offenders” at once?Items with fewer than 4 spam messages are not ticked – this is flexible within the plugin, but not yet configurable.
4. I have an idea for making this better, what should can I do?Discuss it, implement it, share it.
5. Does all this IP address checking add more load onto my poor server?The benefits far outweigh the costs. It’s a small increase at the front end, but a massive decrease overall. Comparing an IP address is mathematically simple task, so it is very fast. Storing the comment, sending it to the Akismet service and then removing it from the database is much more work.
6. Could this be an end to spam?No. The number of zombie machines out there is too large to block them all, this method reduces spam from zombies that know about your website, so it directly saves your resources whilst reducing the number of calls your server makes to the Akismet service.
7. How many machines can this method block?Currently IP addresses drop off the end of the list after 400 are added, so the least recent disappear. This is adjustable in the software but not user-configurable yet.
8. If I’m blocking spam at source, will capability of Akismet be diminished, because it might not see new variants of spam as they emerge?I don’t know. I doubt it. Maybe Matt can add detail without giving too much away.
9. Can it block legitimate comments from non-spammers?It’s possible, but improbable. In cases where the spammer comes through a proxy, the IP address of the proxy might get banned, so anyone attempting to connect to the site through that proxy would get a “403″. Similarly, in shared IP pools for dial-up users, it’s feasible (though highly improbable) that a spammer might dial up, spam, become banned and then hang up, relinquishing the IP address to the next user. If that user happens to visit your site then they’ll get a 403. The lower threshhold of ‘n’ spams from an IP address or a domain is there to decrease these possibiities, but it cannot negate the issue.
10. How long does the ban last?Currently there is a rather arbitrary limit of 400 IP addresses in a FIFO queue. When an address gets to the end it’s dropped off the list and is thus allowed to connect again.
11. I think I’m ok with .htacces files, but what if i’m not?Take a backup before you start: cp .htaccess .htaccess.bak in your wordpress root. Then if you want to revert rm .htaccess then cp .htaccess.bak .htaccess.
12. Can I revert to the vanilla Akismet plugin?Yes. No changes are made that affect the standard akismet functions, so you can swap back and forth by replacing the akismet.php file as many times as you like.
13. I want it, but I’m not an uber-geek, is there any hope?Yes. If you think it’s a useful idea, the most helpful thing you can do is blog about it. If people red your blog and like the idea then it will help generate interest. Interest generates ideas, which increase the likelyhood that this could turn into something really useful.
]]> http://boakes.org/akismet-htaccess-extension/feed/ 36 http://boakes.org/akismet-worst-offenders/ http://boakes.org/akismet-worst-offenders/#comments Sun, 04 Jun 2006 12:06:54 +0000 Rich http://boakes.org/akismet-worst-offenders This last few weeks the site has been very heavily hit by comment spammers hawking their usual reprobate websites and wasting internet bandwidth. Akismet has been doing a sterling job of catching this spam and not one message has made it onto the site (I wrote about Akismet’s effectiveness in the pre-launch testing previously). In the bad old days before Akismet I’d have to go through the “unmoderated comments list” in order to find the occasional real comment amidst all the spam, this is no longer necessary, which is wonderful. Comment-Spam-Nirvana has not been reached yet, however.



In order to help keep Akismet working well, and also, to ensure that “false positives” do not go unnoticed it is still necessary to trawl through the “spam list” and look for real comments. So although the problem has been turned on it’s head, the requirement on the responsible user is still the same.

The latest version of the Akismet plugin (v1.15) makes this “de-spamming” process easier, but it still leaves the poor website owner with the responsibility of looking at every single spam message in case there are any real comments that have been mistakenly marked as spam.

Informing the Akismet server about these false positives is important because it helps improve Akismet’s accuracy, which benefits everyone by ensuring fewer false positives – one hand washes the other, so to speak.

So I wrote a small addition to Akismet 1.15 (pictured above) that tries to help. It pre-processes the spam comments and identifies the worst offenders in terms of the domain that’s being advertised, or (perhaps more usefully) the IP Address of the spamming computer.

It’s not uncommon for me to get several hundred spam comments each day, so certain machines and websites are hitting my site many times. What the plugin does is make those worst offenders really obvious, so they can be removed en masse, reducing the ham-hunting to a smaller and more managable task.

Download it here: Akismet 1.15 plus Worst Offenders Extension. A stand alone version is available which works with newer versions of Akismet, see the discussion on the forum for more details.

]]> http://boakes.org/akismet-worst-offenders/feed/ 44 http://boakes.org/analysing-mynicemailatcom/ http://boakes.org/analysing-mynicemailatcom/#comments Sat, 05 Nov 2005 12:55:17 +0000 Rich http://boakes.org/?p=520 I’m hoping to spend some time over the next ‘n’ weeks understanding the data generated from the mynicemailat domain, though I’m a bit busy with real experiments at the moment, so I intend to do it by thinking aloud here as I get the opportunity.

First pass

I’ve quickly run off a couple of basic queries to get things going. All figures here are measured after search bots and referral and comment spammers have been removed from the data, and they’re based on the live log data that StatTraq gives me.

Total times the mynicemailat site was hit: 1729

Distinct IP addresses visiting mynicemailat.com : 1198

Total number of readers who arrived there by clicking on a spam comment: 1123

The gap between 1123 and 1198 is made up by people who arrived from search engines, and webmail providers.

So in terms of raw traffic, the amount generated was negligible, but, even with this small change, my Alexa traffic rank for this site briefly went into the 100,000′s, for about a week. It’s returned to the 500,000 mark now. What this shows is that my day-to-day visitors tend not to be users of the Alexa toolbar.

Questions

Please leave your analysis questions and suggestions here, and I’ll see what I can do.

]]> http://boakes.org/analysing-mynicemailatcom/feed/ 1 http://boakes.org/akismet/ http://boakes.org/akismet/#comments Tue, 25 Oct 2005 11:33:41 +0000 Rich http://boakes.org/?p=489 This website – like any website that allows readers to submit comments – receives comment-spam, usually advertising medicines, gambling, or other vices.

I’ve been trialling a new anti-comment-spam plugin since mid September. It’s called Kismet, it’s from Automattic, (hence Akismet for short) and it’s working very well.

Comment spam is more costly than email spam because it either wastes the time of the website owner, who has to remove it, or it wastes the time of every reader of the website who has to separate the wheat from the chaff.

It’s going to be launched tonight (i.e. Tuesday Afernoon, in Texas).

Effective Comment-Spam Relief

According to Matt Mullenweg, the curator of Automattic, there were “only a dozen or so” active users during the trial that I was involved in, and the system should “become more effective as more people use it”. The basic stats from my trial experience were as follows:



Message Count	Percentage of Total	Explanation
574	100%	The total number of comments this site received since the trial began.
425	74%	The number of those comments that were spam.
6	1.4%	The number of comments that had to be manually marked as spam.
1	0.17%	The number of comments incorrectly identified as spam by Akismet (a.k.a. false positives).

For people whose blog content is predominantly idle chatter, this plugin will remove the need for user moderation, allowing a far more interactive blogging experience between blogger and readers.

How it Works

It’s based on the principle that once a comment spam message is identified by one recipient, and corroborated by others, all similar messages can be marked as spam, reducing the spammer’s potential audience from thousands of people, to the few that report the message when it first arrives. It works, approximately like this…

When a comment is received by a website, it is checked against a worldwide database to see if it matches any messages that are known to be from spammers, this might be based on:

1. the IP addresses that the message originated from,
2. the web addresses being promoted,
3. a string in the content that can be matched by a regular expression,
4. or any other number of potential techniques that have not (yet) been disclosed.

Messages that are considered to be spam are automatically separated, and the moderator then has 15 days to check through them (in case there are any false positives) before they are removed forever. Spam comments are never visible on the site and the spam checking interface is very simple to use.



Threats to it’s Effectiveness

I’ve been thinking such a service should exist for a long time, so I’m keeping my fingers crossed that it stays effective. There are, however, several obstacles that may have to be overcome if it is to be a success.

Spam-Run Duration

Services such as this will change the delivery pattern of comment spam.

This will happen because the time window in which spam comments can get through will be drastically reduced to the very short period of time between start of the spam-run and the point when the spam has been identified and corroborated. In the short time before the spam is reported, messages can get through, so it is likely that spam-runs will become short, high volume bursts.

In lieu of this it may eventually be necessary to check each received message more than once, so that spam messages which are not immediately spotted when they are received can still be automatically discarded.

DDoS Target

The central server(s) may become a prime target for DDoS attacks if/when spammers realise that their spamming is no longer effective. The purpose of a DDoS attack would be to disable the automatic checking of comments, perhaps breaking the system and thereby letting their comments through.

It is likely that spammers would have to coordinate such an attack to coincide with a spam run, rather than relying on luck. The good news is that this would elevate the spammer from nuisance to criminal, so there are some very heavy legal books that can be thrown at anyone silly enough to try it.

Privacy

Some people may be concerned about the fact that every message they receive is sent to a third party for analysis. When one considers that these are supposed to be public comments on a public website, the privacy concern fades a little, but some people do still write private information in comments because the web is used by people, and people make mistakes, so it’s a concern that can’t be completely ignored.

One possible solution to both the Privacy and DDoS issues might be to provide replicated access to the Spam Database (probably on a registration only basis) so that there are multiple sites that could provide the service. Privacy concerns could be offset by enabling the website owner to select which service provider is used, or to provide their own service. Another possible solution to some privacy concerns would be the ability to mark some posts for manual checking only, thus ensuring message privacy.

Comment Censorship

What the service does, effectively, is silence individuals who are misusing the Web, however, there is potential for this capability to be misused, because it becomes feasible to mount a censorship attack on an individual or company – i.e. If you have the programming skill, it’s not too difficult to create a dummy message, mark it as spam and submit it to the service. If the spam-identification mechanisms are too sensitive or simplistic, then it may be possible to censor someone who hasn’t actually sent any spam.

Download & Installation

If you’re familiar with installing WordPress plugins, it’s all a simple process.

You can get the Akismet plugin already, it’s available from the open source software repository that manages all WordPress plugins.

You can also get it from the Akismet download page.

Installation is simple, just add the php file to your wp-content directory and enable it.

Enabling the plugin gets you 80% there, but you’re not done yet.

In order to protect itself against spammers who, Akismet uses an API key. You must obtain a key before the plugin will begin to work. The mechanism by which you can get your keys is what Automattic will launch later today.

Automattic for The People

As websites and personal publishing have flourished, comment spam propagated by a selfish few has become a significant problem for the masses. Akismet redresses the balance, at least for WordPress users.

By automatically curtailing spam publication, Akismet takes the wheels off the comment-spam gravy train. Hereafter, spammers will have to look for non-WordPress powered blogs to hawk their wares.

Akismet Launches

As akismet has launched, several other testers and early adopters have begin to comment on it, so if you’d like to read a little around the subject perhaps some of these musings will suffice:

• Craig Hartel also tested it.
• As did Michael Hampton.
• Elliott Back asks some good questions of Matt.
• Ben Gillbankshas turned off all other anti-spam measures (FWIW I have too).
• Oskar Syahbana has just installed it prior to going on a break – so any spam that gets through will show up on his blog, a baptism by fire.
• Scott Yang thinks up some good reasons for, and against, using it.
• Tack Mackenzie also immediately mentions the privacy issue.
• Chétan Kunte has just enabled it and had the good sense to ask readers to mention if comments are not getting through
• Eric Setiawan thinks we might be able to forget that spam was ever a problem.
• Aaron Brazell had some misgivings, but these seem to have helped clarify that Akismet really is free.
• N. Godbout was unsure about why it’s relevant to people who don’t blog on wordpress.com.
• Ryan Kennedy wasn’t initially keen on the default 15 day spam retention policy, and highlights the need for a config panel.
]]> http://boakes.org/akismet/feed/ 32 http://boakes.org/my-nice-spam-domain/ http://boakes.org/my-nice-spam-domain/#comments Sat, 17 Sep 2005 11:00:14 +0000 Rich http://boakes.org/?p=481 Hopefully I just spoiled a spammer’s whole week. How? I bought the domain that he’s trying to promote (MyNiceMailAt.com) before he did.

When I receive comment spam which doesn’t obviously link to a gambling, pornographic or pharmaceutical site then I usually do a little investigation to see what’s on the site, who owns it, why they’re spamming me, etc.

In the last 24 hours I’ve been hit by comment spam promoting MyNiceMailAt.com.

1. I tried to look at mynicemailat.com, and it didn’t exist; so
2. I tried to find the domain ownership records, and they didn’t exist either.

MyNiceMailAt.com was an unregistered domain, being promoted by a spammer.

1. I like to do my bit for hindering spammers; so
2. I bought the domain, before the spammer could; so
3. the entire spam run has been a waste of the spammer’s time and resources.

Useful Info

• If you’d like to learn more about comment spam, I highly recommend Ann Elisabeth’s SpamHuntress blog.
• This is not the first time such action has been taken by a spam recipient, last year jagk.com was similarly snapped up, and now has a regularly updated spam blacklist that you can add to your .htaccess file (if you don’t run your blog server, tell your administrator about it).
]]> http://boakes.org/my-nice-spam-domain/feed/ 42 http://boakes.org/spam-indirection/ http://boakes.org/spam-indirection/#comments Thu, 25 Aug 2005 07:30:24 +0000 Rich http://boakes.org/?p=455 I think this may be a mildly new twist on email and referral spam: using referral spam to advertise a message that’s stored in an unused public newsgroup, thus avoiding spam filters.

This morning I checked the stats on this website, where there was a referring link from:

http://health.groups.yahoo.com/group/weight-loss-group/message/2


“Now”, I wondered aloud, “why would a health group be linking to me?”.

So I followed the link to see what the association was, and LO, it came unto me like a shining beacon:

I need to buy weight loss pharmaceuticals!

The indirection is as follows:

1. Create a web based newsgroup
2. Leave your advertising messge on the group
3. Request a couple of web pages from many thousands of websites.
4. Sell drugs to desparate victims who see and believe the message*
5. Profit

* The key to this working is that many websites include a list of the latest referrers, so the site’s users will potentially click these referrers and see the message. It’s a bit desparate but if you can get the link in front of a million eyeballs and only 0.0001% click and buy, then that’s 100 customers, or perhaps more appropriately, 100 victims who hand over their credit card details.

The real killer blow, the thing that makes this technique work, is that most referral spam blocking algorithms block the whole domain which the spam advertises. In this case the domain is yahoo.com. I could potentially block health.groups.yahoo.com but my software won’t do that by default, I’d need to tweak it. This technique is just the spammers exploiting the tiniest of loopholes to get their advertising across in the time before the hole is plugged.

Blacklisted

Incidentally, the incoming connections are from two IP addresses, 80.77.84.108 and 80.77.86.209 – the second of which is already blacklisted By Mat Sullivan’s realtime blacklist SORBS.

]]> http://boakes.org/spam-indirection/feed/ 3 http://boakes.org/latest-referral-spam-domains/ http://boakes.org/latest-referral-spam-domains/#comments Fri, 10 Jun 2005 09:33:47 +0000 Rich http://boakes.org/?p=291 Referral and Comment spam continues unabated, with several new domains this week, and several new machines doing the spamming. Here I present two regularly updated URl’s that list the domains and ip addresses of the machines that are hitting me, and the two scripts that generate them.

This first list shows the 10 most recently discovered referral spam domains that have attempted to get a link from my stats pages: http://boakes.org/referral-spam-domains

This list shows the 10 most recently discovered machines that have sent comment spam me: http://boakes.org/comment-spam-machines

To make the lists easy to use in scripts (e.g. for automatically keeping blacklists up to date) both are in plain text format, with one item per line. The list terminates with a blank line.

If other people would like to use the same script on their WordPress blog then there’s great potential for a federated blacklist – many hands make light work.

Suggestions for scipt improvements are welcomed; and please let me know if you’re using either of them.

Comment Spam Machines

This script will work with any wordpress installation since it uses the standard wordpress comments table. You can copy the script below or download the latest version.

The machines that have most recently spammed this site are:

require_once('wp-config.php');
global $wpdb;
$q="SELECT min(comment_date) as mindate, comment_author_ip FROM wp_comments where comment_approved='spam' group by comment_author_ip order by mindate desc limit 10";
$lines = $wpdb->get_results( $q );
foreach ($lines as $line) {
echo( “” . $line->comment_author_ip . “\r\n”);
}?>

Referral Spam Domains.

This script is more limited in it’s use (currently) because it pulls it’s data from a table of spam daomains that is maintained by a StatTraq extension. Hopefully with the next relewase of StatTraq this will become more mainstream. You can copy the script below or download the latest version.

The domains that have most recently attempted to gain referrals from this site are:

require_once('wp-config.php');
global $wpdb;
$q="select domain from rjb_spamdomains order by id desc limit 10";
$lines = $wpdb->get_results( $q );
foreach ($lines as $line) {
echo( “
1. ” . $line->domain . “

“);
}
?>

Current Referral Spam Machines

I’ve not scripted this one separately (contact me if it would be useful) but here’s a list of the machines that have engaged in referral spam of this site during the last 24 hours.

require_once('wp-config.php');
global $wpdb;
$q="select ip_address from wp_stattraq where spam=1 and DATE_SUB(CURDATE(),INTERVAL 1 DAY) < access_time group by ip_address order by line_id desc";
$lines = $wpdb->get_results( $q );
foreach ($lines as $line) {
echo( “
1. ” . $line->ip_address. “

“);
}
?>

]]> http://boakes.org/latest-referral-spam-domains/feed/ 2 http://boakes.org/curious-referrals/ http://boakes.org/curious-referrals/#comments Sun, 15 May 2005 21:27:21 +0000 Rich http://boakes.org/curious-referrals An article on SpamHuntress.com got me thinking laterally about some of the referral stats I’m seeing, and I’ve started to notice something curious.

I’m seeing referrals from what appear to be valid blogs (i.e. I check my inbound connections to see if they’re spammers or not, and some of them are valid – if they were from spammers they’d be stripped from the stats) and yet, when I look within the page which is supposed to actually link here, there is no link. This is curious.

So, a hypothesis: these links could be down to a referral spammer who’s not happy that I’ve mentioned their lame antics in other posts.

I’m guessing the spammer could put a comment or trackback on someone elses blog and use a url from my site in doing so, but not put down an email address that’s mine, so I’d not be notified upon acceptance or rejection of the comment.

I’m further presuming the content of the message would then be moderated by the site owner, in some cases generating a clickthrough to my site, and the message subsequently removed since it would be apparent that it’s not from me.

So I’d see the referral, but not the deleted link.

This will, of course, be proved or disproved if others are seeing such a curious phenomenon, or can come up with an explanation; or if anyone who’s receiving such a message/trackback finds this post and confirms my suspicion.

]]> http://boakes.org/curious-referrals/feed/ 3 http://boakes.org/comment-spam-rip/ http://boakes.org/comment-spam-rip/#comments Wed, 06 Apr 2005 15:42:24 +0000 Rich http://boakes.org/web-spammers-losing-revenue A couple of years ago, a web page with an open comment form would have been used for commenting intelligently and considerately on the subject at hand or sending a message to the page author. Then, with the advent of Google’s page rank system, comment forms became the subject of massive misuse, because pagerank gave a higher rank to web pages based on the number of times they appeared on other sites. Spammers would use automated tools to mercilessly link and relink their sites on any and every open form. Having a high pagerank means coming first in search results, which for a commercial site equates to more sales: so comment spamming had rich rewards.

To combat this, web page authors have built all kinds of systems into their web sites, but none have been universally effective. The recent introduction of the rel=”nofollow” directive may be paying off, and the days of the web spammer may be numbered.

For non-tech-heads rel=”nofollow” tells Google and other search engines that the owner of a web page site does not recommend the linked page – it’s not a negative recommendation, but it’s not a positive recommendation either.

Just now I saw a comment that got through all the automated spam killers that I have on this site. It said:

Great site http://example.com <a href=”http://example.net” rel=”nofollow”>example</a> [url=http://example.org]example[/url]

This is interesting because it shows that web spammers are having to worry about the best tactics to get themselves listed, where previously they would be able to spam just about any web page with an open comment form. Here the spammer was testing my site (and probably several thousand others at the sime time) to work out if it’s possible to get a link displayed, using three common techniques.

Sorry spammer, your days of revenue are limited. Go and get a job and become a worthwhile part of society.

]]> http://boakes.org/comment-spam-rip/feed/ 0 http://boakes.org/extending-stattraq-and-spam-control/ http://boakes.org/extending-stattraq-and-spam-control/#comments Fri, 01 Apr 2005 09:20:28 +0000 Rich http://boakes.org/extending-stattraq-and-spam-control I spent a while chatting with StatTraq author Randy Peterman over the last few days, we’re both interested in how to remove (or hide) referral spam from site usage statistics in order that they may be a realistic and meaningful reflection of actual human visitors.

After a little hacking I’m pleased to say that I now have a version of StatTraq that can automatically identify referral spam as it arrives. Every hit is still still added to the database, but hits from spammers are marked as such, which means that when the stats are viewed, spam can be excluded.

Interestingly this also means that spam statistics can also be analysed.

I’m going to be working with Randy (as time permits us both) in order to get these additions integrated into the main release.

I already have a cleanup-script so existing stat-traq users will be able to cleanse their database and not lose any of their existing data about visitors or spammers.

Beta testers, developers & people with requests and suggestions ; please leave a message.

]]> http://boakes.org/extending-stattraq-and-spam-control/feed/ 5 http://boakes.org/15-new-referral-spam-domains/ http://boakes.org/15-new-referral-spam-domains/#comments Thu, 17 Mar 2005 23:25:19 +0000 Rich http://boakes.org/15-new-referral-spam-domains Over the last few days the referral spam tide has ebbed and flowed: for about a week, everything that came in was pointing to doobu.com, then today, there was a definite change in tactic as 15 new poker related domains (see the existing story on referral spammers for the updated table of details) were added to the ever growing list.

I’m starting to wonder if it’s worth creating an RSS feed that would update as I discover each new referral spammer. This might then provide a viral trust network so that queries could be rejected based on the referrer – as soon as a spammer is positively identified, any requests featuring it would just get Forbidden. I already have a db table in my StatTraq MySQL server, so knocking up an RSS generator should be a doddle. Raise your hand with a comment anyone who’d like to encourage me to develop it, or I’ll just keep commenting idly.

]]> http://boakes.org/15-new-referral-spam-domains/feed/ 4 http://boakes.org/new-referral-spam-tactic/ http://boakes.org/new-referral-spam-tactic/#comments Wed, 09 Mar 2005 17:10:28 +0000 Rich http://boakes.org/new-referral-spam-tactic Today I’ve stopped getting hit by named domains for referral spam. At first glance this sppears to be good news.

Instead, it appears that I’m being referral-spammed by an un-used IP address; perhaps in the hope that when it is associated with a DNS entry at some point in the future, my logging software will kindly run a Reverse DNS lookup, and thus the offending site will get it’s linkage.

This may also be related to rel=”nofollow”, in an attempt to find a loophole in the code which ignores specific domains. Or an attempt to avoid search engines blacklists.

It certainly seems a bit different, but all the other signs point to the same spambots as in all the existing spam. The referrer, for the record is http://12.163.72.13/.

]]> http://boakes.org/new-referral-spam-tactic/feed/ 1 http://boakes.org/another-light-analysis-of-referral-spammers/ http://boakes.org/another-light-analysis-of-referral-spammers/#comments Wed, 02 Feb 2005 08:59:09 +0000 Rich http://boakes.org/another-light-analysis-of-referral-spammers The “Account Terminated” swamp got a little muddier over the last few days. It appears that those of us who’ve started to blog about what we’re finding are starting to get a lot of hits from others who are being targetted, and the scammers effectiveness is therefore being reduced as folk realize that the original message is just a ruse.

The scammer, in an attempt to stay one step ahead has now added some new messages to his repartee, the list new includes:

1. Account Terminated
2. Hosting service temporarily unavailable
3. – Suspended until further notice –
4. “” [an empty string]
5. Account locked
6. Service Unavailable!

Genius.

What’s more, the opening line now has all kinds of alternative forms too:

1. The Following strange article is term of service closing.
2. The entire file is now in closing order
3. The Following explanation is in suspension
4. This former info is currently under investigation
5. This entire rumor is currently in closing order

Also, the scammer has now added an official looking feedback form to the page that even includes a captcha code, just to add an air of authenticity and professionalism. Make no mistake, I’m seeing hundreds of referrals every week from this URL – this is not an above board operation.

Here’s the killer blow, which illustrates why we’re all seeing this spam – the google pagerank for psxtreme.com is currently 7/10 – the site has no content, no history of service, and yet it’s pagerank is at a level that will push it almost to the top of any web search. If you’re playing a percentage game with (for example) people who fancy a quick 10 minutes gambling on the net, then this is the thing that gets your site visited more than others. You have to annoy a lot of people to achieve it, but those who seek personal wealth don’t often see the bigger picture.

]]> http://boakes.org/another-light-analysis-of-referral-spammers/feed/ 0 http://boakes.org/referral-spammers/ http://boakes.org/referral-spammers/#comments Thu, 20 Jan 2005 10:09:59 +0000 Rich http://boakes.org/referral-spammers So; more tales from the “Account Terminated” swamp. I’m looking through my referrer spam from the last 24 hours and I decide to do a little whois checking.

Update: A regularly updated text file containing the 10 most recent domains to hit my servers is available here: http://boakes.org/referral-spam-domains

A few things I noticed:

1. Owner Jane Phill lives at “61 Street, Tel Aviv, Israel, IL, 49992″ according to the reachcasino.com information but at “61 Street, NYC NY, US, 10024″ according to the goapplyonline.com information. It’s a pretty safe bet that this is therefore a false address.
2. Owner Thomas Reece lives at “249 W 89 Street, NYC, NY, US, 10024″ according to the crepesuzette.com listing; a nicely similar address to Jane Phill’s false NYC address (and now also Randy Bill’s).
3. online-deals-4u.info has a different owner, but look (!) the admin contact is (as in the other cases) a “contact##@support-24×7.biz” entry – AND the IP address resolves to the same machine.
4. The admin contact for every one of these domains is an email address at “team-support-24×7.net” which is owned by a “Monika Stanes” (MS2183-GANDI) the team-support-24×7.net domain is administrered by gandi.net.
5. Cheat-Elite.com has a different owner, but just happens to resolve to the same address as Jane Phill’s domains – and hey look, it’s administered by good old team-support-24×7.net.
6. Try actually visiting the administrators web site… www.support-24×7.biz, team-support-24×7.net, support-4u.net, top-support.net – spot the common thread? Yep – it’s gandi.net. the one exception here is “marketing-support.info which currently has no DNS entry, so a quick whois lookup on them and (ahem) bingo it’s spammers registrar of choice gandi.net, again.
7. The server IP address for freakycheats.com (Thomas Reece) is the same as for most of Jane Phill’s domains, so as of 2005-02-02 we can see a definite link between these false identities.
8. As of St Patrick’s Day 2005 our old friend Jane Phill has turned up again, and now she’s working for “Marketing Ltd” and has registered 15 new poker-related domains.

About the Servers

It’ s possible to find out a little more about the servers which are being redirected to (rather than just the whois information on the domain name)…

219.150.118.16	The most prevalent IP address which is associated with several “domain owners” is assigned by CHINATELECOM-ha.
216.171.143.122	This one appears to be US based.
161.58.59.8	This one’s a Verio Customer.
64.234.220.141	And this little piggy is hosted by WebStream in the USA.

The table of ne’er-do-well’s and their domains.

Website	Owner	Admin Contact	IP resolves to…	Date of check
1st-advantage-credit-repair.com	Jane Phill	Sandra, Drake (NIC-13522) contact76@support-24×7.biz	219.150.118.16	2005-01-20
123-home-improvement-equity-loans.com	Jane Phill	Sandra, Drake (NIC-13522) contact76@support-24×7.biz	219.150.118.16	2005-01-20
acceptcreditcardsrealtime.com	Jane Phill	Sandra, Drake (NIC-13522) contact76@support-24×7.biz	219.150.118.16	2005-01-20
all-calmortgage.com	Jane Phill	Sandra, Drake (NIC-13522) contact76@support-24×7.biz	219.150.118.16	2005-01-20
alumnicards.com	Jane Phill	Sandra, Drake (NIC-13522) contact76@support-24×7.biz	219.150.118.16	2005-01-20
credit-cards-credit-cards-credit-cards.net	Jane Phill	Sandra, Drake (NIC-13522) contact76@support-24×7.biz	219.150.118.16	2005-01-21
creditsharpie.com	Jane Phill	Sandra, Drake (NIC-13522) contact76@support-24×7.biz	219.150.118.16	2005-01-20
exclaim4creditcardprocessingmerchantaccount.com	Jane Phill	Sandra, Drake (NIC-13522) contact76@support-24×7.biz	219.150.118.16	2005-01-20
fast-cash-quick-money-easy-loan.com	Jane Phill	Sandra, Drake (NIC-13522) contact76@support-24×7.biz	219.150.118.16	2005-01-21
goapplyonline.com	Jane Phill	Sandra, Drake (NIC-13522) contact76@support-24×7.biz	219.150.118.16	2005-01-20
hasslerenterprises.net	Jane Phill	Sandra, Drake (NIC-13522) contact76@support-24×7.biz	219.150.118.16	2005-01-20
home-equity-loans-mortgage-refinancing.com	Jane Phill	Sandra, Drake (NIC-13522) contact76@support-24×7.biz	219.150.118.16	2005-01-20
internet-merchant-account-pro.com	Jane Phill	Sandra, Drake (NIC-13522) contact76@support-24×7.biz	219.150.118.16	2005-01-20
lowest-interest-rate-credit-cards-online.com	Jane Phill	Sandra, Drake (NIC-13522) contact76@support-24×7.biz	219.150.118.16	2005-01-20
lowinterestratecreditcards.net	Jane Phill	Sandra, Drake (NIC-13522) contact76@support-24×7.biz	219.150.118.16	2005-01-20
mortgagemarketinginc.com	Jane Phill	Sandra, Drake (NIC-13522) contact76@support-24×7.biz	216.171.143.122	2005-01-20
mortgagequestaz.com	Jane Phill	Sandra, Drake (NIC-13522) contact76@support-24×7.biz	219.150.118.16	2005-01-20
ps2cool.com	Jane Phill	Sandra, Drake (NIC-13522) contact76@support-24×7.biz	219.150.118.16	2005-01-21
reachcasino.com	Jane Phill	Sandra, Drake (NIC-13522) contact76@support-24×7.biz	219.150.118.16	2005-01-20
repaircreditonline.net	Jane Phill	Sandra, Drake (NIC-13522) contact76@support-24×7.biz	219.150.118.16	2005-01-20
tecrep-inc.net	Jane Phill	Madisyn, Trevin (NIC-15252) contact4@support-24×7.biz	219.150.118.16	2005-01-25
chat-nett.com	Phill Davis	Madisyn, Trevin (NIC-15252) contact4@support-24×7.biz	219.150.118.16	2005-02-04
terashells.com	David Lee	Daisy, Meghan (NIC-14050) contact26@support-24×7.biz	219.150.118.16	2005-02-04
911easymoney.com	Thomas Reece	Brycen, London (NIC-17655) contact70@team-support-24×7.net	161.58.59.8	2005-01-20
condodream.com	Thomas Reece	Reece, Thomas (NIC-21871) contact100@team-support-24×7.net	64.234.220.141	2005-01-20
crepesuzette.com	Thomas Reece	Reece, Thomas (NIC-21871) contact100@team-support-24×7.net	161.58.59.8	2005-01-20
flafeber.com	Thomas Reece	Reece, Thomas (NIC-21871) contact100@team-support-24×7.net	64.234.220.141	2005-01-20
freakycheats.com	Thomas Reece	Reece, Thomas (NIC-21871) contact100@team-support-24×7.net	219.150.118.16	2005-02-02
mediavisor.com	Reece, Thomas	Reece, Thomas (NIC-21871) contact100@team-support-24×7.net	64.234.220.141	2005-01-20
royalmailhotel.com	Thomas Reece	Sandra, Drake (NIC-13522) contact76@support-24×7.biz	64.234.220.141	2005-01-20
spoodles.com	Thomas Reece	Reece, Thomas (NIC-21871) contact100@team-support-24×7.net	64.234.220.141	2005-01-20
sportsparent.com	Thomas Reece	Reece, Thomas (NIC-21871) contact100@team-support-24×7.net	161.58.59.8	2005-01-20
stmaryonline.org	Reece, Thomas	Reece, Thomas (moniker21871) contact100@team-support-24×7.net	64.234.220.141	2005-01-20
cheat-elite.com	Susan Hanes	Gloria, Elisabeth (NIC-17237) contact77@team-support-24×7.net	219.150.118.16	2005-01-20
online-deals-4u.info	Angie Ashanti	Angie Ashanti (C6716231-LRMS) contact11@support-24×7.biz	219.150.118.16	2005-01-20
rulo.biz	Philip Ivan	Philip Ivan (MONIKER15251) contact3@support-24×7.biz	219.150.118.16	2005-01-22
best-buy-site-4u.info	Rogelio Victor	Rogelio Victor (C6717792-LRMS) contact95@support-24×7.biz	219.150.118.16	2005-01-24
psxtreme.com	Randy Bill	Dalia, Rylee (NIC-15667) contact1@marketing-support.info	219.150.118.16	2005-02-02
yelucie.com	Robert	Graham, Harry (NIC-18451) contact62@support-4u.net	219.150.118.16	2005-02-06
crescentarian.net	Dan	Titus, Taryn (NIC-20220) contact49@top-support.net	219.150.118.16	2005-02-06
6q.org	Ernest Darius	Ernest Darius (moniker16004) contact66@marketing-support.info	219.150.118.16	2005-02-10
smsportali.net	Lee	Michaela, Adan (NIC-15253) contact5@support-24×7.biz	219.150.118.16	2005-02-10
future-2000.net	Jim Fox	Leonel, Morgan (NIC-21487) mail29@support-2000.net	219.150.118.16	2005-02-12
ronnieazza.com	Susan Lee	Evelin, Porter (NIC-14080) contact56@support-24×7.biz	219.150.118.16	2005-02-12
highprofitclub.com	Kareem Adrienne	Adrienne, Kareem (NIC-10459) karadr56@tech-corner.us	67.184.17.116	2005-03-06
doobu.com	Jaylene Nicolette	Nicolette, Jaylene (NIC-13114) contact11@support-24×7.biz	67.184.17.116	2005-03-06
poker-tables-best-deals.com	Marketing Ltd	Phill, Jane (NIC-8754) contact61@support-4u.net	209.59.165.114	2005-03-17
texas-holdem-poker-now.com	Marketing Ltd	Phill, Jane (NIC-8754) contact61@support-4u.net	209.59.165.114	2005-03-17
poker-online-anytime.com	Marketing Ltd	Phill, Jane (NIC-8754) contact61@support-4u.net	209.59.165.114	2005-03-17
pacific-poker-top-place.com	Marketing Ltd	Phill, Jane (NIC-8754) contact61@support-4u.net	209.59.165.114	2005-03-17
how-to-play-poker-quick.com	Marketing Ltd	Phill, Jane (NIC-8754) contact61@support-4u.net	209.59.165.114	2005-03-17
poker-hands-secrets.com	Marketing Ltd	Phill, Jane (NIC-8754) contact61@support-4u.net	209.59.165.114	2005-03-17
poker-games-top-ranked.com	Marketing Ltd	Phill, Jane (NIC-8754) contact61@support-4u.net	09.59.165.114	2005-03-17
world-series-of-poker-1996.com	Marketing Ltd	Phill, Jane (NIC-8754) contact61@support-4u.net	209.59.165.114	2005-03-17
poker-rules-easy-4u.com	Marketing Ltd	Phill, Jane (NIC-8754) contact61@support-4u.net	209.59.165.114	2005-03-17
wsop-allabout.com	Marketing Ltd	Phill, Jane (NIC-8754) contact61@support-4u.net	209.59.165.114	2005-03-17
free-poker-great-value.com	Marketing Ltd	Phill, Jane (NIC-8754) contact61@support-4u.net	209.59.165.114	2005-03-17
free-texas-hold-em-best-deals.com	Marketing Ltd	Phill, Jane (NIC-8754) contact61@support-4u.net	209.59.165.114	2005-03-17
party-poker-leading-site.com	Marketing Ltd	Phill, Jane (NIC-8754) contact61@support-4u.net	209.59.165.114	2005-03-17
samiuls.com	Mark Ltd	Phill, Jane (NIC-8754) contact61@support-4u.net	209.59.165.114	2005-03-17
world-poker-tour-1998.com	Marketing Ltd	Phill, Jane (NIC-8754) contact61@support-4u.net	209.59.165.114	2005-03-17
vpshs.com	Phill	Phill, Jane (NIC-8754) contact61@support-4u.net	219.150.118.16	2005-03-18
ca-america.com	Jane Ltd	Dario, Ashlee (NIC-16233) contact43@marketing-support.info	67.184.17.116	2005-03-19
vrajitor.com	James	Katy, Kyra (NIC-18205) contact5@support-4u.net	219.150.118.16	2005-03-21
ro7kalbe.com	Bill Ltd	Jaylon, Juan (NIC-16724) contact53@marketing-support.info	219.150.118.16	2005-03-21
bnetsol.com	Bob Ltd	Lilliana, Meredith (NIC-13722) contact78@support-24×7.biz		2005-03-21
registrarprice.com	Sam	Sarina, Emma (NIC-14805) contact84@support-24×7.biz		2005-03-21
buy-2005.com	Margo	Emilee, Lindsey (NIC-15936) contact81@marketing-support.info		2005-03-22
buy-2005-top.com	Bill owe	Jacob, Jarvis (NIC-17691) contact3@team-support-24×7.net		2005-03-24

]]> http://boakes.org/referral-spammers/feed/ 9