Lunacy Unleashed

Notes from the field in the War on Spam

Bad Behavior 2.0.9

Make a Donation.

Bad Behavior 2.0.9 has been released. It is a strongly recommended upgrade for all users.

This release is likely the final release in the 2.0 series as I make a major change in the development process; see below for details on this change.

This release addresses a further set of “false positive” reports received from various users which affect some uncommon circumstances.

New in this release (since 2.0.8):

  • A workaround has been placed for a problem with the Clearswift Web Policy Engine. Users behind this proxy server are no longer blocked.
  • A workaround has been placed for a bug in the LiveJournal OpenID process which Six Apart refuses to fix. Logins using OpenID will no longer fail.
  • A workaround has been placed for bugs in some versions of Internet Explorer and Safari web browsers which caused them to be blocked after leaving a comment on WordPress. These requests are no longer blocked.
  • A spam prevention feature was causing users to be blocked from their own blogs when they also subscribed to their own feed, or when they accessed the site with multiple web browsers at the same time; it has been disabled for rework.

Download Bad Behavior now!

The 2.0 series of Bad Behavior will be maintained as a legacy branch, with only bug fixes, false positive fixes and security fixes applied to this branch, if any such fixes are needed. No new checks for spammers will be added.

Shortly I will introduce a “development” 2.1 series on a much shorter development cycle, with days or perhaps even hours between releases. In this branch I’ll be experimenting with new spam prevention features, rolling them out quickly and rolling back quickly in case of actual trouble. I’ll also be rolling out a new packaging method which I’ve discussed previously, that will make Bad Behavior even more platform-independent than it currently is, and allow for the “core” to be updated separately from the “glue” which connects it to your host platform.

Once features prove themselves through development and testing to be stable, they’ll be rolled forward into a “stable” 2.2 series, intended for those users who are averse to the risks of blocking legitimate users or having the occasional crash. While I work very hard to ensure that every release, however labeled, does not crash, and does not generate false positives, things occasionally happen which are outside my control.

This parallel development scheme will help balance the needs of the two primary groups of Bad Behavior users.

The first group needs enterprise-grade code which ideally never blocks a single legitimate request and can quickly be rolled into production environments with a high degree of confidence. The tradeoff is the same as it has always been: to prevent any chance of false positives, Bad Behavior’s stable branch will permit some spam, anywhere from 0.1% to 10%, to pass through, and will require a backup solution such as Akismet. Even so, it will drastically reduce the amount of time and money spent managing spam, especially for deployments of dozens or hundreds or thousands of sites.

To serve this class of users more effectively, I’m also studying the feasibility of offering support contracts for enterprise users of Bad Behavior. Services offered under such contracts might include installation assistance, on-call support, hotfix development and deployment, and per-incident support. If your organization may need such a service, stay tuned for more details in the near future.

The second group, I believe, is the majority of Web sites: those for whom a rare blocked user is merely an annoyance rather than a critical problem, and who have much lower tolerance for spam because they aren’t being paid to manage their own blogs, wikis and forums. As much as possible, Bad Behavior’s development branch will limit spam for this class of users to 0.5% missed. The tradeoff is that you will be asked to do what you already do: to report any problems you encounter, whether they be missed spam or blocked users or plain old crashes.

And for users who would like to have their cake and eat it too, the development and stable versions will be installable side-by-side on the same site, and you will be able to switch back and forth between them at the click of a button.

Finally, prior to the first stable 2.2 release, I will be reworking all of Bad Behavior’s documentation and moving Bad Behavior from its current home to a new site dedicated solely to Bad Behavior. So you all will have to update your feed URLs to the new location soon. (Mailing list readers won’t have to do anything.)

In the meantime, Bad Behavior remains a user-supported project, with all code released under the GNU General Public License. If you find Bad Behavior valuable, please consider making a financial contribution. I develop Bad Behavior in my limited spare time, and every contribution means I can devote more time to its development.

January 8, 2007 Posted by | Bad Behavior, Blog Spam, LifeType, MediaWiki, Spam, WordPress | Comments Off on Bad Behavior 2.0.9

Bad Behavior 2.0.8

Make a Donation.

Bad Behavior 2.0.8 has been released.

This version contains updates for various “false positive” reports and is recommended for all users.

Updated in this release (since 2.0.7):

  • Verizon Wireless EV-DO users are no longer blocked.
  • Blocked requests will be subject to a two-second delay before a response is sent. (See below.)
  • Some blackhole lists previously used in Bad Behavior have been scaled back or removed.
  • The address for the Bad Behavior Blackhole has been added. (See below.)
  • Some new spambots have been identified and blocked.

In recent days spam attacks have been on the rise, with one especially obnoxious bot delivering requests so fast that some sites have been taken offline by them. While the requests aren’t especially numerous or resource-intensive, the most common software used by Web hosting providers is very inefficient at serving dynamic pages such as PHP-based Web sites. So even a moderate number of requests can take a whole server down, or lead the hosting provider to take the site down before the whole server goes down.

Bad Behavior now counters this by introducing a short two second delay to blocked requests, before the HTTP response is sent. Since most spambots wait for the response before going on to the next request, this should sufficiently slow down most of the overly aggressive spambots and give Web site operators some breathing room. While I would have liked to put in a delay of a minute or more, there remains the slight chance that an actual human being would be blocked, and they should be able to get a response back in a reasonable time.

With respect to realtime blackhole lists, all of the existing lists target e-mail spam, and since spambots who send link spam are almost always also sending e-mail spam through the same servers, these are a fairly effective means of blocking link spam. However, since they target e-mail spam, they also block legitimate users. The primary issue here is that while an IP address may be added to a blackhole list quickly, it is not removed quickly — or at all — once the spam stops. Thus, people with dynamic IP addresses are unfairly blocked because some other customer was sending spam.

Bad Behavior Blackhole, which should go online within the next few weeks, is designed specifically for link spam. It adds IP addresses to its database quickly when actual spam is received, and in addition, drops the IP addresses once the spam stops. This helps prevent dynamic IP customers from being blocked because another user’s computer was sending spam. Once Bad Behavior Blackhole is online, all other realtime blackhole lists will be dropped from Bad Behavior.

Download Bad Behavior now!

As always, if you find Bad Behavior valuable, please consider making a financial contribution. I develop Bad Behavior in my spare time, and every little bit means I have more spare time to devote to its development.

And don’t forget to subscribe to the RSS feed or the mailing list. (They’re the same content.)

December 15, 2006 Posted by | Bad Behavior, Blog Spam, Drupal, ExpressionEngine, LifeType, MediaWiki, Spam, WordPress | Comments Off on Bad Behavior 2.0.8

Spam Surge

In the last two days I’ve seen a tenfold increase in the amount of spam being delivered, both that is being blocked, and isn’t being blocked, by Bad Behavior.

While the spam started around the same time as I released Bad Behavior 2.0.7 yesterday, there doesn’t appear to be any correlation between the two. I inspected a few of the spams and they seem like the same old stuff, just cranked into very high gear.

I’ve personally seen over 3,000 spam attempts in the last day, with over 200 missed. This is spam that Bad Behavior is not yet capable of catching without blocking legitimate users as well.

This is why I have been working on the Bad Behavior Blackhole, in order to identify and block spam by its sources, wherever they are.

The Bad Behavior Blackhole is a feature that, once fully up and running, can identify known sources of blog spam and wiki vandalism and pre-emptively block them without affecting legitimate users.

Unfortunately, time constraints have not permitted me to put in much work on Bad Behavior Blackhole, as I’ve had to work on things which bring in revenue. As I’ve said before, while tens of thousands of people use Bad Behavior, only a few dozen have ever actually contributed back.

If you find Bad Behavior valuable, and you want to see this project up and running sooner rather than later, please contribute to its further development.

Thank you in advance for your support.

Update: Slashdot has coverage of the massive spam increase, which is hitting e-mail spam as well.

November 7, 2006 Posted by | Bad Behavior, Blog Spam, MediaWiki, Spam, WordPress | Comments Off on Spam Surge

Bad Behavior 2.0.7

Make a Donation.

Bad Behavior 2.0.7 has been released.

I’ve got a nice roundup of bug fixes this time around. Most people should upgrade right away to take advantage of the fixes and additional spam protections.

New in this release (since 2.0.6):

  • A bug (apparently in MediaWiki) which caused blank lines to appear on rendered pages in MediaWiki 1.7 has been worked around.
  • In version 2.0.6 four blackhole lists were added and incoming POST requests screened against them. Two of these lists generated significant hits in which primarily non-U.S. users who hadn’t actually sent any spam were being blocked. They have been removed. (Once Bad Behavior Blackhole is up and running, the other two may be removed as well, even though they’re performing fine.)
  • Two tests which catch some spambots and content thieves which were present in Bad Behavior 1 were inadvertently dropped from Bad Behavior 2. One of the hazards of rewriting something from the ground up. These tests have been restored.
  • A check which blocks users behind a Microsoft ISA Server 2004 proxy server, and one other type of proxy server I forget the name of, has been moved to strict mode only. Please disable strict mode if you are expecting traffic from such a source, and please contact Microsoft for a hotfix, if they’ve bothered to fix the bug in their software.
  • An additional IP address range for Google has been whitelisted. If you ever receive spam from an IP address owned by Google, please notify me immediately.
  • Several additional spambots have been identified and blocked.

Download Bad Behavior now!

As always, if you find Bad Behavior valuable, please consider making a financial contribution. I develop Bad Behavior in my spare time, and every little bit means I have more spare time to devote to its development.

And don’t forget to subscribe to the RSS feed or the mailing list. (They’re the same content.)

Also please note: Due to excessive levels of spam here on wordpress.com, which doesn’t use Bad Behavior anymore, I’ve had to close comments and pings entirely. You can reach me at badbots@nospam.ioerror.us without the nospam.

November 6, 2006 Posted by | Bad Behavior, Blog Spam, Blogging, Spam, WordPress | Comments Off on Bad Behavior 2.0.7

Bad Behavior 2 for Drupal

A user wrote in to let me know that Bad Behavior 2 has finally been ported to Drupal.

The work is pretty early and needs some spit and polish, but you can get the early results from the Drupal site.

September 28, 2006 Posted by | Bad Behavior, Blog Spam, Drupal, Spam | Comments Off on Bad Behavior 2 for Drupal

Bad Behavior Blackhole Update

About a year ago I started a project called the Bad Behavior Blackhole. The purpose of the blackhole was to list known sources of blog spam and to publish that data for the use of bloggers who wanted to make use of it to prevent spam.

Due to lack of time, I put the project on hold indefinitely. But I’ve been slowly working on it, off and on, over the last few months. Mostly off, again, due to lack of time. As I’ve said before, I have to spend most of my time on things that pay the bills, and historically, fighting spam hasn’t really been one of them, unfortunately.

With Bad Behavior 2.0.6 this week, I released a new feature which checks POST requests against third-party spam blacklists. This has proven quite effective in stopping a lot of the spam that wasn’t otherwise caught, but it does have a few drawbacks.

First, since I don’t maintain any of the lists, it’s difficult for me to help anyone get removed from the lists, other than providing links back to the blacklist providers. I’ve seen a few positive hits which I don’t want to be blocking, such as dynamic IP addresses which once sent a spam two or three years ago and have been blacklisted ever since. (The list involved, list.dsbl.org, will be dropped in the next release, and you can edit the code and remove it yourself if you’re having problems with it.)

I envision the Bad Behavior Blackhole as much more responsive than other blacklists, as the users likely to be affected aren’t going to really know what’s going on, or why they should be blocked because somebody sent a spam back in 2003.

Specifically, Bad Behavior Blackhole will have the following features:

  • Immediate removal for anyone upon request, the first time. Removal will be delayed for further requests from the same IP, to prevent spammers from removing themselves and sending more spam.
  • Blacklisting only for a specific period of time, and only while spam is actually flowing from a given IP address. Once the spam stops, the address will be delisted automatically after a short time. If it restarts, then the address is relisted.
  • List sources which are actually sending spam, as well as sources which are demonstrated to have compromised security, such as open proxy servers and Trojaned machines, before they can send spam.
  • Usable from any platform. This covers Movable Type, WordPress, and just about anything else you can think of. Adding support for realtime blackhole lists to any given program is at most a 15-minute hack.

It needs about a day and a half worth of work to finish up and do the initial rollout. (Didn’t you notice the link for it was dead?) But as I said before, I’ve been delaying it due to lack of time. And this is where you come in. I work on Bad Behavior and related projects primarily as I have time, and I can afford to devote more time to it when more people contribute to its development.

Over the past couple of months I’ve been quietly setting up a honeypot blog, and collecting other sources of data on blog spammers, to feed the realtime blackhole list. The data is coming in. At this point it just needs to be connected to the Bad Behavior Blackhole, tested and released. Once this is done we’ll have a much more responsive list which actually keeps spammers out, keeps blocking of legitimate users to an absolute minimum, and provides an easy removal method for the rare person who might be blocked.

If you’d like to see this project completed sooner than later, contribute to further development of the Bad Behavior Blackhole.

And again, thank you all for your continued support in the war on web spam. Bad Behavior could not continue without it.

September 22, 2006 Posted by | Bad Behavior, Blog Spam, Spam | 8 Comments

I spotted some Bad Behavior

People ask me about making money from AdSense all the time. While I usually will offer little tips and tricks that I’ve learned along the way, one thing I want to make sure that new AdSense publishers know is what NOT to do.

The number one thing that you should NOT do is STEAL OTHER PEOPLE’S CONTENT. Yes, I know the guy who sold you the video or the eBook said it was okay. Guess what, he has your $97 bucks, and you’re about five minutes away from being up shit creek without a paddle, as you lose your web hosting, your domain names, and most importantly, your AdSense account, all because you ripped someone off.

If I catch you stealing my content, your ass is grass. (This obviously doesn’t apply if I gave you permission to use it.)

This content was stolen from Michael Hampton.

Copyright © 2006 Michael Hampton. All rights reserved. This material may not be published, broadcast, rewritten or redistributed.

September 19, 2006 Posted by | AdSense, Advertising, Bad Behavior, Blog Spam, Google, Link Farm, Personal, Spam, Splog | 2 Comments

I spotted some Bad Behavior

People ask me about making money from AdSense all the time. While I usually will offer little tips and tricks that I’ve learned along the way, one thing I want to make sure that new AdSense publishers know is what NOT to do.

The number one thing that you should NOT do is STEAL OTHER PEOPLE’S CONTENT. Yes, I know the guy who sold you the video or the eBook said it was okay. Guess what, he has your $97 bucks, and you’re about five minutes away from being up shit creek without a paddle, as you lose your web hosting, your domain names, and most importantly, your AdSense account, all because you ripped someone off.

If I catch you stealing my content, your ass is grass. (This obviously doesn’t apply if I gave you permission to use it.)

This content was stolen from Michael Hampton.

Copyright © 2006 Michael Hampton. All rights reserved. This material may not be published, broadcast, rewritten or redistributed.

September 19, 2006 Posted by | AdSense, Advertising, Bad Behavior, Blog Spam, Google, Link Farm, Personal, Spam, Splog | 2 Comments

Bad Behavior 2.0.6

Make a Donation.

Bad Behavior 2.0.6 has been released.

About four weeks ago I provided a pre-release copy of Bad Behavior 2.0.6 to a select group of testers in order to evaluate a new method of blocking spam, and it’s proved quite successful at blocking a large chunk of spam. On my testbed it blocked 953 spams and missed about 50. So I expect it to cut the spam flow even further.

I said last month I wasn’t generally releasing it immediately so that I could determine whether it blocked any legitimate users. It did indeed block two people that I know of: one was resolved in moments through the fix-it-yourself link, and the other was myself, while using a Wi-Fi access point. I determined that someone had recently sent spam through the same AP, causing the blockage. It had also caught a third person, before the pre-release, whose computer was actually sending spam at the time.

So I’m releasing 2.0.6 generally. If you received a pre-release copy, this copy is unchanged, and you don’t need to do anything.

New in this release (since 2.0.5):

  • A new blocking method using realtime blackhole lists is being used to determine if a post originates from a known spam source, open proxy, etc. GET requests are not screened. Links are provided to blackhole list removal procedures through the fix it yourself link.

Download Bad Behavior now!

As always, if you find Bad Behavior valuable, please consider making a financial contribution. I develop Bad Behavior in my spare time, and every little bit means I have more spare time to devote to its development.

And don’t forget to subscribe to the RSS feed or the mailing list. (They’re the same content.)

September 18, 2006 Posted by | Bad Behavior, Blog Spam, MediaWiki, Spam, WordPress | 11 Comments

Bad Behavior 2.0.6 Test

Make a Donation.

Bad Behavior 2.0.6 is currently being tested.

Many of you have noticed the large upturn in spam flow in the past few weeks. Bad Behavior 2.0 to date has blocked much of it, but has not been able to block nearly as much of it as I would like.

I am currently testing a new spam blocking method which looks, for the moment, to be catching virtually all of the remaining uncaught spam which I am seeing.

I’m not releasing it immediately, though, so that I can evaluate whether it is generating any false positives, and if so, whether the affected users are able to clean their computers of the viruses and other malware which they contain, and whether this is sufficient to resolve the problem.

It will probably be about a week before I have enough data to be sufficiently satisfied with the false positive handling to put it out for general release, but so far I haven’t seen anything which would qualify as a false positive. It did catch one human being whose computer, it turned out, was sending out thousands upon thousands of e-mail and blog spams.

But if you’d like to get your hands on this code early, I am offering a pre-release package to anyone who has previously contributed financially at least $5.00 to Bad Behavior development (or anyone who contributes now). Just e-mail me and I’ll get your copy sent out.

Keep in mind that I haven’t fully evaluated whether the new code will generate false positives, though the preliminary results are that it should not stop anyone who isn’t actually sending spam, so keep a copy of the previous release around in case you don’t like it or have problems.

And don’t forget to subscribe to the RSS feed or the mailing list. (They’re the same content.)

August 21, 2006 Posted by | Bad Behavior, Blog Spam, Spam, WordPress | 9 Comments