Lunacy Unleashed

Notes from the field in the War on Spam

Bad Behavior is Moving

It’s time for you to update your feed links and bookmarks. Bad Behavior is moving to a new home!

Continue reading

Advertisements

January 25, 2007 Posted by | Bad Behavior | Comments Off on Bad Behavior is Moving

Bad Behavior 2.0.9

Make a Donation.

Bad Behavior 2.0.9 has been released. It is a strongly recommended upgrade for all users.

This release is likely the final release in the 2.0 series as I make a major change in the development process; see below for details on this change.

This release addresses a further set of “false positive” reports received from various users which affect some uncommon circumstances.

New in this release (since 2.0.8):

  • A workaround has been placed for a problem with the Clearswift Web Policy Engine. Users behind this proxy server are no longer blocked.
  • A workaround has been placed for a bug in the LiveJournal OpenID process which Six Apart refuses to fix. Logins using OpenID will no longer fail.
  • A workaround has been placed for bugs in some versions of Internet Explorer and Safari web browsers which caused them to be blocked after leaving a comment on WordPress. These requests are no longer blocked.
  • A spam prevention feature was causing users to be blocked from their own blogs when they also subscribed to their own feed, or when they accessed the site with multiple web browsers at the same time; it has been disabled for rework.

Download Bad Behavior now!

The 2.0 series of Bad Behavior will be maintained as a legacy branch, with only bug fixes, false positive fixes and security fixes applied to this branch, if any such fixes are needed. No new checks for spammers will be added.

Shortly I will introduce a “development” 2.1 series on a much shorter development cycle, with days or perhaps even hours between releases. In this branch I’ll be experimenting with new spam prevention features, rolling them out quickly and rolling back quickly in case of actual trouble. I’ll also be rolling out a new packaging method which I’ve discussed previously, that will make Bad Behavior even more platform-independent than it currently is, and allow for the “core” to be updated separately from the “glue” which connects it to your host platform.

Once features prove themselves through development and testing to be stable, they’ll be rolled forward into a “stable” 2.2 series, intended for those users who are averse to the risks of blocking legitimate users or having the occasional crash. While I work very hard to ensure that every release, however labeled, does not crash, and does not generate false positives, things occasionally happen which are outside my control.

This parallel development scheme will help balance the needs of the two primary groups of Bad Behavior users.

The first group needs enterprise-grade code which ideally never blocks a single legitimate request and can quickly be rolled into production environments with a high degree of confidence. The tradeoff is the same as it has always been: to prevent any chance of false positives, Bad Behavior’s stable branch will permit some spam, anywhere from 0.1% to 10%, to pass through, and will require a backup solution such as Akismet. Even so, it will drastically reduce the amount of time and money spent managing spam, especially for deployments of dozens or hundreds or thousands of sites.

To serve this class of users more effectively, I’m also studying the feasibility of offering support contracts for enterprise users of Bad Behavior. Services offered under such contracts might include installation assistance, on-call support, hotfix development and deployment, and per-incident support. If your organization may need such a service, stay tuned for more details in the near future.

The second group, I believe, is the majority of Web sites: those for whom a rare blocked user is merely an annoyance rather than a critical problem, and who have much lower tolerance for spam because they aren’t being paid to manage their own blogs, wikis and forums. As much as possible, Bad Behavior’s development branch will limit spam for this class of users to 0.5% missed. The tradeoff is that you will be asked to do what you already do: to report any problems you encounter, whether they be missed spam or blocked users or plain old crashes.

And for users who would like to have their cake and eat it too, the development and stable versions will be installable side-by-side on the same site, and you will be able to switch back and forth between them at the click of a button.

Finally, prior to the first stable 2.2 release, I will be reworking all of Bad Behavior’s documentation and moving Bad Behavior from its current home to a new site dedicated solely to Bad Behavior. So you all will have to update your feed URLs to the new location soon. (Mailing list readers won’t have to do anything.)

In the meantime, Bad Behavior remains a user-supported project, with all code released under the GNU General Public License. If you find Bad Behavior valuable, please consider making a financial contribution. I develop Bad Behavior in my limited spare time, and every contribution means I can devote more time to its development.

January 8, 2007 Posted by | Bad Behavior, Blog Spam, LifeType, MediaWiki, Spam, WordPress | Comments Off on Bad Behavior 2.0.9

More Bad Behavior verbose logs needed

Last week I asked for people to send in Bad Behavior verbose logs if they had readers from certain countries.

First, I want to say thank you to everyone who sent in logs.

Second, it looks like I need logs from a different set of countries than those originally posted.

If your blog is hosted in any of the following countries, or has readers from these countries, then I need your verbose logs: Belarus, China, Hungary, Iran, Malaysia, Qatar, Romania, Russia, Saudi Arabia, Singapore, Sri Lanka, Switzerland, Ukraine.

All data submitted will be held in strict confidence, in encrypted storage, used solely to develop the software, not shared with anyone else, and destroyed within 90 days.

I still also need your verbose logs, regardless of country, if you have previously had problems with people unable to access your blog while at work, but who can reach your site at home when you used Bad Behavior’s strict mode.

To enable verbose logging: Go to Options > Bad Behavior, and tick Verbose logging. Let it collect data for two full business days (Monday-Friday). Then use your web host’s phpMyAdmin to export the log: Select the wp_bad_behavior table, click Export, click SQL format, and then Download to your computer. E-mail the .SQL file to badbots at ioerror dot us. You can then disable verbose logging if you wish.

Thank you for your help! This data will help stop more spammers while ensuring that legitimate readers are able to read your blog and leave comments without trouble.

December 19, 2006 Posted by | Bad Behavior, Blog Spam, LifeType, WordPress | Comments Off on More Bad Behavior verbose logs needed

Cyveillance Bad Behavior

“Online risks are becoming more complex and pervasive by the day,” says the home page of the Cyveillance web site.

Indeed, one of the risks is that your Web site might be targeted by Cyveillance.

The company says that it crawls the Internet looking for phishing scams, identity theft, illegal credit card numbers, trademark and copyright infringement, and more. It’s also been known to work on behalf of the government to spy on whistleblowers who expose waste, fraud and abuse. Some have even alleged that Cyveillance bots attempt to illegally hack into Web servers.

Cyveillance uses robots which crawl your Web site pretending to be a legitimate Web browser and completely ignoring your robots.txt file. Then it tries to figure out whether you’ve downloaded any illegal music, or said something bad about some company. It then sends you threatening letters ordering you to take your site down, even when you haven’t done anything wrong.

Bad Behavior doesn’t just block spammers. It’s meant to target any bot which overloads a Web site, attempts to hack in, delivers spam, acts in an unethical manner, etc.

By relying on analyzing the HTTP requests themselves, rather than simply the IP address, Bad Behavior has blocked Cyveillance almost from the very first release last year, regardless of what IP address range they move to in order to hide their connections. An audit of six months worth of logs shows that Bad Behavior was able to successfully identify and block Cyveillance bots even when they used previously unknown IP address ranges.

December 15, 2006 Posted by | Bad Behavior, Cyveillance | Comments Off on Cyveillance Bad Behavior

Bad Behavior 2.0.8

Make a Donation.

Bad Behavior 2.0.8 has been released.

This version contains updates for various “false positive” reports and is recommended for all users.

Updated in this release (since 2.0.7):

  • Verizon Wireless EV-DO users are no longer blocked.
  • Blocked requests will be subject to a two-second delay before a response is sent. (See below.)
  • Some blackhole lists previously used in Bad Behavior have been scaled back or removed.
  • The address for the Bad Behavior Blackhole has been added. (See below.)
  • Some new spambots have been identified and blocked.

In recent days spam attacks have been on the rise, with one especially obnoxious bot delivering requests so fast that some sites have been taken offline by them. While the requests aren’t especially numerous or resource-intensive, the most common software used by Web hosting providers is very inefficient at serving dynamic pages such as PHP-based Web sites. So even a moderate number of requests can take a whole server down, or lead the hosting provider to take the site down before the whole server goes down.

Bad Behavior now counters this by introducing a short two second delay to blocked requests, before the HTTP response is sent. Since most spambots wait for the response before going on to the next request, this should sufficiently slow down most of the overly aggressive spambots and give Web site operators some breathing room. While I would have liked to put in a delay of a minute or more, there remains the slight chance that an actual human being would be blocked, and they should be able to get a response back in a reasonable time.

With respect to realtime blackhole lists, all of the existing lists target e-mail spam, and since spambots who send link spam are almost always also sending e-mail spam through the same servers, these are a fairly effective means of blocking link spam. However, since they target e-mail spam, they also block legitimate users. The primary issue here is that while an IP address may be added to a blackhole list quickly, it is not removed quickly — or at all — once the spam stops. Thus, people with dynamic IP addresses are unfairly blocked because some other customer was sending spam.

Bad Behavior Blackhole, which should go online within the next few weeks, is designed specifically for link spam. It adds IP addresses to its database quickly when actual spam is received, and in addition, drops the IP addresses once the spam stops. This helps prevent dynamic IP customers from being blocked because another user’s computer was sending spam. Once Bad Behavior Blackhole is online, all other realtime blackhole lists will be dropped from Bad Behavior.

Download Bad Behavior now!

As always, if you find Bad Behavior valuable, please consider making a financial contribution. I develop Bad Behavior in my spare time, and every little bit means I have more spare time to devote to its development.

And don’t forget to subscribe to the RSS feed or the mailing list. (They’re the same content.)

December 15, 2006 Posted by | Bad Behavior, Blog Spam, Drupal, ExpressionEngine, LifeType, MediaWiki, Spam, WordPress | Comments Off on Bad Behavior 2.0.8

Bad Behavior is WordPress 2.1 Ready

The forthcoming release of WordPress 2.1 includes internal changes which may cause some plugins to stop functioning. Such plugins will need to be upgraded along with your WordPress 2.1 upgrade.

Bad Behavior 2 has been checked against these changes and found to be ready for WordPress 2.1 without any changes required. Users upgrading to WordPress 2.1 do not need to upgrade Bad Behavior for compatibility.

Versions prior to 2.0 are unsupported, have not been checked for compatibility and will not be checked. Any remaining version 1 users are strongly urged to upgrade to a current release as soon as possible.

December 12, 2006 Posted by | Bad Behavior, Blog Spam, WordPress, WordPress 2.1 | Comments Off on Bad Behavior is WordPress 2.1 Ready

Bad Behavior verbose logs needed

I’m now working on improved spam detection for certain circumstances where enabling Bad Behavior’s strict mode would block the spam, but would also block legitimate readers of your blogs.

For this work I need verbose Bad Behavior logs from some of you, in order to analyze the legitimate readers and differentiate them from spammers.

All data submitted will be held in strict confidence, in encrypted storage, used solely to develop the software, not shared with anyone else, and destroyed within 90 days.

If either you or most of your blog’s readers are from any of the following countries, then I need your verbose logs: China (not Taiwan), Qatar, Saudi Arabia, Singapore, Switzerland, United Kingdom.

I also need your verbose logs if you have previously had problems with people unable to access your blog while at work, but who can reach your site at home, regardless of country, when you used Bad Behavior’s strict mode.

To enable verbose logging: Go to Options > Bad Behavior, and tick Verbose logging. Let it collect data for two full business days (Monday-Friday). Then use your web host’s phpMyAdmin to export the log: Select the wp_bad_behavior table, click Export, click SQL format, and then Download to your computer. E-mail the .SQL file to badbots at ioerror dot us. You can then disable verbose logging if you wish.

Thank you for your help! This data will help stop more spammers while ensuring that legitimate readers are able to read your blog and leave comments without trouble.

P.S. Please vote for Homeland Stupidity, my other blog, in the 2006 Weblog Awards! You can vote again, even if you voted yesterday.

December 10, 2006 Posted by | Bad Behavior, Blog Spam, WordPress | Comments Off on Bad Behavior verbose logs needed

Vote for me in the 2006 Weblog Awards

One of my other blogs, Homeland Stupidity, has been selected as a finalist in the 2006 Weblog Awards Best Centrist Blog category.

This is wonderful news because it means I have the potential to pull in a large number of new readers to my flagship blog, increasing my revenue and freeing up more time for me to work on Bad Behavior. (A new release will be out in the next few days.)

Please help me out by voting for Homeland Stupidity today, and once per day every day through December 15th, when the voting closes.

December 8, 2006 Posted by | Bad Behavior, Personal | Comments Off on Vote for me in the 2006 Weblog Awards

Spam Surge

In the last two days I’ve seen a tenfold increase in the amount of spam being delivered, both that is being blocked, and isn’t being blocked, by Bad Behavior.

While the spam started around the same time as I released Bad Behavior 2.0.7 yesterday, there doesn’t appear to be any correlation between the two. I inspected a few of the spams and they seem like the same old stuff, just cranked into very high gear.

I’ve personally seen over 3,000 spam attempts in the last day, with over 200 missed. This is spam that Bad Behavior is not yet capable of catching without blocking legitimate users as well.

This is why I have been working on the Bad Behavior Blackhole, in order to identify and block spam by its sources, wherever they are.

The Bad Behavior Blackhole is a feature that, once fully up and running, can identify known sources of blog spam and wiki vandalism and pre-emptively block them without affecting legitimate users.

Unfortunately, time constraints have not permitted me to put in much work on Bad Behavior Blackhole, as I’ve had to work on things which bring in revenue. As I’ve said before, while tens of thousands of people use Bad Behavior, only a few dozen have ever actually contributed back.

If you find Bad Behavior valuable, and you want to see this project up and running sooner rather than later, please contribute to its further development.

Thank you in advance for your support.

Update: Slashdot has coverage of the massive spam increase, which is hitting e-mail spam as well.

November 7, 2006 Posted by | Bad Behavior, Blog Spam, MediaWiki, Spam, WordPress | Comments Off on Spam Surge

Bad Behavior 2.0.7

Make a Donation.

Bad Behavior 2.0.7 has been released.

I’ve got a nice roundup of bug fixes this time around. Most people should upgrade right away to take advantage of the fixes and additional spam protections.

New in this release (since 2.0.6):

  • A bug (apparently in MediaWiki) which caused blank lines to appear on rendered pages in MediaWiki 1.7 has been worked around.
  • In version 2.0.6 four blackhole lists were added and incoming POST requests screened against them. Two of these lists generated significant hits in which primarily non-U.S. users who hadn’t actually sent any spam were being blocked. They have been removed. (Once Bad Behavior Blackhole is up and running, the other two may be removed as well, even though they’re performing fine.)
  • Two tests which catch some spambots and content thieves which were present in Bad Behavior 1 were inadvertently dropped from Bad Behavior 2. One of the hazards of rewriting something from the ground up. These tests have been restored.
  • A check which blocks users behind a Microsoft ISA Server 2004 proxy server, and one other type of proxy server I forget the name of, has been moved to strict mode only. Please disable strict mode if you are expecting traffic from such a source, and please contact Microsoft for a hotfix, if they’ve bothered to fix the bug in their software.
  • An additional IP address range for Google has been whitelisted. If you ever receive spam from an IP address owned by Google, please notify me immediately.
  • Several additional spambots have been identified and blocked.

Download Bad Behavior now!

As always, if you find Bad Behavior valuable, please consider making a financial contribution. I develop Bad Behavior in my spare time, and every little bit means I have more spare time to devote to its development.

And don’t forget to subscribe to the RSS feed or the mailing list. (They’re the same content.)

Also please note: Due to excessive levels of spam here on wordpress.com, which doesn’t use Bad Behavior anymore, I’ve had to close comments and pings entirely. You can reach me at badbots@nospam.ioerror.us without the nospam.

November 6, 2006 Posted by | Bad Behavior, Blog Spam, Blogging, Spam, WordPress | Comments Off on Bad Behavior 2.0.7