DrunkenBlog

Diary of a Dying Inbox

Years ago when spam started to get really going, people started saying it was going to kill email off as a useful and viable medium for communications. I saw bits and pieces of this happening to those around me, but it wasn't such a big deal. I was very careful with my addresses, or at least with who got which one, and it just wasn't that harsh on my end. This has changed, in a very severe way, over the last few years.

Many people will say "If you'd just do this and this you won't have a problem", but I've done many of those things and I still have the problem, and it's just growing worse. My inbox is literally dying off, as I find my behavior changing more and more. And it's not just my inbox, I'm deluged on many fronts, as are many others.

Up until a few months ago, this was an entirely manageable problem. You see, I used to live via email. I'm on something like 50 email lists, some of which I follow closely, others of which have degenerated to the point that I check in them once a month to see if anything has changed. I get hundreds upon hundreds of messages to various accounts each day, most of which just get shuffled into their folders via rules.

However, I do a lot of work via email too. I get documents, proposals, images, schedules, etc. Most people who know me know I don't really do phones very well, and while the odds of getting a call back are good, the odds of actually catching me are low. But I do check my email religiously... or rather I did. You basically have two options for spam blocking, server-side or client-side, although some use a combination of the both.

Most client-side solutions have started to kinda... suck... over the last while. I'm very suspicious that some of this is intentional, because for awhile there they got very good. Then came those spams that, and I swear to gawd, seemed to be about nothing more than screwing up your filters. I know spammers were testing their messages against solutions to see what would and wouldn't get through, and these messages didn't seem to be about that... they just seemed to exist to screw up your filters enough to either make you give up, or become ineffective enough that the floodgates would open for the rest of the spam. Either way, it's seemed to have kinda worked.

Ever email client worth its salt had some sort of baysean-esque system for learning what was and wasn't spam. As an example, Mac OS X's Mail has a similar system which worked well for many people, but if my inbox of people asking for other solutions is any indication, it started to kinda suck with 10.3. Whether this is because of changes made to it in Panther, or a combination of the new uber-spam, I don't know. There are some better, paid solutions, but when you're paying $30-$50 just to be able to read your email, something is seriously screwed up. Personally, I think ThunderBird has the best built-in spam protection I've seen, but everyone knows ThunderBird is still kind of an afterthought for OS X.

Things on the server-side, which is the type I use, have gotten much smarter in a way as new types of spam-checking have come online. Before, you were basically limited to:

• Running the email past a blacklist-service, to see whether it's 'signature' had been reported before, or if the host had. This basically meant you had to hope the spam had become widespread enough to make it into a service, or it was coming from a known source... and you had to be really careful about some of those sources as some of the best were also a little overzealous.
• Your own training of your filters based on the type of mail, and from whom, you normally get, just like you'd generally do client-side.
• Blocking out half the known world, like South Korea, China or Russia from even contacting you at all. This has become less and less effective as broadband has grown in the USA, and it's easier for the spammers to pay people in Russia who send out the spam through say, your neighbor's hacked Windows box.

While other aspects have gotten smarter, one of the big things to come online was checking against the domains being linked in the spam, similar to how anti-comment-spam systems work on websites. Blacklist the site being mentioned in the email, and no matter what form it happens to take next time, or who it says it from, if that URL is mentioned, *poof. It really was a great idea, especially in terms of the large amounts f 'phishing' spam that was going out. Unfortunately...

• The types of spam that don't include any form of URL whatsoever have risen dramatically over the last little while. These generally have to do with penny-stocks, or other weird shite, and hopefully someone has started a website to track these miracle stocks mentioned in these spams to see how they actually perform.
• There's a DNS hit whenever a URL needs to be checked, and some of these spams, for whatever reason, are including monstrous amounts of links. This can start to put some vicious load on wherever you're doing your lookups, and slows everything way down. Additionally, after you've had it around for awhile and built up a selection of blacklisted domains, this can start to get expensive in terms of CPU cycles, and the process can already add quite a bit of load.

Another bring problem is the sheer size of spam at the moment. When your average spam message is 8k-16k, something freakish is going on, and I know I have started to avoid pulling down all my mail whenever I'm on a very slow connection and haven't checked for a few days.

Some people have this way worse than me, which means I'd have to imagine they're close to giving up on email altogether. In terms of the rate of spam, there's certainly been some surges lately on my end. I'm currently getting about 1,800 spams per day. Even assuming my spam solution had a 99% accuracy rate, that would mean ~18 spams were getting into my various inboxes every day instead of the spam folder.

However, it's not even close to 99% at the moment, and I actually do take the time to train my filters once, and sometimes twice or three times per day. It's more like 85% accuracy, which means on average 270 spams are getting through each day.

It's horrid, and causes me to lose more time than you'd think because they are just spewed all over my inbox. I have to check to make sure I'm not not missing something mixed in with the flood of spam sitting there unread, which eats up way more time than it should. Additionally, I have to go through that spam that has been flagged and moved to a separate folder, scanning at least semi-carefully so that I'm not missing an important message. It's happened way too often than I'd like, because humans really suck at this sort of thing.

As a confession, I brought some of this upon myself by trying to be clever. Ever notice how site after site wants you to use your email addy as your login? That worked OK for me, until I kept remembering which email I'd actually used for that site... so I came up with nice little mnemonics for the username of the email address, like say, 'stupidstore [ a t ] drunkenblog [ dot] com'. Give it a nice chopped-mnemonic variation for the password, and I was basically set.

It not only let me see how people were getting to that email, or worse, selling it, but it let me remember how to get into where without having to consult a spreadsheet. (as an aside, ZDnet is evil about giving your info away -- avoid) This worked perfectly fine, until spammers stumbled upon the habit of just tacking any name they could think of onto a domain and sending their spam to it. This means that if you send something to adam@theblog, I'll get it. However, when this first started, spam filters were actually kinda rocking and it wasn't such a big deal... at least on my end.

That's all pretty much changed, and nothing more than a fond memory at this point. And I'm screwed in the sense that I have probably hundreds of those things out there, and I'm really not looking forward to turning off the catch-all functionality and going to every single site and updating all of that info so that their stuff gets to me. Others are in much of the same boat -- their current email might be getting slammed with spam, but changing it to something else requires going to countless sites, like their bank, shopping sites, everywhere, and changing that info.

And don't even start in with me about those evil Windows users -- many of the really popular email viruses at the moment are spread socially, which means the person getting infected actually has to do something to start spreading the hurt. Technical issues just don't really apply.

Back when I had something like two readers, the blog was often me just posting weekly statistics on how many of what virus crossed paths with my various inboxes that ClamAV had rendered neutered. I basically stopped doing it, mostly because my solution for getting the count was so hacky that I kept having to enter in the new virus names to be checked. I stopped doing it, and I really don't have the desire to do it now.

The problem hasn't really gone away, and has actually gotten much, much worse, even if there is a lull from time to time. Without going into a bunch of stats, I can say that over the last month, my various inboxes, and sites for people I have up, have averaged around 785 actual viruses getting stripped per day. Luckily, I don't deal with a huge amount of sites, but I can honestly see why a lot of hosts don't want to deal with server-side virus processing and such. Every message has to be scanned against a database of known virus signatures, and what was once basically an effortless chore for the weakest of servers can start to lay down some real hurt.

The side-effects of the processing can just be one hell of a bitch sometimes too, like the dreaded double-extension problem. The idea is that, lets say I send you a file named 'MySexyPic.jpg.exe'. Many Windows installs are set to have extensions turned off by default, so you'd be well within your rights to think you're opening a picture... because the .exe is hidden. You've now just been 0wned, and there are craploads of these nasty types of things emailed around, and it's a good idea to cull things that try to exploit it.

Unfortunately, while I am very mindful about the extensions of files I'm sending to people, many aren't, and I end up with them sending me something like 'MySexyPic.jpg.zip', which promptly gets stripped and means I have to go searching for it from the stripped and quarantined attachments. It's not the end of the world, but I have gotten dreadfully sick of dealing with it and having to be mindful of it... and I'm not the only one.

This is probably the most serious, mostly because I use MovableType and haven't yet switched to something else... and there's a reason why many hosting companies have started to literally ban it from their servers.

I've mentioned my reasons for using it before over other things I've looked at moving to, but the big one was static pages. If you use something like WordPress, your pages are all nice and dynamic via PHP, which is really nice in a way. Basically just means you change something, and it's instantly that way on the site, at the cost of some increased load on the server.

However, with MovableType if you're very... restrained... you can have everything be nice and static. This is sucky in a way, because if you change something on the page it has to be regenerated as a static file, but great in a way if you're expecting lots of traffic... because the server is basically handing stuff out hand over fist with very little overhead. It's just a tradeoff, but one this site benefited from greatly.

I mentioned comment spam, which just about anyone with a public blog has had to deal with. Mine was no big deal for awhile, probably averaging about three a day. Then I just started to get hammered and would wake up to find 200+ spams strewn about the blog. This caused me to get off my ass and finally install MT-Blacklist, which was a damn godsend. It was still there, and you still had some monster spam hits from time to time, and yes every once in awhile it caused some problems with people submitting comments, but it made everything manageable.

What's become unmanageable is the load placed on the server, namely because some spammer out there has gotten greedy beyond all means and just spews out spam via concurrent http connections. There's no trickle here, it's just a brutal frontal assault of comment spam. Anyone who has a popular MovableType-based site has seen what lots of concurrent comments being left, or even concurrent searches, can do to a server. For most blogs and such this isn't that big of a problem, but for a larger one it is an absolute killer.

The big problem here is Perl, or rather that MovableType spawns a new instance of it whenever you do a search, or in the case of a new comment, spawns a new instance and then rebuilds the page. In the case of something like MT-Blacklist, when a new comment comes it a new Perl process is spawned... which has to do all kinds of regex and other checks against a big list of no-no sites and patterns, and then if it goes through, rebuilds the page for others to see.

When these things are a trickle, as they usually are (even if you're getting tens of thousands of hits to something, only a fractional percentage will leave comments, trackbacks, etc), things are pretty peachy. Imagine what happens if you have 50 people all conducting a search at the same time, or, horrors of horror, 50 people all leaving comments being smacked against that list of no-no's. Now imagine it's a comment spammer just spewing these things out, hitting the blog with as many connections as apache will allow, and things are insane.

Now imagine you're on a shared host with multiple other sites, possibly other blogs being hit by similar things, and you can imagine why many bloggers have been annoyed at being told by their host they're not welcome as a customer anymore. DrunkenBlog has gone pseudo-down a few times over the last week, and it was entirely related to this. Seriously, nothing will stop your heart faster than logging in and seeing a load of over 200, all related to one MovableType-based site.

MovableType posted a fix for the issue, which basically involved making everything dynamic, which makes sense as while you induce extra load by being fully dynamic, it is absolutely nothing compared to what's happening now. Of course this has basically killed my reason for continuing to use it, so now it's just a matter of inertia and waiting for WordPress to hit a stable v1.5. And no, I don't hate MovableType or anything, I just know I'm going to WP eventually, not the least of which is OS X's freaky Perl issues which makes getting a local install of MovableType going a nightmare and of two things (the other being when I tried to get Evolution to install) that became such time sinks I eventually just gave up until something fundamental changes that causes me to revisit it.

And don't even get me started on TypeKey. TypeKey just sucks for a whole variety of reasons, and I eschew it completely, even when it means I'm not leaving a comment on some site that is TypeKey-only comments enabled. I'd Grrrr at the them, but the comments I leave at various places aren't exactly anything to write home about, so I'm sure they're not missing anything.

About the only real thing I've seen being on the issue is a new tagging scheme for links in comments, that Google and others have announced, which would theoretically ruin the incentive that comment spammers have for, well, comment spamming. The idea is that if you tag links in comments with this magical tag, google and others won't index it and allow it to influence their rankings... which is probably the primary goal of most of the comment spammers.

However, as we've seen with things like the rise of referrer spam, there are plenty of other goals that can be accomplished via comment spam, such as phishing and or just getting random people here and there to go somewhere you want them to, that can be accomplished simply because the cost of comment spamming is virtually nil. When I actually cared about some things, vast amounts of it was coming out of zombied home machines, and really they're just simple httpd POSTs.

You also have the problem that most of these 'solutions' come about well after the problem has passed being a nuisance, and well after there are so many installs and such out there that whatever 'new' thing comes out will take ages to really come into its own. I'm sure you've gone to random blogs and seen them littered with comment spam, either because the person doing them gave up the fight or has just moved on... in this case it'll also be about upgrading, or taking the time to add the tag.

Playing devil's advocate, one might say that this new tagging initiative could have more of a dent than others because larger, and more popular sites, usually have a higher PageRank and them moving to the new system will have a disproportionate effect. However, playing the devil's advocate of the devil's advocate, one might note that many 'popular' sites and blogs have taken to not allowing comments at all, and if they do, are generally pretty careful about weeding spam off their blogs straight quick, and thus aren't even that big of a factor right now... yet it's still worth the spammers while to be doing it.

As a side-note, I don't blame larger sites for doing it, I've often considered it myself. The usual reason is that as you start getting larger, you start attracting... extreme... views, and if you're really lucky you start attracting trolls. The secondary reason is more of a marketing thing, in that if you have comments, people might be tempted to leave one instead of going to their site and linking to you with their comments. The third reason is also marketing-related, and ties into the first, and basically comes down to many just not wanting to get whatever they've thrown up torn apart in the comments, especially when it's warranted.

For this point though, let's stick to the first.

I have a pretty thick skin at this point, but every once in awhile I'll get an email from someone saying they would have left a comment, but didn't want to get ripped apart... which makes me consider removing comments. DrunkenBlog has a very 'diverse' readership in it's way, people from every platform and every skill level, and it's a shame if some people are literally afraid to comment, or find themselves cringing at what they see in the comments... At the same time I value the feedback, even when it's negative, and I listen to it and see where it's warranted and not.

You know, I have it on my to-do list to go back and edit the Apache section of Yin & Yang, simply because of the comments, to something that gives the same idea of services running at escalated privilege levels but in a way that both gets the point across and pisses few people off. Unfortunately it's very low on the to-do list, but it's there.

Besides, without the comments, the blog would have many more typos than it does now.

This is the newest one, or at least the severity is new. Someone, somewhere, has lately decided that referral spamming is where it's at. If you're not aware, there's an HTTP header field which allows the browser to send where you are coming from to where you are going to, in the same way that it sends along what browser you're using and platform.

This was originally a thing for testing, but has become an invaluable tool for advertising-related activities, and just figuring out how the hell people are getting to where they're getting to on your site. This one is a lot worse for others than it is for me. I don't exactly try to do a lot with advertising revenue on the blog... hell, I turned all revenue-generating aspects of the site off for the duration of the last traffic spike for various reasons.

But I still use them, for various purposes, all of which have essentially become null and void over the course of this month. It's just grown exponentially, almost daily, and I have the sneaking suspicion there aren't that many people actually behind it, it's much too uniform. You can try to grep it out to have something coherent to look at, but it's just becoming much too large of a problem to do so. And some of the blog-related services out there for whom is linking to whom just don't fill the void, which means I'm just dead in the water here.

If you're curious about why someone would actually care about hitting referrer logs... I'm honestly not sure what has actually changed recently, as this seemed to rise several years ago but petered to a nuisance. The basic idea is that lots of blogs, and other sites display, in various ways, who is getting to their site. This not only gives the referrer spammer a pagerank boost if the site happens to get googled while it's being shown, but visitors reading a blog might click a link if it's there, much as they do with Trackbacks. It's evil.

*rubs temples*

Communications I know, love, and rely on are just becoming more and more of a hassle. I don't check email nearly as much as I used to, simply because the act of checking it brings with it the unspoken agreement that I'm going to be spending anywhere from 10 seconds to a few minutes dealing with spam, and a few times a day much longer. It's becoming a burden, not an enabling technology, which means like many others I'm starting to gravitate towards alternatives... like instant messaging.

It hasn't really been a conscious choice, it's just sort of happened... I don't look forward hitting send and receive in my mail client anymore, and other people seem to have gravitated towards hitting me with with an IM instead of an email, in the expectation that I'll just get it when I come back. This often isn't idea at all, as there are serious benefits to getting it via email the way most IM currently works... it enables many more options in how I actually get it. Same deal for mailing list replacements... there are forums, etc., but tradeoffs go with that.

Unfortunately, it's not a situation I see improving any time soon. On the technical side of things, we've seen some new things introduced but no real silver bullets. SPF, Domain Keys, etc. Of course the idea to some is that using the new technologies in combination can lead to a solution greater than the sum of it's parts, ala Voltron, but because of the way things work on the internet as a whole... even if these things do pan out it'll be years until they have any serious effect. My inbox doesn't really have years. Don't even come to me with the ridiculous 'sender-pays' scheme, where you're charged postage for something that is, for all intents and purposes, data just like everything else. That way madness and stagnation lies.

The vast majority of spam comes from a relative handful of people, which you'd think would make it relatively easy to hunt these bastards down, and then shut them down. Unfortunately, the government has shown it's not really up to the job:

• There's been little legislation, which I guess I should quantify by saying there's been little effective legislation. There's been some at both the State and Federal level... but most of it does very little to actually stop the tide and instead just makes it easier for ZDnet to sell my information to Colgate or whoever else they're whoring themselves to this week, even though you have specifically taken a screenshot showing you clicked the box saying don't send me a damn thing.
• A lot of spam has money behind it -- which means there is a trail. Sometimes it's a weird and convoluted trail, but it's still a trail, simply because there's a lot of money there. If they're pointing you at a site that takes money in some way, there's a trail going back somewhere that can be followed. It rarely is, at least not to the degree that would be necessary to really start stamping on these people. My understanding is that most of this type of spam originates through Americans, or citizens of countries where this is considered evil, but goes out through people in countries where people could care less... people rarely disappear off the spammer halls of fame.
• The damn foreign ISPs often couldn't care less about what is and isn't going out over the networks, it's just too lucrative in a way. I'm not talking about a Bittorrent site in Sweden, but rather Chinese/Russian/Asian ISPs which get hammered with spam complaints, then take a month to deal with them after much pressure is applied. They do get around to kicking the spammer off their network, but of course they just somehow end up popping back up on the same network doing the same things. You might not be aware, but for a lot of these ISPs it's just a known quantity, and they have seperate prices for bandwidth being used normally and bandwidth being used for evil... and they just happen to charge a lot more for the bandwidth being used for evil. But they do sell it. We're not even going to mention Russia. The Russian government doesn't give a crap that there's some phishing site siphoning off credit card numbers, they don't even have laws against most of it.
• Much of, if not all of the above is exacerbated by the fact that more resources are put into tracking down a teenager snagging a copy of SpiderMan 2 than are at tracking down the people killing the infrastructure. More money, more lobbying, more everything is being pushed at China to enact new copyright laws than to shut down this kind of crap on their ISPs.

It might not seem like that big of a deal to you in the grand scheme of things, but this stuff is having a very real effect. More than 75% of all email sent now, globally, is spam. Some day it's more like 80%, but what's a few percentage points when it comes to spam? It doesn't really matter, it's just expected to get worse in 2005, and virus counts have grown at almost the exact same rate as spam has. Hell, half the spam now actually contain a virus...

All this crap: email spam, comment-spam, referrer spam, is actually affecting services you can get, and that are being offered for what price. It's actually affecting tuition costs at some Universities. A trickle of spam is almost negligable in the grand scheme of things, to the point of being 'free', but we're actually paying for it now. Faster hardware to handle the load, newer software, and worst of all: time. Software, hardware, all those options are relatively plentiful... but time is a finite resource.

You can only slice the 'time pie' so thin before the wedge just crumbles and you have no choice but to move on... Which means IM services better step up and start to kick ass this year, as I am starting to believe 2005 will be the year my inboxes die.

 Posted by drunkenbatman
    January 20, 2005, at 05:40 AM