Marty vs. The Hacker [Guest Post by my brother, Marty]

In early October, somebody (hereafter called “Dickhead”) placed malware at the top of smbc-comics.com and captainexcelsior.com. It seemed likely at the time that they had come in through one of our home computers or a hacked email account since these 2 sites don’t share the same password. I hadn’t logged into captainexcelsior.com in a while, but Zach or Chris Jones had to update the name to Captain Stupendous. So, we changed the passwords, carefully distributed them through SMS and not email, and cleaned the site of old php based software and some php backdoors they had installed (their own copy of Zend and a fake forum). Similarly, we changed the passwords on smbc-comics.com, cleaned up old php software, and removed a few backdoors. We also aggressively virus/malware scanned our Windows boxes.

The crack on captainexcelsior.com happened again very soon. Again we changed the password, distributed fake logins to Zach and Chris in email so we could watch for potentially broken email accounts, and waited. Nothing ever happened again. More on this one later.

Late October, SMBC Comics gets more malware, again simply placed at the top of our index page. The index.php file was owned by apache, which was mysterious, but I found a few gaping holes that had crept into our setup, closed those, and assumed they could have easily exploited them. Then I dredged through all directories and carefully removed all old files. They had installed their own copy of Zend, likely with well hidden backdoors, so that was removed and replaced. They had put in a few backdoors in the .htaccess file. Amazingly, they had not touched crontab or added a public key for later login. Kinda odd. I changed passwords again and distributed them to Zach through SMS. I was pretty damn confident things were over.

November 1 – All hell breaks loose. I had completely underestimated what was going on. They installed more malware, except the attacker screwed up. He was installing it carefully in my Zend/Cache.php file so that even if my index.php file looked clean, when the site generated a page and cached it, the malware would sneak in. But, it had a syntax error causing smbc-comics.com to go blank rather than serve anything. Well, at this point everything was carefully cleaned so it had to be worse than feared. I logged in as root and saw that a successful root login attempt in /var/log/secure and called our hosting company to shutdown the server immediately and wipe it clean.

At that point, Dickhead must have gotten pissed. Suddenly I couldn’t check my email on my home linux server. I have an ssh shell already open and try to sudo and can’t. I try to su, and can’t. I try opening a new shell to my user account, and can’t. SHIT. Call my wife “TURN OFF THE SERVER!”. She does. Ok, now we know what’s been going on for the past 2 months. They had a keylogger getting any passwords that originated from my home server. Why they killed my passwords and didn’t just stay silent and let me feel that my home turf was safe, I’ll never know. I had already made up my mind to wipe that box, but they didn’t know that.

Immediately run home. Our new server was already up (thanks iWeb!). I used 3 different virus scanners on my Vista box at home and wrapped it with an off box firewall to watch for any weird traffic. Then I pulled smbc-comics.com back online doing the same careful cleanup process from before. Then I crashed around 12:30am (keep in mind I’m bringing all this up while bouncing a baby and having another one running around me). [Zach's note: Marty doesn't run a baby factory. He has two kids]

Next morning, go to log in to smbc-comics.com… root password not working! Shit! And /etc/passwd and /etc/shadow were modified at 2:43! SHIT! And all my ssh terminals have been pulled down! holy shit holy shit holy shit holy shit. Call Zach, “PANIC”. I get our root password reset and log in to get a glimpse of what’s going on before I send another bring-it-all-down request to iWeb. Start fishing around the logs. Some of my security mechanisms hadn’t tripped.. HOW’D THEY DO IT! Totally panicky now. I don’t see anything in the logs, perhaps they cleaned those? There is now only one place where my root password has been typed: my home windows box. Key logger? Remote desktop access in the middle of the night? Checking through logs and my gmail account, I dont see anything. I came up with an elaborate notion of how they could have gotten in without tripping my security measures on the smbc box or my home box

Then I realized they’re 3 hours ahead.

So, so, so, /etc/shadow was modified at 11:43pm (my time) last night not 2:43 am… that was me! And my ssh terminals going down: PuTTY naturally times out under certain configurations (I had forgotten that). And the final piece: root password not working. I had been bouncing a baby as I typed it in several times from a sheet of paper in front of me. When I handed the baby off and looked closely, my ‘e’ was actually a ‘z’. ALL A RED HERRING?! ALL THAT PANIC WAS IN MY HEAD!? WELL DONE DICKHEAD, YOU GOT IN MY HEAD. YOU FUCKING HACKED INTO MY HEAD.

Later that day, they put the old smbc-comics.com harddrive (the one that had been hacked) into an external USB port so I could look at the smoldering rubble. I met up with Ed Roper, a security ninja at my day job, and we looked carefully through the old drive. I can’t give details for fear they’ll be used against me, but it was clear they used a root kit and replaced many important binaries, but I dont understand why they didn’t do a better job of pruning logs.

So, how did all this start? I’m not fully sure, but I suspect they got in through a keylogger when I contracted a Flash-based piece of malware a few months ago on a Windows XP box. I eventually realized what happened, cleaned that box, and changed passwords. They would have had several hours of access — plenty of time to root my home box. Once they had my home box, it would only be a matter of time before I logged into smbc-comics.com (and captainexcelsior.com, and others).

We’re much more vigilant now, which mostly means I still don’t feel confident I’m out of the woods. I can conjure wild ways they could still be in other places in my life, so I’m covering those tracks now. Only time will tell (and hopefully some carefully laid mines).

We also found some IP addresses. We’re not looking for any Internet vigilantism (as cool as that sounds). But, if these boxes are serving malware, they should come down.

77.78.248.57 from Bosnia

83.133.122.54 from Germany

89.149.242.16 from Germany

In the remote chance that information from somebody would lead to an arrest or at least a kick in the nuts, I’m putting up a Nintendo 3ds (when it comes out) for anybody who gives us information that leads to Dickhead(s) getting arrested.

-Marty

This entry was posted in geek, Guest Post, Marty and tagged , , , , , , . Bookmark the permalink.

32 Responses to Marty vs. The Hacker [Guest Post by my brother, Marty]

  1. Mark says:

    That is an awesome story.

  2. Camus Dude says:

    Fuck, dudes! Also, Marty you’re like fucking Chuck Bartowski!

  3. Mitchell says:

    This feels like such an imbecilic question, especially after all the brilliant jargon above, but I want to ask: should visitors of SMBC be worried about having been infected with the malware placed on the site?

    • ZachWeiner says:

      Heya! If you had a virus, you’d probably know by now – the trojans were sent out for just a couple hours a few weeks ago, and they were pretty nasty. But, to be safe, it’s probably smart to run a virus scan.

      • @abcdariu says:

        Yeah, it was real nasty. I was running a ubuntu/XP dual boot, just doing the daily check of webcomics and them bam! Antivirus, firewall, everything goes crazy, a java splash-screen pops and the machine turns off. After that, it couldnt get to grub anymore, so I was stuck outside the OS =P

        • ZachWeiner says:

          Ack! Sorry! I have a promise out to everyone who got trojaned: If you see me at a con, I’ll draw a picture of myself begging forgiveness, no charge.

      • Ryan says:

        How about putting the voteys in the RSS feed? Then at least some of us will be safe!

  4. Pingback: Tweets that mention The Weinerworks ยป Marty vs. The Hacker [Guest Post by my brother, Marty] -- Topsy.com

  5. Nathan says:

    I work for a datacentre and this crap happens all the time. Trust me, you guys didn’t piss anyone off. Consider it collateral damage because Russia and China don’t have burgeoning dotcon industries, resorting to making money from rooting other machines and infecting desktop PCs to create massive zombie networks to spam.

    My advice, consider using mod_security (with Apache) and find some defensive rules. That can help prevent your server from being hacked, especially if it’s due to holes in the web software (such as wordpress), etc. Also, keep using public-key authentication as much as possible. Less prone to attacks. Noone is sniffing your personal traffic (except perhaps the NSA), so using SMS to send passwords is about as safe as sending it over googletalk/AIM/etc.

    Feel free to shoot me an email if you have any questions.
    I wish you guys best of luck.

    Oh, lastly, make backups! Make them often! If you’re still reading this, then you’re not making backups.

    • lts says:

      Don’t worry, I had someone from the NSA *assure* me that even if they did monitor us, they don’t have near enough interns to actually go through it, so they don’t.

  6. Mike says:

    I’m pretty sure I understood most of that, even if I’m not confident in my ability to do it myself. Sucks that you had to go through it, but you handled it like a champ.

  7. Aggrazel says:

    Yeah what Nathan said is true. I deal with this kindof stuff all the time. Usually get called in after the fact to clean up some big mess such as you are describing.

    They may never have had your passwords. I’d be surprised if they did. Most site hacks are done because of holes in web code. Especially suspicious as you say your index.php was owned by apache, meaning the hacker simply controlled apache and made it do their dirty work.

    mod_security is a great recommendation. Also, I recommend http://www.configserver.com/cp/csf.html … it helps lock your site down for malicious looking activity.

  8. Pingback: channelATE.com: Comics and Cartoons by Ryan Hudson - I want a baby.

  9. John says:

    Thanks for the above posts, should be helpful even to those not currently experiencing hacker difficulties…

  10. Olax says:

    So… when will The Forum (capitalized for worth) be back?

  11. Marty says:

    Thanks for the suggestions. I’ll look at mod_security today.

    The forums will be back up layer today. From a security standpoint, they’re the freakiest.

  12. Ed says:

    Well you definitely did the right thing by informing eveyone, you could have just dealt with it in secret and kept it quite to save face, but you made sure we would be on alert for suspiscious things, so thanks

    • ZachWeiner says:

      Believe me, the though occurred to me.

      But, I figure it’s better for everyone to come clean. That keeps more people safe, and shows that I’m not (always) a dick.

  13. Max says:

    Marty, I’m just curious…how certain are you that those IPs aren’t spoofed? Offhand, I don’t know of any large-scale proxy servers or similar bad dudes that make their home in Bosnia but those German IPs look like they were produced by YourFreedom.

  14. D says:

    If you have the malware/rootkit still, contact me via email. I regularly analyze malware and work heavily in reverse engineering (Win32 programming, the PE/ELF file formats, x86 assembly, etc). A friend actually referred me to this page, and I thought, at the least, I could identify the danger of the malware, its origins, and other information, which, may in turn contribute to finding what’s behind the entire ordeal.

  15. Owen says:

    Wow, somebody IS a dickhead.
    Congratulations on dealing with it so well (and so thoroughly). Here’s hoping the moron dies a hilarious death.

  16. Ian says:

    While mod_security for apache is pretty good, I find vegetarianism the best defense against hackers. I’ve had a few sites ownd over the past years but ever since I changed my diet to vegetarianism six months ago I haven’t had anything happen to me. Seems obvious that this must have been the factor.

    Also the sms passwords is a good idea, but don’t rely on it every time, throw in a good mix of transmission: sms, hard line, face-to-face, etc – spice it up and be inconsistent. While this attack was pretty much just a drive-by, a targeted attack is much harder to deal with. By making your actions variable and unpredictable you make it harder to future dick-head to figure out whats going on.

    • Zak says:

      I agree, since switching to linux and veganism, I have transcended mortality, gained psychokinesis and can now absorb all forms of energy.

  17. Jeanne says:

    Your response to the possible threat at the end made me think of [url=http://img.chan4chan.com/img/2009-05-02/1241229099359.jpg]this[/url]. In all seriousness though, thanks a lot for working so hard to get everything fixed as fast as possible. It really does mean a lot to see how much you guys care about the site and keeping our computers free of trojans.

  18. Tim Baldwin says:

    Those IP Addresses are unlikely to get you anywhere. :-(
    Unfortunately, they are all associated with SPAM activity. This indicates that they are probably machines that have already been hacked, and are set up to give other hackers a “safe” starting point.

    The Bosnian address is interesting, as it seems to be a cable modem IP Address, but is it very likely that it is just some computer in Bosnia that got hacked, and other hackers are using it as a repeater for these types of attacks. :-\

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>