In early October, somebody (hereafter called “Dickhead”) placed malware at the top of smbc-comics.com and captainexcelsior.com. It seemed likely at the time that they had come in through one of our home computers or a hacked email account since these 2 sites don’t share the same password. I hadn’t logged into captainexcelsior.com in a while, but Zach or Chris Jones had to update the name to Captain Stupendous. So, we changed the passwords, carefully distributed them through SMS and not email, and cleaned the site of old php based software and some php backdoors they had installed (their own copy of Zend and a fake forum). Similarly, we changed the passwords on smbc-comics.com, cleaned up old php software, and removed a few backdoors. We also aggressively virus/malware scanned our Windows boxes.
The crack on captainexcelsior.com happened again very soon. Again we changed the password, distributed fake logins to Zach and Chris in email so we could watch for potentially broken email accounts, and waited. Nothing ever happened again. More on this one later.
Late October, SMBC Comics gets more malware, again simply placed at the top of our index page. The index.php file was owned by apache, which was mysterious, but I found a few gaping holes that had crept into our setup, closed those, and assumed they could have easily exploited them. Then I dredged through all directories and carefully removed all old files. They had installed their own copy of Zend, likely with well hidden backdoors, so that was removed and replaced. They had put in a few backdoors in the .htaccess file. Amazingly, they had not touched crontab or added a public key for later login. Kinda odd. I changed passwords again and distributed them to Zach through SMS. I was pretty damn confident things were over.
November 1 – All hell breaks loose. I had completely underestimated what was going on. They installed more malware, except the attacker screwed up. He was installing it carefully in my Zend/Cache.php file so that even if my index.php file looked clean, when the site generated a page and cached it, the malware would sneak in. But, it had a syntax error causing smbc-comics.com to go blank rather than serve anything. Well, at this point everything was carefully cleaned so it had to be worse than feared. I logged in as root and saw that a successful root login attempt in /var/log/secure and called our hosting company to shutdown the server immediately and wipe it clean.
At that point, Dickhead must have gotten pissed. Suddenly I couldn’t check my email on my home linux server. I have an ssh shell already open and try to sudo and can’t. I try to su, and can’t. I try opening a new shell to my user account, and can’t. SHIT. Call my wife “TURN OFF THE SERVER!”. She does. Ok, now we know what’s been going on for the past 2 months. They had a keylogger getting any passwords that originated from my home server. Why they killed my passwords and didn’t just stay silent and let me feel that my home turf was safe, I’ll never know. I had already made up my mind to wipe that box, but they didn’t know that.
Immediately run home. Our new server was already up (thanks iWeb!). I used 3 different virus scanners on my Vista box at home and wrapped it with an off box firewall to watch for any weird traffic. Then I pulled smbc-comics.com back online doing the same careful cleanup process from before. Then I crashed around 12:30am (keep in mind I’m bringing all this up while bouncing a baby and having another one running around me). [Zach's note: Marty doesn't run a baby factory. He has two kids]
Next morning, go to log in to smbc-comics.com… root password not working! Shit! And /etc/passwd and /etc/shadow were modified at 2:43! SHIT! And all my ssh terminals have been pulled down! holy shit holy shit holy shit holy shit. Call Zach, “PANIC”. I get our root password reset and log in to get a glimpse of what’s going on before I send another bring-it-all-down request to iWeb. Start fishing around the logs. Some of my security mechanisms hadn’t tripped.. HOW’D THEY DO IT! Totally panicky now. I don’t see anything in the logs, perhaps they cleaned those? There is now only one place where my root password has been typed: my home windows box. Key logger? Remote desktop access in the middle of the night? Checking through logs and my gmail account, I dont see anything. I came up with an elaborate notion of how they could have gotten in without tripping my security measures on the smbc box or my home box
Then I realized they’re 3 hours ahead.
So, so, so, /etc/shadow was modified at 11:43pm (my time) last night not 2:43 am… that was me! And my ssh terminals going down: PuTTY naturally times out under certain configurations (I had forgotten that). And the final piece: root password not working. I had been bouncing a baby as I typed it in several times from a sheet of paper in front of me. When I handed the baby off and looked closely, my ‘e’ was actually a ‘z’. ALL A RED HERRING?! ALL THAT PANIC WAS IN MY HEAD!? WELL DONE DICKHEAD, YOU GOT IN MY HEAD. YOU FUCKING HACKED INTO MY HEAD.
Later that day, they put the old smbc-comics.com harddrive (the one that had been hacked) into an external USB port so I could look at the smoldering rubble. I met up with Ed Roper, a security ninja at my day job, and we looked carefully through the old drive. I can’t give details for fear they’ll be used against me, but it was clear they used a root kit and replaced many important binaries, but I dont understand why they didn’t do a better job of pruning logs.
So, how did all this start? I’m not fully sure, but I suspect they got in through a keylogger when I contracted a Flash-based piece of malware a few months ago on a Windows XP box. I eventually realized what happened, cleaned that box, and changed passwords. They would have had several hours of access — plenty of time to root my home box. Once they had my home box, it would only be a matter of time before I logged into smbc-comics.com (and captainexcelsior.com, and others).
We’re much more vigilant now, which mostly means I still don’t feel confident I’m out of the woods. I can conjure wild ways they could still be in other places in my life, so I’m covering those tracks now. Only time will tell (and hopefully some carefully laid mines).
We also found some IP addresses. We’re not looking for any Internet vigilantism (as cool as that sounds). But, if these boxes are serving malware, they should come down.
18.104.22.168 from Bosnia
22.214.171.124 from Germany
126.96.36.199 from Germany
In the remote chance that information from somebody would lead to an arrest or at least a kick in the nuts, I’m putting up a Nintendo 3ds (when it comes out) for anybody who gives us information that leads to Dickhead(s) getting arrested.