Over the past few days the Apache server on my personal server has konked out a few times. Normally I’m in the middle of something else, annoyed, and running mostly on caffeine and willpower. So I just kick it over, restart it, and call it finished. But today it was hung again and I decided to bite the bullet and figure out what was up. After a couple of “how the hell did you find that out” kind of questions I figured I would post this for posterity.
It looks like the problem is that spambots are posting to the server and then keeping connections open forever (or at least for a long time). So how do you figure out something like that? Apache2 has a server status interface that it publishes via HTTP, but it’s turned off in most of the default configurations I’ve seen. You’ll probably have something like the following somwhere in your apache2.conf:
<Location /server-status> SetHandler server-status Order deny,allow Deny from all Allow from 127.0.0.1 </Location>
I add “ExtendedStatus On” below that section so that I get a detailed list of requests in the output as well. The page can only be loaded from localhost, which means I have to setup an SSH tunnel from my desktop in order to hit it.
Once you have that you setup you’ll be able to see what pages a request is hitting, how long it’s been going for (the “SS” column in the status table) and what IP it’s coming from. Then it’s just a simple “iptables -I INPUT -s $ip -j DROP” and that ip effectively no longer exists as far as you’re concerned. As long as you don’t mind dropping chunks of the Internet out of existence as far as your server is concerned in order to keep your web server up (turns out I’m fine with it) everything should be peachy. Sure I could also change the Apache config to not allow that behavior, but when my own poorly written junk needs to hold a connection to the server open for 5 minutes I would have problems myself. See, that’s what us in the biz call “engineering” right there.
Here’s the list of IPs I’ve banned so far cause they were hitting my comment form too agressively: