Webserver compromised

[kevinscof]

IM at a loss right now, so I am asking for advice. Im running Fedora Core 10 with Apache, mysql, php, perl. My email is Postfix, Dovecot. Im running a web server with aproximately 130 websites each with their own public IP. I have setup my iptables to only allow the following ports in : 21, 25, 80, 110. For my outgoing I have set ports 21, 25, 53, 80, 110. Im rejecting everything else. My system has been compromised but I have ran chkrootkit and rkhunter and it comes back with no rootkits. My log is showing requests going out on random ports above 40000. My system will come to a crawl and I look in the tmp folder and I will find a random txt file and usually a hidden folder. I am unable to figure out how they are getting in, but im guessing over port 80. any advice on what I should start checking into would be greatly appreciated.

[pete_1967]

First and foremost you should look where the network card is located and pull the plug. After that you can start forensics. It doesn't really matter which port they are using after they've taken over your system. You can't trust any binary because they may have been replaced with the crackers' own.

Longer you keep the machine connected to network, more you harm not only your clients, but also other internet users by allowing the intruders to use your server for their activities. Depending on where you live, you allowing the system stay connected may also be illegal because you are knowingly allowing your machine to be used for (possibly) illegal activities. Few years back I called FBI to sort out a university whose students were trying to spread viruses and refused to cut their connections. Day after I called FBI, the activity stopped, because it is criminal for an organisation to allow such activities run if they are informed about them.

P.S. And running out of the date, unsupported operating system as a server platform isn't the smartest ever solution either.

[kevinscof]

well that was a lot of help.. I apreciate that.

[stevea]

I do kernel work and have a background in security profiles - but not day to day admin.

Running ftp (I assume that's your port 21) is generally problematic wrt security,
I am aware of two privilege escalation exploits against the Linux kernel since F10 (IIRC), so any exploit that can execute code must be assumed to have access to root privileges.
Apache security is a specialize topic far beyond my realm, however if you haven't kept up with Apache updates and it's security measures, then a port 80 exploit is quite possible.

An application like yours would be far better served with a server distro, like CentOS rather than Fedora. Even then you need to keep up with updates.

---

Do check the files times against logs. And of course the security log - but all of that can be revised by a privileged exploit.

[William Haller]

Pete's comment might not have been helpful in the sense you wanted, but it was the correct course of action anyway. Pull the box off the net. Make an image of the disk drive for forensic analysis.

Switch to CentOS/RHEL or an up to date Fedora release with the intention of keeping all updates applied. Restore files from a time previous to the breach. Make sure everything is working.

Seriously check whether you need FTP access other than anonymous via vsftpd. If you need users to be able to upload, go to certificate based authentication and generate the security certificates. Then switch to SFTP instead for uploads. Check through all your clients scripts and packages they're using if they have the ability to install anything to their websites to make sure the versions of software they are running don't have security holes. I'm sure there are more items that others could list, but this is a basic start. Make sure any password access to your client websites is equally strong and you have no root suid scripts associated with them.

Get the box off the net, load a new formatted drive with the current software versions you need and keep it current, restore from an old backup. If clients whine about missing stuff since your last backup - carefully evaluate each item you port across to make sure it isn't infected. If you have no regular backups that are in a known uninfected state - rebuild from scratch - at least for any site that isn't plain jane non-scripted safe stuff. Then you can worry about examining the differences between the last safe copy and the current copy to see what changed.