Fedora 14 server under sshd [net] attack... Help!

[Jburke]

My Fedora 14 server was recently attacked by an sshd [net] attack. I was able to run a $ps aux the first time and $kill the pid. Tonight when I noticed the attack occuring again I tried to $ps aux and when I went to $kill pid it changed pid's. It did this several times and I could not stop it with $kill or $kill -9. I installed denyhosts and disabled root ssh login. I also locked down all users but my own with $passwd -l *username*.

After I pruned inactive user accounts properly by changing their pass, locking account, deleting mail, deleting .bash folders, and ssh access I began to unlock accounts. I noticed 1 account in particular when I log into it the attack occurs again... but how if I disabled root login? So I then wrote a bash script to log the users activity via $script and e-mail information to myself. I logged onto the user's username and screwed around with bash commands to get a log going. The attack began again and was still able to log in as root... I checked the logs e-mailed to me and my IP, apache, yum, and every other log imaginable and got no info. I believe the attack is a script (possibly PERL or Php) but it is hidden from $ps somehow.

So I locked the username where the attack keeps occuring and created him a new one and the attack is no longer occuring. However I ran a $locate with keywords .mozilla and .gnome2 because I saw both of those on username in question; but they were empty. I noticed that these hidden directories were on pretty much every username but 2 inactive users, my username, 1 user that always connects proxied, and wordpress's username. I created a new username and saw that it too had these files. Has anyone else seen this and know where I can begin to find out how the attack is occuring. I have absolutely no clue and I am running out of ideas.......

[trekkie690]

assuming your box got hacked. they may have already changed your ps command to not show their process id. its actually quit easy my friend did it on a windows box with a batch script. They may also be within any DMZ your coming from for the attack, so check othr systems in your DMZ for issues. Also FC14 is old and very much no longer supported (which could be the reason behind the ability to hack) either manually rebuild/patch or upgrade to FC16 or 17 since they are supported at this time.

Assuming its a production network and you cannot just shut down the box and investigate, get on the box and do a tcpdump and watch the attack. (the fun part in analyzing the traffic!!) might give you more incite or use wireshark...same thing but gui form. then you'll see the IP/domain they're coming from ($10 says China or Eastern Europe!). Then a simple ACL will block them on the firewall, or at least while they're coming from that IP. course i now sshd.conf (i think) ensure you set it to sshv2, limit the number of connections and set timeout, and ..etc (google ssh security) to limit other further actions.

the .mozilla is where firfox, thunderbird, filezilla...etc store their information. the .gnome2 has things like your KEYS stored in it.

also look in the /var/logs file for more details. Its annoying but if you read through some of the files in there they can tell you exactly what your looking for, i just dont know which. messages, maillog, security, sshd...etc

[Jburke]

The PS being compromised is a possiblity... I will look into this. As for the server having networking security issues I assume it is also a possibility but is out of my power to control. The server is a GoDaddy Virtual server and they state Fedora 14 is the newest available on their Virtual Servers so I did not attempt to upgrade or manually compile by source. I checked my bash script of the users activities and found nothing pertaining to sshd[accepted] and sshd[net] but when I log onto the user name even though my Android device proxied through Orbot (Tor network) it still occurs. The .firefox and .gnome2 are new and were not included in the initial server setup. They were installed via apt rather than yum; I compiled apt by source and don't have a log of it for some strange reason (probably an error in compilation on my behalf). I am almost 100% the exploit is PERL related as I am using a lot of optional modules that were installed for my WebOS. Please let me know any other ideas that you think are a possibility.

[bendib]

Once this is resolved, I strongly recommend you clean your install of any modifications (hopefully) with the following: yum update -y && yum reinstall * -y

This will update your OS, and then proceed to download and reinstall everything.

[stevea]

First - what does /var/log/secure* say ? It will show if anyone login'd via ssh. But of course if they have root they can delete entries there.

The fact the sshd kept 'changing' pids is normal, when you try to login, then sshd forks a new process. You can kill them all the pkill -9 sshd but inittab/systemd may well start another. It's whack-a-mole until you stop the service.

You should IMNSHO NEVER expose password based authentication sshd to the Inet, but b/c it attracts script kiddies. If you use keys instead of passwwords, and disable password login - then they little sh*ts can't even get a problem without the correct key. Search the forum, the config has been posted before.

Don't assume you were hacked - look for real evidence.

I would not follow bendib advice since this will only 'fix' binaries of packages and there are a zillion attempts to exec binaries form other paths from w/in the pakckages. If you have a real hack - then rebuild from bare metal and only restore any personal files you trust.

[6e617a696d]

First thing I would do (and it's kinda standard for me on enterprise and non enterprise servers I manage):

1) limit login attempts
2) PermintRootLogin no (this should be default)
3) Consider changing default port to something non standard
if this is not an option what I would also do is add this to iptables:

iptables -N SSH_CHECK
iptables -A INPUT -p tcp --dport 22 -m state --state NEW -j SSH_CHECK
iptables -A SSH_CHECK -m recent --set --name SSH
iptables -A SSH_CHECK -m recent --update --seconds 60 --hitcount 4 --name SSH -j DROP

What it does is:
1) Create a new chain SSH_CHECK, and all incoming SSH connection (TCP port 22) will go into this chain to test the condition.
2) Condition is, for any source IP address there cannot be more than 3 SSH connection attempts within a 60 seconds window.
3) If condition has been met, then all packets from that source IP address will be dropped.
4) That source IP can only connect again if condition is cleared again, i.e. there has been 60 seconds of quiet time.