Server Hacked and the root password didn't work

[u-noneinc-s]

This is the first hit in google (excluding a couple paid for MS ads). http://en.wikipedia.org/wiki/Rootkit
The best remedy is reformat/reinstall because you can never be sure you got rid of all traces. The best defense is strong password, better yet strong passphrases, better still is the use of keys. install denyhosts and or use /etc/hosts.allow and /etc/hosts.deny.
There are only 2 user accounts on my machine, and no root login allowed. There is only 1 IP allowed to access my machine through /etc/hosts.allow, and /etc/hosts.deny I have set to sshd : all which denies all access unless otherwise listed in the hosts.allow.

[JoeyJoJoe]

actually I've just been doing a bit more reading and I've been lead to beleive that this might not actually sort of the problem... :S

I'm unsure of what to do myself now...
my /etc/ssh/sshd_config file has a commented line "# PermitRootLogin Yes"... I'm wondering if it is worth uncommenting it and changing it to no.

[jhetrick62]

I agree, I have mine set to PermitRootLogin No and uncommented. But also run Denyhosts with tcp-wrappers to add problem IP address to my host.deny file.

Jeff

[u-noneinc-s]

my /etc/ssh/sshd_config file has a commented line "# PermitRootLogin Yes".

Test it. try to login to your own box with# ssh root@localhost and see if you can actually get logged in. I think you'll find that even though the root login=yes is commented out, it is allowed by default. Anyway, better safe than sorry so uncomment it and change it to no

[JoeyJoJoe]

for some reason my sshd_config has it un-commented in and set to no (which I must have done when I first put fedora on), where as my sshd_config~ is commented out and set to yes... why is there two? what does the "~" add to it... and would you need to get rid of both in order to stop it?

[JoeyJoJoe]

On a more "on topic" note I should say to "gmg" you really should pick a secure password for your root... I'm assuming that once they used the ssh vunerability it was probs a brute force to actually get the password (I'm unware of too many ofther ways of doing it and don't think you could use just that... if i'm wrong correct me). Anywho, you should have a password which contains at least on character outside of the alpha-numeric set (like _ - + = / . , > < *) some numbers and if possible a word that isn't in the dictionary... like "jeriomonian"... just some advice...

[jhetrick62]

The file ending in ~ is the original file, while the other is the one that you modified when you installed it. Most files that you change, automatically create backups of themselves. Check your other config files and you will see the same.

If you follow Firewing's server setup page, it instructs you to add the line PermitRootLogin No.

Jeff

[gmg2006]

Thanks guys for this wonderful discussion. I can SSH now to the server and getting more data investigating the cause of the problem. Just one thing to ask, are the log files are in /var/log/ folder? is that right?

In that folder, there are secure* files and i saw the ip addresses who successfully login to the server and tried to login. Is there any other files or log that i need to check?

[Firewing1]

Yup. /var/log/messages should log all SSH logins/outs. There was a link earlier about rootkits, but basically a hacker will replace certain files on your computer so that you can't do something or so that he will always have root access in the future. It seems in this case both happened: The hacker knows your root password and you can't change it back. You can install chkrootkit and rkhunter to scan for them, but for repairing them IMHO the safest way is to simply reinstall and then start fresh w/ better security settings. Here's how to install them (I'd do this right away on your fresh system... They scan usually once a week automatically with Cron jobs.):

su -
yum install rkhunter chkrootkit

As for the IP it means nothing because (a) the hacker could have hacked another computer which he/she instructed to hack yours, or (b) the hacker simply disconnects his modem for a couple minutes and he has a new IP.

To solve it, I'd reinstall, run

yum update

RIGHT away, to get the latest updates. Then, install chkrootkit and rkhunter as shown above, install clamav, and as my howto shows, set your SSH "PermitRootLogin No", and change the default port away from 22. Next, install denyhosts:

yum install denyhosts

and now you're all set.

Firewing1

[johannlo]

Just my 2 cents - if you can control who needs SSH access, you can set it to only accept connections w/ a valid key i.e. even if they know the password unless they have a copy of a key that the SSH server recognises it won't work. There are a few ways of doing it, e.g. key + pw associated with that particular key (as opposed to user)
- Swatch is a neat way of monitoring your log files. If u want to get fancy u can create a rule like as soon as the log indicates more than x login attempts from the same IP, insert a temporary rule into Iptables blocking that IP for a set duration. Note that I have never gotten this to work but hey I'm no expert

Swatch - http://www.linuxjournal.com/article/4776

[u-noneinc-s]

Tripwire was another good one. Does anybody know if anyone has stepped up and offered to maintain it yet? I remember there was talk of it but never heard anthing more (tight around the release of FC5)

[gavinw6662]

if you need to get the data off of there, I would start with a live cd and get all data off of there. CentOS now has a pretty handy one, or just use the trusty old knoppix. Then I would follow everyone elses instructions and do a fresh install, always the best choice in this case. After getting linux running, I would install chkrootkit and rkhunter and tripwire to hopefully monitor your system. Well not to mention choose a very strong password too. Always remember, 8 or more characters comprised of letters,numbers (caps non caps), punctiation. Those are always the hardest to crack.

[brunson]

tripwire is a great tool, but requires a fair amount of manual interaction whenever you update the system, which happens a lot in fedora. I doubt, even if it had a maintainer, that those who really could benefit from it would have the attention to detail to use it. The package is pretty well useable as it is, if no one uses it now, why bother investing time and effort into it?

[u-noneinc-s]

Tripwire is a great tool, but requires a fair amount of manual interaction whenever you update the system, which happens a lot in fedora. I doubt, even if it had a maintainer, that those who really could benefit from it would have the attention to detail to use it. The package is pretty well useable as it is, if no one uses it now, why bother investing time and effort into it?

Maybe I should re-install it and try it again. I had it running on FC3, but when I upgraded to FC5 "something" seemed to change (don't remember what exactly). Then a thread popped up here at FedoraForum about it not supportng FC5 and nobody maintaining it so I uninstalled it. Is tripwire-2.4.0.1-x86-bin.tar.bz2 still current?
(Rhetorical question, I can google it). I hope I can remember how to install it. I think I saved a text file for that

By "a fair amount of interaction" I guess you mean verifying the listed changes in filesystems, users and groups etc? I rather enjoyed that part of it. It kept me on my toes. Checked root mail (logwatch, rkhunter and tripwire) daily.

[civic_si]

there are programs out there that will detect root kits. I would try this before reinstalling my server. Every machine I have open with ssh thats not behind a firewall I use deny.hosts to block unwanted access. It will save alot of headaches later.