Severe Rootkit Vulnerability suspected in Fedora 2.91 / openssh-3.9p1-3

[DeeplyWorried]

I have been struggling with trying to run a secure sshd server since
Sept. 24, and am pretty certain of my procedure and deduction. But I
don't know how wide-spread this problem is. So please check your
server to see if the same deadly problem is happening to you.

I had been running RedHat 9 on my apache/sshd server with static IP
for 6 months. On Sept. 24 I found my server was rootkitted. This is
not surprising in hindsight because I was not keeping up with updates,
and this is not what I am going to talk about. What I am talking about
is that on Sept. 24, I reformatted my / (/dev/hda1), did a clean
installation of Fedora core 3, test 2 from CD (ISO downloaded from
http://fedora.redhat.com/download/mirrors.html) with Firewall on and
only port 22 getting though (SELinux disabled - doesn't make a
difference though), within 1 minute that a ssh client connects to my
server, a sniffer found this out and rootkitted my brand new FC3T2
server.

Note that I was extra careful not using any executable not under
/dev/hda1. In fact I have just one other partition, /dev/hda3, which
is mounted under /home. I never used any commands in /home.

I found I was rootkitted by checking the file sizes of /bin/grep,
/bin/ls, /usr/bin/ssh. I took down their values when the machine is
just installed, which are 75684, 81272, 222248 bytes, respectively.
After infection, the file sizes are 77264, 85200, 227816. I wrote a
script to check this automatically.

#!/bin/csh
#
# Check the file sizes of Fedora Core 3, Test 2
#
# fc3t2
#

set array = ( /bin/grep 75684 \
/bin/ls 81272 \
/usr/bin/ssh 222248 )

set i = 1
while ( $i < $#array )
set file = $array[$i]
@ j = $i + 1
set HealthySize = $array[$j]
if ( -Z $file == $HealthySize ) then
# echo $file is healthy. #
else
echo $file is hacked\! file size should be $HealthySize\!
ls -ltr $file
endif
@ i = $i + 2
end

Those who are running Fedora Core 3, Test 2 with sshd (openssh-3.9p1-3
built on openssl-0.9.7a-39) open to the entire universe, and have
established at least one ssh client connection to it, please check
your /bin/grep, /bin/ls, /usr/bin/ssh file sizes !!!!!

The infected files are attached behind. Please be extra careful in
handling them. This virus is not actively knocking on port 22, it is
only passively listening: only after at least one ssh connection is
made so the sniffer heard it, then does it pound on your server. I
found this out by 'iptables -L -v -n'. Only when the port 22 line has
non-zero packet count does my server gets infected.

Now here is the really interesting part. I ran Rootkit Hunter
(http://www.rootkit.nl/projects/rootkit_hunter.html) and it tells me
my openssl-0.9.7a-39 from stock FC3T2 distribution may be vulnerable.
So I downloaded the newest
http://www.openssl.org/source/openssl-0.9.7d.tar.gz and
ftp://ftp.openbsd.org/pub/OpenBSD/Op...h-3.9p1.tar.gz
to compile a static sshd:

wget http://www.openssl.org/source/openssl-0.9.7d.tar.gz -O \
/tmp/openssl-0.9.7d.tar.gz
cd /tmp
rm -rf openssl-0.9.7d
tar xvfz /tmp/openssl-0.9.7d.tar.gz
cd openssl-0.9.7d
./config
make
make test
make install

wget ftp://ftp.openbsd.org/pub/OpenBSD/Op...h-3.9p1.tar.gz \
-O /tmp/openssh-3.9p1.tar.gz
cd /tmp
rm -rf openssh-3.9p1
tar xvfz /tmp/openssh-3.9p1.tar.gz
cd openssh-3.9p1
mv /etc/ssh/ /etc/ssh.orig
./configure --help
./configure \
--with-ldflags=-static \
--with-ssl-dir=/usr/local/ssl/ \
--sysconfdir=/etc/ssh/ \
--with-xauth=/usr/X11R6/bin/xauth
make
make install
cp /etc/rc.d/init.d/sshd /etc/rc.d/init.d/sshd.orig

emacs /etc/rc.d/init.d/sshd &
replace /usr by /usr/local

cp /etc/ssh/sshd_config /etc/ssh/sshd_config.orig
cat <<EOF>>/etc/ssh/sshd_config
Port 22
Protocol 2
HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_dsa_key
PermitRootLogin no
UsePAM no
EOF

/etc/init.d/sshd restart

The above would satisfy all Rootkit Hunter 1.1.8 entries
(/usr/local/bin/rkhunter -c --createlogfile) except the
" - OpenSSL 0.9.7a [ Vulnerable ]" line, which is
moot because I installed the newer openssl-0.9.7d.tar.gz in
/usr/local/ssl/ and linked against that. I did the
above compilation with Firewall set rejecting port 22 connections, then
after sshd restarted, set Firewall to accept port 22 again.

And the machine still gets infected in one minute after client
connections !!! This means it's not just a problem of FC3T2, but the
newest and best openssl-0.9.7d.tar.gz and/or openssh-3.9p1.tar.gz. So
this could get really serious for anyone running Linux server.

I also did clean installations of Fedora Core 2, Test 1; Fedora Core
2, Final; the respective file size checking scripts are behind.

#!/bin/csh
#
# Check the file sizes of Fedora Core 2, Test 1
#
# fc2t1
#
set array = ( /bin/grep 75668 \
/bin/ls 76004 \
/usr/bin/ssh 223900 )
set i = 1
while ( $i < $#array )
set file = $array[$i]
@ j = $i + 1
set HealthySize = $array[$j]
if ( -Z $file == $HealthySize ) then
# echo $file is healthy. #
else
echo $file is hacked\! file size should be $HealthySize\!
ls -ltr $file
endif
@ i = $i + 2
end



#!/bin/csh
#
# Check the file sizes of Fedora Core 2, Final
#
# fc2
#

set array = ( /bin/grep 75716 \
/bin/ls 76932 \
/usr/bin/ssh 208872 \
/bin/su 101390 )

set i = 1
while ( $i < $#array )
set file = $array[$i]
@ j = $i + 1
set HealthySize = $array[$j]
if ( -Z $file == $HealthySize ) then
# echo $file is healthy. #
else
echo $file is hacked\! file size should be $HealthySize\!
ls -ltr $file
endif
@ i = $i + 2
end


All are infected in the same way. In my opinion there is an unknown
bug in openssl / openssh source tree and the hacker is exploiting it.

I'm running out time dealing with this because I have other jobs to
do. If I were younger I would read /bin/grep assembly code and figure
out who did it. For now I will just shut down sshd. This is a heads-up
for those in the serious security business.

[Jman]

Very thorough analysis, but I'm not sure what happened in your case.

You may want to try chkrootkit to search for these kind of things. Get the example yum.conf from the FAQ, uncomment dag, and

yum install chkrootkit

.

Moved to Security.

[ewdi]

okay, this is just a suggestion for the future.

Disable Direct Root Login! that is very helpfull

create one user with wheel and only allow that user to su into root.
it will at least give you double protection, put chkrootkit into cron and makes it email it to you everyday.

[blammo]

Such a severe vulnerability hasn't been reported or investigated? I run rkhunter regularly and it shows the same vulnerablilities with openssl and ssh. The rkhunter FAQ explains that these are probably false-positives because distributions like Redhat and BSD choose to patch the programs and rkhunter thinks they are the original vulnerable versions when in fact they are the safe patched versions. With only port 22 open in your firewall on a brand spanking new installation? Getting slammed so quickly and nobody has heard about it? I don't know.

[DeeplyWorried]

Yum is not yet working for fc3t2. But I downloaded the source:

wget ftp://ftp.pangeia.com.br/pub/seg/pac/chkrootkit.tar.gz -O \
/tmp/chkrootkit.tar.gz
cd /tmp
rm -rf chkrootkit-0.44
tar xvfz /home/Beowulf/chkrootkit/0.44/chkrootkit.tar.gz
cd chkrootkit-0.44
make sense
./chkrootkit

and nothing was found.

[DeeplyWorried]

I already put

PermitRootLogin no

in /etc/ssh/sshd_config
and tested 'ssh -l root' doesn't connect.

[DeeplyWorried]

I indeed think as you do and doubt my sanity some time.

My question to people who installed FC3T2: is your
/bin/grep 75684 bytes or 77264 bytes ? If 77264 bytes
please check its initial size just after installation.

The second question is: since I 'installed everything',
are there daemon processes changing key command
file sizes? I think this is unlikely - but now I have to ask.

[Dog-One]

This all took place with i386 version not x86_64 correct?

[DeeplyWorried]

Yes it is the i386 ISO, and "install everything".

I also checked it is not from up2date because the
/bin/grep date was not changed even though the
file size changed.

[pigpen]

Is there a list with up-to-date md5 hashes of all binaries of fedora core (2 in my case)?
I ask because rkhunter produces the following output on my fedora box:

[root@pcxxxxx rkhunter]# grep NOT /var/log/rkhunter.log
[01:37:27] /bin/netstat Hash NOT valid (My MD5: 0d219306a39b315e2a3472f703c3275e, expected: 21d52be31b7242bd2a8a2d05ec4069d3)
[01:37:30] /sbin/depmod Hash NOT valid (My MD5: 306d4f31623d6d8aa3c3b5ce4073075c, expected: 1dd725c82bc1dfec1c33ac553986f6e7)
[01:37:31] /sbin/ifconfig Hash NOT valid (My MD5: 2414d669ab6ed5b49a081321d46d4a0a, expected: 5f3062e2c86cb8412b0d582e1077df22)
[01:37:31] /sbin/insmod Hash NOT valid (My MD5: 7ccba496f763eeced8f320d0ec9cd3a1, expected: b907a1ca2bc08aef86e570f52e76adb7)
[01:37:31] /sbin/modinfo Hash NOT valid (My MD5: a93bc7120a13d51d67d76ec0d93d4f63, expected: f8bc4e505c691f4e6807a1faffffb72b)

Additionally, chkrootkid complains about hidden processes:

Checking `lkm'... You have     2 process hidden for readdir command
You have     2 process hidden for ps command
Warning: Possible LKM Trojan installed

which turned out to be mysql:

[root@pcxxxxx rkhunter]# /usr/lib/chkrootkit-0.43/chkproc -v -v
PID  3947: not in readdir output
PID  3947: not in ps output
CWD  3947: /var/lib/mysql
EXE  3947: /usr/libexec/mysqld
PID  3948: not in readdir output
PID  3948: not in ps output
CWD  3948: /var/lib/mysql
EXE  3948: /usr/libexec/mysqld
You have     2 process hidden for readdir command
You have     2 process hidden for ps command

is this normal or should I be worried?

[DeeplyWorried]

I just found out the reason of the file size change. It is false alarm.
I would like to apologize and thank those who helped. Because I also
searched on google but didn't find anything, I hope this post help
people realize what is going on.

There is a cron.daily/ job called 'prelink'. It searches for dynamical libraries
on the disk, then modify the binary to make it load faster. All binaries listed
in /etc/prelink.conf are affected (without chaning the file dates either).

My accusation of openssh was ill founded. The phenomenon of file size
change after one ssh client connection was pure coincidence. Indeed,
'prelink' was unearthed after a final experiement, with the ethernet cable
pulled out. The file sizes still changed.

Actually I did look in cron.hourly/ early on but did not find anything that
can explain the behavior. I never thought it could be in cron.daily/ since
to my highly agitated mind, the attack seemed to occur more frequently.

'prelink' is a good idea, but should be more publicized. It can really
cause health problem for people like me.

[Dog-One]

It's very possible to get hacked. Been there, done that with RH9 and RH7.1.

In the case of the RH9, someone captured a partial ssh login to my University account. Since my username there and the one on my Linux box wasn't the same, but similar and the passwords were the same, they did a little poking until they got in. I traced the shell command logs and found all kinds of rootkits loaded but it looked much like they all failed to take hold. I only happened to notice it because I spotted a login with my username as failed with an Internet IP, then another and another, then one that was successful.

In the case of the RH7.1 breakin, that was really strange. I just happened to be watching traffic with iptraf when I noticed a connection to port 68533 from an IP that I couldn't traceroute back to. For one, I didn't think it was even possible to connect to a port above 65535, let aone one that had no listening daemon associated with it. I kept watching the packets going in and out of this port and kept noticing the hard drive light blipping every so often. Finally, I just pulled the plug on the Internet, dug around in my system and found zippo. After having a few bad dreams (thanks to the movie, "Enemy of the State"), I just reloaded the whole darn thing and started fresh with everything inbound blocked by iptables (or was it ipchains then?). The whole thing just seemed like a NSA job or something--somebody that really knows way, way more than me was just having a little fun with me.

So these days when I read things that say, "it's not a matter or if, it's when", rings true. When you connect to the Big Pipe, you will get hammered, eventually. Keep good backups and be ready to reload at a moments notice.

If you run a dedicated Linux firewall, put some --log entries in your iptables script and notice how many unsolicited connection attempts come in. It still blows me away. I honestly wish that ISPs would provide an upstream user configurable firewall that would allow you to block all that crap BEFORE it chews up half your bandwidth.

[crackers]

Originally Posted by Dog-One

when I noticed a connection to port 68533 from an IP that I couldn't traceroute back to. For one, I didn't think it was even possible to connect to a port above 65535, let aone one that had no listening daemon associated with it.

That is very strange - I've done all sorts of math on the 68533 number and can't seem to get it to map to any known daemon process. I'll go out on a limb and guess it was a buffer-overflow hack...

[Dog-One]

What did it for me was trying to find where the intruder came from. As I recall, when I did a traceroute back to the IP, is was something like 17 hops away and it did have a valid DNS entry--odd but valid. The very next day I did some more poking and found that the last 4 hops of the previous nights traceroute didn't even exist on the Internet anymore, let alone any of the DNS entries. So how does one turn off, not just an IP, but several segments of domain names and IPs in a single day?

The only thing I could figure is that the intruder had some connections with the powers that be and when he noticed me trying to traceroute back to him, he vanished without a trace. The whole incident left me twisting in the wind--kind of like waking up from a dream and finding yourself naked, standing on stage in an auditorium with a packed house. I went several weeks with a default install running, too afraid of loading anything important on my machine.

Shorty after the whole mess, I was told by my wife that she had recently submitted a packet to get her security clearance at her new job--that's when the light came on. So when you think security, think security levels. Most likely, no one is at a security level that is totally safe. Many people may be at a security level that is safe from the casual hacker; a few safe from the experienced hacker, but no one's system is so good that it can't be compromised by the folks that really know. You just have to decide what's good enough and only put things on that machine that are within that level of security.

[blammo]

Hey people, I wouldn't worry too much about this stuff! Ports are like *******s, every once in a while somebody's going to come along and smell 'em! I just spent an hour watching my logs of someone from Asia trying to logon to my SSHD daemon, as root, and I have that disabled in the config. Am I safe? I queried the IP for info, and it returned some non-standard replies and I give a hoot! Should I send a complaint? Yea, right! Just watch your ass!

Be afraid folks, be very afraid!!!