"illegal module type: so" in /var/log/secure on login fail

[Crysm]

I recently installed Fedora 9, and it ran fine up until I broke it earlier today.

When looking over the System > Authentication app from the desktop menu, I clicked the revert button, mistakenly thinking it would merely undo my changes since I'd opened the window.

Then I started experiencing problems logging in via SSH, so I rebooted. I can now no longer log in to any user, not even root. When I try in runlevel 3, it immediately brings me to the login prompt again. In runlevel 5, it brings up the desktop background without icons or bars and just sits there. And in SSH, the session immediately ends.

I can, however, boot directly into single-user mode from grub.


After googling around, I managed to pick out a few files that might offer insight, but I can't tell what to do about it.

This, I think, is what caused the problem, particularly because the errors I see (in the log) on login first show up shortly thereafter: (from /var/log/secure)

Sep  6 17:36:21 onca login: pam_unix(login:session): session opened for user root by LOGIN(uid=0)
Sep  6 17:36:21 onca login: ROOT LOGIN ON tty1
Sep  6 17:37:13 onca userhelper[8347]: pam_timestamp(system-config-services:session): updated timestamp file `/var/run/sudo/root/tty1'
Sep  6 17:37:13 onca userhelper[8350]: running '/usr/sbin/system-config-services ' with root privileges on behalf of 'root'
Sep  6 17:39:00 onca login: pam_unix(login:session): session closed for user root
Sep  6 17:39:37 onca login: pam_unix(login:session): session opened for user root by LOGIN(uid=0)
Sep  6 17:39:37 onca login: ROOT LOGIN ON tty1
Sep  6 17:39:54 onca userhelper[9738]: pam_timestamp(system-config-users:session): updated timestamp file `/var/run/sudo/root/tty1'
Sep  6 17:39:54 onca userhelper[9741]: running '/usr/share/system-config-users/system-config-users ' with root privileges on behalf of 'root'
Sep  6 17:42:14 onca userhelper[9778]: pam_timestamp(system-config-authentication:session): updated timestamp file `/var/run/sudo/root/tty1'
Sep  6 17:42:14 onca userhelper[9782]: running '/usr/share/authconfig/authconfig-gtk.py ' with root privileges on behalf of 'root'
Sep  6 17:43:13 onca userhelper[9816]: PAM (system-config-users) illegal module type: so
Sep  6 17:43:13 onca userhelper[9816]: PAM (system-config-users) no control flag supplied
Sep  6 17:43:13 onca userhelper[9816]: PAM (system-config-users) no module name supplied
Sep  6 17:43:13 onca userhelper[9816]: pam_timestamp(system-config-users:session): updated timestamp file `/var/run/sudo/root/tty1'
Sep  6 17:43:13 onca userhelper[9819]: running '/usr/share/system-config-users/system-config-users ' with root privileges on behalf of 'root'
Sep  6 17:43:30 onca su: PAM (su) illegal module type: so
Sep  6 17:43:30 onca su: PAM (su) no control flag supplied
Sep  6 17:43:30 onca su: PAM (su) no module name supplied
Sep  6 17:43:30 onca su: PAM (su) illegal module type: so
Sep  6 17:43:30 onca su: PAM (su) no control flag supplied
Sep  6 17:43:30 onca su: PAM (su) no module name supplied
Sep  6 17:43:30 onca su: PAM (su) illegal module type: so
Sep  6 17:43:30 onca su: PAM (su) no control flag supplied
Sep  6 17:43:30 onca su: PAM (su) no module name supplied
Sep  6 17:43:30 onca su: PAM (su) illegal module type: so
Sep  6 17:43:30 onca su: PAM (su) no control flag supplied
Sep  6 17:43:30 onca su: PAM (su) no module name supplied
Sep  6 17:43:30 onca su: pam_unix(su:session): session opened for user readonly by root(uid=0)

When I try to log in, I get these lines added to /var/log/secure:

Sep  6 19:31:41 onca login: PAM (login) illegal module type: so
Sep  6 19:31:41 onca login: PAM (login) no control flag supplied
Sep  6 19:31:41 onca login: PAM (login) no module name supplied
Sep  6 19:31:41 onca login: PAM (login) illegal module type: so
Sep  6 19:31:41 onca login: PAM (login) no control flag supplied
Sep  6 19:31:41 onca login: PAM (login) no module name supplied
Sep  6 19:31:41 onca login: PAM (login) illegal module type: so
Sep  6 19:31:41 onca login: PAM (login) no control flag supplied
Sep  6 19:31:41 onca login: PAM (login) no module name supplied
Sep  6 19:31:41 onca login: PAM (login) illegal module type: so
Sep  6 19:31:41 onca login: PAM (login) no control flag supplied
Sep  6 19:31:41 onca login: PAM (login) no module name supplied
Sep  6 19:31:43 onca login: pam_unix(login:session): session opened for user root by LOGIN(uid=0)
Sep  6 19:31:43 onca login: Permission denied

And by googling a bit, I came accross this file here, which I suspect to be affected, though I really know almost nothing about it: /etc/pam.d/login

#%PAM-1.0
auth [user_unknown=ignore success=ok ignore=ignore default=bad] pam_securetty.so
auth	include	system-auth
account	required	pam_nologin.so
account	include	system-auth
password	include	system-auth
# pam_selinux.so close should be the first session rule
session	required	pam_selinux.so close
session	required	pam_loginuid.so
session	optional	pam_console.so
# pam_selinux.so open should only be followed by sessions to be executed in the user context
session	required	pam_selinux.so open
session	required	pam_namespace.so
session	optional	pam_keyinit.so force revoke
session	include	system-auth
session	optional	pam_ck_connector.so

If anyone could offer any direction on how to reverse this, it would be much appreciated!
I really don't want to have to wipe the system, as it hosts (among other things) a multi-terabyte RAID array.

[Skunk Worx]

the login file looks ok. The actual content is the same as mine less some tab vs. spaces issues which I assume come from your pasting or the forum.

Mine uses spaces only, no tabs, and the columns are spaced at 12 and 25 (the "optional", "include", "required" column is at 12, the pam_* keywords are at column 25).

Also I think the "authconfig-tui" command should be possible from the single user prompt. I think the defaults are "Use Shadow Passwords" and "Local Authorizations are sufficient".

It would be interesting to know what files were modified around the time of the failure; for example /etc/passwd or /etc/shadow; and also the files in the directories /etc/pam.d and /etc/sysconfig.

You can run the "ls -alrt" command in the directories to see a list of files reverse ordered by modification time and the "ls -l" command on the passwd and shadow files to see when they were modified.

I see the complaints about system-config-users which includes config-util. Both are in the /etc/pam.d directory. Check system-config-users and config-util and system-auth.

Here's some sample output from my 64 bit amd box.

[root@localhost pam.d]# ls -alrt /etc/pam.d/ | tail -3
-rw-r--r--   1 root root      97 2008-07-11 02:02 wireshark
drwxr-xr-x   2 root root    4096 2008-08-11 23:02 .
-rw-r--r--   1 root root     890 2008-09-06 21:39 system-auth-ac
drwxr-xr-x 116 root root   12288 2008-09-06 21:39 ..
[root@localhost pam.d]# ls -l /etc/passwd /etc/shadow
-rw-r--r-- 1 root root 1940 2008-08-02 21:58 /etc/passwd
-r-------- 1 root root 1461 2008-07-28 15:14 /etc/shadow
[root@localhost pam.d]# ls -alrt /etc/sysconfig/ | tail -5
drwxr-xr-x   7 root root  4096 2008-08-13 20:01 .
-rw-r--r--   1 root root   304 2008-09-06 21:17 system-config-users
-rw-r--r--   1 root root    60 2008-09-06 21:37 network
-rw-r--r--   1 root root   290 2008-09-06 21:39 authconfig
drwxr-xr-x 116 root root 12288 2008-09-06 21:39 ..
[root@localhost pam.d]# cat /etc/pam.d/system-config-users
#%PAM-1.0
auth        include     config-util
account     include     config-util
session     include     config-util
[root@localhost pam.d]#  cat /etc/pam.d/config-util 
#%PAM-1.0
auth		sufficient	 pam_rootok.so
auth		sufficient	 pam_timestamp.so
auth		include		system-auth
account		required	pam_permit.so
session		required	pam_permit.so
session		optional	pam_xauth.so
session		optional	pam_timestamp.so
[root@localhost pam.d]#  cat /etc/pam.d/system-auth
#%PAM-1.0
# This file is auto-generated.
# User changes will be destroyed the next time authconfig is run.
auth        required      pam_env.so
auth        sufficient    pam_unix.so nullok try_first_pass
auth        requisite     pam_succeed_if.so uid >= 500 quiet
auth        required      pam_deny.so

account     required      pam_unix.so
account     sufficient    pam_localuser.so
account     sufficient    pam_succeed_if.so uid < 500 quiet
account     required      pam_permit.so

password    requisite     pam_cracklib.so try_first_pass retry=3
password    sufficient    pam_unix.so sha512 shadow nullok try_first_pass use_authtok
password    required      pam_deny.so

session     optional      pam_keyinit.so revoke
session     required      pam_limits.so
session     [success=1 default=ignore] pam_succeed_if.so service in crond quiet use_uid
session     required      pam_unix.so
$ ls -l /lib64/security/pam_unix.so
-rwxr-xr-x 1 root root 48872 2008-05-21 01:50 /lib64/security/pam_unix.so
[root@localhost pam.d]# ls -l /lib32/security/pam_unix.so 
ls: cannot access /lib32/security/pam_unix.so: No such file or directory
[root@localhost pam.d]# ls -l /lib/security/pam_unix.so 
ls: cannot access /lib/security/pam_unix.so: No such file or directory
[root@localhost pam.d]# grep 512 /etc/pam.d/*
/etc/pam.d/system-auth:password    sufficient    pam_unix.so sha512 shadow nullok try_first_pass use_authtok
/etc/pam.d/system-auth-ac:password    sufficient    pam_unix.so sha512 shadow nullok try_first_pass use_authtok
[root@localhost pam.d]# grep 512 /etc/sysconfig/*
/etc/sysconfig/authconfig:PASSWDALGORITHM=sha512

[Crysm]

Thanks for the response!

I didn't paste /etc/pam.d/login, I recreated it by hand because I found it later, it was small, and the process of (mount, copy, unmount)x2 whenever I want to transfer a file (via USB HDD, my only option) was a bit annoying. That should account for the space/tab discrepancy.

authconfig-tui shows "Use shadow passwords" and "Local authorization is sufficient" to be the only selected options, matching what you said.


I took the time to copy everything this time around, but I just piped output to a file and trimmed to the relevant bits.

ls -alrt /etc/pam.d/
(snip)
-rw-r--r--   1 root root      97 2008-07-16 05:41 system-config-language
-rw-r--r--   1 root root     890 2008-09-06 20:25 system-auth-ac
drwxr-xr-x   2 root root    4096 2008-09-06 20:56 .
drwxr-xr-x 106 root root   12288 2008-09-07 02:10 ..

/etc/shadow and /etc/passwd have been changed today, but after the incident. I did try resetting the root password with passwd to see if that fixed anything, but it didn't.

ls -l /etc/shadow /etc/passwd
-r-------- 1 root root 1397 2008-09-06 18:35 /etc/shadow
-rw-r--r-- 1 root root 1923 2008-09-06 18:21 /etc/passwd

ls -alrt /etc/sysconfig
(snip)
drwxr-xr-x   7 root root  4096 2008-08-22 15:02 .
-rw-r--r--   1 root root   304 2008-09-06 18:21 system-config-users
-rw-r--r--   1 root root   290 2008-09-06 20:25 authconfig
drwxr-xr-x 106 root root 12288 2008-09-07 02:10 ..

/etc/pam.d/system-config-users and /etc/pam.d/config-util match your files.
/etc/pam.d/system-auth varies somewhat:
I'm also noticing that the last line of /etc/pam.d/system-auth is simply "so", which doesn't look to be in keeping with the syntax of the rest of the file, and matches the log entry (though it is also a type of file extension...).

cat /etc/pam.d/system-auth

#%PAM-1.0
# This file is auto-generated.
# User changes will be destroyed the next time authconfig is run.
auth        required      pam_env.so
auth        sufficient    pam_unix.so nullok try_first_pass
auth        requisite     pam_succeed_if.so uid >= 500 quiet
auth        required      pam_deny.so

account     required      pam_unix.so
account     sufficient    pam_localuser.so
account     sufficient    pam_succeed_if.so uid < 500 quiet
account     required      pam_permit.so

password    requisite     pam_cracklib.so try_first_pass retry=3
password    sufficient    pam_unix.so md5 shadow nullok try_first_pass use_authtok
password    required      pam_deny.so

session     optional      pam_keyinit.so revoke
session     required      pam_limits.so
session     [success=1 default=ignore] pam_succeed_if.so service in crond quiet use_uid
session     required      pam_unix.so
so

ls -l /lib/security/pam_unix.so
-rwxr-xr-x 1 root root 47420 2008-05-21 04:52 /lib/security/pam_unix.so

"grep 512 /etc/pam.d/*" and "grep 512 /etc/sysconfig/*" return nothing. After noting the differences between your /etc/pam.d/system-auth and mine, I tried this instead:

grep md5 /etc/pam.d/*
/etc/pam.d/system-auth:password    sufficient    pam_unix.so md5 shadow nullok try_first_pass use_authtok
/etc/pam.d/system-auth-ac:password    sufficient    pam_unix.so md5 shadow nullok try_first_pass use_authtok

grep md5 /etc/sysconfig/*
/etc/sysconfig/authconfig:PASSWDALGORITHM=md5

[Crysm]

....Solved! I did "authconfig-tui", marked the MD5 box, and removed the last line (simply "so") from /etc/pam.d/system-auth.
Login works, SSH, FTP, Samba... everything I need. Thanks a bunch! You put me on the right track to fix it.

[Skunk Worx]

Very interesting. I wonder how that dangling "so" got into the /etc/pam.d/system-auth file?

I figured it had to be somewhere around there. I had a problem recently where the passwd algorithm did not match and sure enough you have it too.

Glad to have helped you along the path!
---
John