systemd - pid not readable (yet?) after start - multiple services

[modal1]

I've been trying to correct the following on FC16 3.3.x-x.fc16.i686:

/var/log/messages:

systemd[1]: PID file /var/run/akmodsd.pid not readable (yet?) after start
systemd[1]: Unit akmods.service entered failed state
systemd[1]: PID file /run/nmbd.pid not readable (yet?) after start
systemd[1]: PID file /var/run/httpd/httpd.pid not readable (yet?) after start
sendmail[1529]: WARNING: Xspamassassin: local socket name /var/run/spamass-milter/spamass-milter.sock missing
systemd[1]: PID file /run/sendmail.pid not readable (yet?) after start

Oddly, all of these (with the exception of akmods, which doesn't concern me much as X works and I don't often use a GUI on this box) work without error after a manual restart and httpd,nmdb and sendmail do not fail to run on boot despite the warning about the pid not being readable.

spamassassin/spamass-milter fail to function but the socket file does exist despite the error that sendmail is generating:

ll /var/run/spamass-milter/spamass-milter.sock
18981 srwxr-xr-x 1 sa-milt sa-milt 0 Jun 19 12:25 /var/run/spamass-milter/spamass-milter.sock

Upon receiving a message sendmail generates the following error:
/var/log/maillog:

Milter (spamassassin): local socket name /var/run/spamass-milter/spamass-milter.sock unsafe
Milter (spamassassin): to error state

But after a manual restart (systemctl restart/service restart - doesn't matter) both are functional.

/lib/systemd/system/spamass-milter.service:

[Unit]
Description = Mail filter for SpamAssassin
Wants = spamassassin.service
After = syslog.target local-fs.target network.target remote-fs.target nss-lookup.target spamassassin.service
Before = sendmail.service
Before = postfix.service
Conflicts = spamass-milter-root.service

[Service]
Type = simple
UMask = 0022
Environment = SOCKET=/run/spamass-milter/spamass-milter.sock SOCKET_OPTIONS= EXTRA_FLAGS=
EnvironmentFile = -/etc/sysconfig/spamass-milter
EnvironmentFile = -/etc/sysconfig/spamass-milter-postfix
User = sa-milt
ExecStart = /usr/sbin/spamass-milter $SOCKET_OPTIONS -p $SOCKET $EXTRA_FLAGS
[Install]
WantedBy = multi-user.target

/lib/systemd/system/spamassassin.service:

[Unit]
Description=Spamassassin daemon
After=syslog.target network.target remote-fs.target nss-lookup.target

[Service]
Type=forking
PIDFile=/var/run/spamd.pid
EnvironmentFile=-/etc/sysconfig/spamassassin
ExecStartPre=-/sbin/portrelease spamd
ExecStart=/usr/bin/spamd --pidfile /var/run/spamd.pid $SPAMDOPTIONS
StandardOutput=syslog
StandardError=syslog

[Install]
WantedBy=multi-user.target

and for comparisons sake:
/lib/systemd/system/httpd.service:

[Unit]
Description=The Apache HTTP Server (prefork MPM)
After=syslog.target network.target remote-fs.target nss-lookup.target

[Service]
Type=forking
PIDFile=/var/run/httpd/httpd.pid
EnvironmentFile=/etc/sysconfig/httpd
ExecStart=/usr/sbin/httpd $OPTIONS -k start
ExecReload=/usr/sbin/httpd $OPTIONS -t
ExecReload=/bin/kill -HUP $MAINPID
ExecStop=/usr/sbin/httpd $OPTIONS -k stop

[Install]
WantedBy=multi-user.target

/lib/systemd/system/smb.service

[Unit]
Description=Samba SMB Daemon
After=syslog.target network.target nmb.service winbind.service

[Service]
Type=forking
PIDFile=/run/smbd.pid
LimitNOFILE=16384
EnvironmentFile=-/etc/sysconfig/samba
ExecStart=/usr/sbin/smbd $SMBDOPTIONS

[Install]
WantedBy=multi-user.target

Any thoughts on what could be causing this issue?

[george_toolan]

/run is a temporary file system. Is devtmpfs loaded correctly and can the program write to the file system or is SELinux blocking the access attempts?

ls -l /run

[modal1]

As far as I can tell it is:

ps auxwwww|grep dev
root        15  0.0  0.0      0     0 ?        S    Jun19   0:00 [kdevtmpfs]

SElinux is disabled.

ll /run
total 96K
 1191 drwxr-xr-x  36 root    root    1.3K Jun 19 12:45 .
    2 dr-xr-xr-x. 23 root    root    4.0K Jun 20 01:00 ..
12898 drwxr-xr-x   2 root    root      80 Jun 19 12:23 abrt
12899 -rw-r--r--   1 root    root       4 Jun 19 12:23 abrtd.pid
15076 -rw-r--r--   1 root    root       5 Jun 19 12:23 atd.pid
13389 -rw-r--r--   1 root    root       4 Jun 19 12:23 auditd.pid
12697 drwxr-xr-x   2 avahi   avahi     80 Jun 19 12:23 avahi-daemon
14577 -rw-r--r--   1 root    root       5 Jun 19 12:23 chronyd.pid
12580 drwxr-xr-x   2 root    root      40 Jun 19 12:23 console
18717 drwxr-xr-x   2 root    root      60 Jun 19 12:23 ConsoleKit
18726 -rw-r--r--   1 root    root       5 Jun 19 12:23 console-kit-daemon.pid
13928 -rw-r--r--   1 root    root       5 Jun 19 12:23 crond.pid
13982 ----------   1 root    root       0 Jun 19 12:23 cron.reboot
12569 drwxr-xr-x   3 root    lp        80 Jun 19 12:23 cups
12704 drwxr-xr-x   2 root    root      60 Jun 19 12:23 dbus
14214 -rw-r--r--   1 root    root       5 Jun 19 12:23 dhclient-eth0.pid
 9360 prw-------   1 root    root       0 Jun 19 12:23 dmeventd-client
 9359 prw-------   1 root    root       0 Jun 19 12:23 dmeventd-server
12581 drwxr-xr-x   2 root    root      40 Jun 19 12:23 faillock
16155 drw-------   2 root    root      60 Jun 19 12:23 fcm
17285 -rw-------   1 root    root       5 Jun 19 12:23 fcoemon.pid
12078 -rw-r--r--   1 root    root       4 Jun 19 12:23 gpm.pid
12571 drwx--x---   2 root    apache    60 Jun 19 12:23 httpd
 7221 drwxr-xr-x   9 root    root     220 Jun 19 05:23 initramfs
12575 drwxr-xr-x   2 root    root      40 Jun 19 12:23 libgpod
15610 -rw-------   1 root    root      32 Jun 19 12:23 lldpad.pid
 8859 drwxr-xr-x   6 root    root     120 Jun 20 03:46 lock
12577 drwx------   2 root    root      40 Jun 19 12:23 lvm
13673 srwxr-xr-x   1 root    root       0 Jun 19 12:23 mcelog-client
13674 -rw-r--r--   1 root    root       4 Jun 19 12:23 mcelog.pid
 8900 drwx--x---   2 root    root      40 Jun 19 05:23 mdadm
13796 -rw-r--r--   1 root    root       5 Jun 19 12:23 messagebus.pid
10414 drwxr-xr-x   2 root    root      60 Jun 19 12:23 mount
12572 drwxrwxr-x   2 root    root      40 Jun 19 12:23 netreport
12578 drwxr-xr-x   2 root    root      40 Jun 19 12:23 net-snmp
17261 drwxr-xr-x   2 root    root      60 Jun 19 12:23 nmbd
17174 -rw-r--r--   1 root    root       5 Jun 19 12:23 nmbd.pid
12579 drwx--x---   2 root    openvpn   40 Jun 19 12:23 openvpn
 1350 drwxr-xr-x   2 root    root      40 Jun 19 12:23 plymouth
12583 drwxr-xr-x   2 root    root      40 Jun 19 12:23 portreserve
12584 drwxr-xr-x   2 root    root      40 Jun 19 12:23 ppp
12586 drwxr-x---   2 root    root      40 Jun 19 12:23 pptp
15874 -r--r--r--   1 root    root       0 Jun 19 12:23 rpcbind.lock
15876 srw-rw-rw-   1 root    root       0 Jun 19 12:23 rpcbind.sock
16009 -rw-r--r--   1 rpcuser rpcuser    5 Jun 19 12:23 rpc.statd.pid
12587 drwxr-xr-x   2 root    root      40 Jun 19 12:23 samba
12588 drwxr-xr-x   2 root    root      40 Jun 19 12:23 saslauthd
12589 drwxrwxr-x   2 root    screen    40 Jun 19 12:23 screen
16279 -rw-------   1 root    smmsp     33 Jun 19 12:23 sendmail.pid
12582 drwxr-xr-x   2 root    root      40 Jun 19 12:23 sepermit
12576 drwxr-xr-x   2 root    root      40 Jun 19 12:23 setrans
17296 -rw-r--r--   1 root    root       5 Jun 19 12:23 smbd.pid
16336 -rw-r--r--   1 smmsp   smmsp     49 Jun 19 12:23 sm-client.pid
16012 -rw-------   1 root    root       5 Jun 19 12:23 sm-notify.pid
12590 drwx--x--x   2 sa-milt sa-milt   60 Jun 19 12:25 spamass-milter
17906 -rw-r--r--   1 root    root       5 Jun 19 12:25 spamd.pid
17146 -rw-r--r--   1 root    root       5 Jun 19 12:23 sshd.pid
13998 -rw-r--r--   1 root    root       5 Jun 19 12:23 syslogd.pid
 9172 drwxr-xr-x   9 root    root     260 Jun 19 12:25 systemd
 8860 drwxr-xr-x   7 root    root     160 Jun 19 12:23 udev
12591 drwxr-xr-x   4 root    root      80 Jun 19 12:39 user
12592 -rw-rw-r--   1 root    utmp    1.5K Jun 20 11:13 utmp
12596 drwx------   2 vnstat  vnstat    60 Jun 20 00:53 vnstat
12597 drwxr-xr-x   2 root    root      40 Jun 19 12:23 vpnc
16696 -rw-r--r--   1 root    root       5 Jun 19 12:23 xinetd.pid

---------- Post added at 12:10 PM ---------- Previous post was at 11:58 AM ----------

I didn't initially get a response here so I posted the same question to reddit/linuxadmin http://www.reddit.com/r/linuxadmin/c...tart_multiple/

[modal1]

Am I the only one with this issue?
This is a fresh install of FC16 (XFCE spin) and occurs with all installed kernels.

[modal1]

[facejoke]

You are not alone ugly ghost

I have the same problem but not on all lsb scripts.

httpd for ex has no pid on /var/run,
so systemctl complaints....

I googled hours without any clue

[jpollard]

I have the same errors recorded, but I also have the file...

Looks like a timing error again- perhaps systemd is looking for the file before it is created. Possibly something to do with the thundering herd scheduling being done, some things will get delayed - For instance, it may be created with the PID in the buffer, but not yet flushed to disk. If systemd looks for it before the flush it will be unable to read it.

Not sure this can be fixed - but have you made a bug report?

[facejoke]

not yest.
It's pertty weird that not more people have this kind of problem which is
very crucial. For now I can't use Fedora 17 in production mode for sure...

[jpollard]

Never should use Fedora for production.

There are always too many things broken, or at best, unstable.

Use CentOS or SL.

systemd is not usable in servers, those people that have keep finding it failing one way or another. One big problem for servers is that it looses log entries, and scrambles up log entries making it hard/impossible to track a cascading failure. Combining that with lost messages and it gets to be unusable.

It does work for simple cases... but running one service per system is the windows way to do things, and is not cost effective.

[facejoke]

I'm locked since 2004 with fedora for
my research project... no choice..

[jpollard]

Then don't update it.

To me, the last dependable Fedora version was 14. Everything since has one thing or another failing (can't boot, network failures, services don't properly start/stop, shutdown hangs...)

[jpollard]

It is impossible to use upstart - Pottering and the rest of the developers buggerd up the services to the point that they won't work without systemd.

F10 had a number of problems, but none remained (yum changed in F10, things improved in 11/12) by the time F14 came out.

But if you think one week to recompile things from F10... why did you update from F10?

Fedora is a moving target, and nothing remains stable very long. There is nothing that says you HAVE to use F17, especially when it doesn't do what you need.

Fedora 18 is coming (eventually) and things will again be unstable. Right now the install process is so bad it doesn't really work except for taking absolutely everything as defaults... even then, a number of reboot problems remain (forcing grub2 into the mix) that that too will be a problem.

RHEL/CentOS 6/SL are currently around the F12+ level. And will remain at that level for quite some time.

[facejoke]

well, for several different reasons I would like to upgrade:
- use kernel 3.6 to take advantage of last RT patches-(even if 3.6 compiles well on FC10 some components like pacemaker cluster became unstable)
- some components requires recent glibc
- most components I use take 3 times less cpu on FC17 than FC10
- anticipate further compilation problems due to a too old system.

BTW I can change OS, but I have no time to test all stuff on CentOS for ex, it will personaly take too much time
to recheck all.

Is your advice would be to upgrade until FC14 and wait for upper versions ?

[jpollard]

My suggestion is to use what works for you.

If you need a newer kernel, you can get it - very little ties the system to the kernel (especially for the F14 level) and it is entirely possible to compile new kernels on it.

But most likely, you will be wasting time rebuilding things on the "latest and greatest" and not getting real work done. Even if you do use 1/3 the CPU time on the latest software, you could easily speed 4 times that just getting it to compile, run, and validated before getting back to doing real work. Doing that work once, should be enough. After that, get the job done. THEN you might spend time updating things to see that it remains valid.

Just because it runs fast, doesn't necessarily mean it is correct.

[facejoke]

indeed, maybe I'm a bit too much hungry to radically change everything, or I receive a lot of pressure
from other people who push me to upgrade....
I already tried to upgrade from 10->11->12->13->14 but it was taking so much time with
a lot of glitches so I said to myself maybe it's better to install a fresh FC17 and step by step compile everthing needed. My challenge being always to spend more time with my own work rather than to be a sysadmin... :o)

If I use some fedora spin like scientific or robotics, shall I face to same problems as FC17 ?

---------- Post added at 10:16 PM ---------- Previous post was at 07:45 PM ----------

Ok found the trick.

if anyone has a "PID file /var/run/XXX.pid not readable (yet?) after start."

it's only a Pid path confusion between the init script and systemd.
if your service config file has another path that systemd standard path (/var/run/httpd for systemd)
so it will complain. the solution is to set PidFile in httpd.conf as /var/run/httpd/httpd.pid


Yeah !!