httpd won't start at boot, but runs fine later

[rholme]

I run tomcat and have used it since Fedora 6. When I used fedora 18, I had a problem that tomcat needed to start before httpd (solved by starting both from /etc/rc.d/rc.local), When I upgraded to fedora 20, I took the suggested step of making EVERYTHING (httpd and tomcat) start correctly using systemctl. All was fine for a while, but now I am back to an OLD problem. HTTPD will NOT start at boot, but can be run later manually using systemctl. It is obvous from the log that httpd is unhappy about some network socket service that needs to be up before it can come up. I am wondering if someone knows which service it is screaming for AND (if possible) how to add that service to the "After" line in the /etc/systemd/system/multi-user.target.wants/httpd.service file. I would add tomcat.service here to force httpd to after apache-tomcat comes up, but would be just guessing as I suspect that is NOT the real culprit.

Here are the relevant excerpts from the /var/log/messages file (hope this is enough for a clue). Date stamp removed to shorten:

------
systemd: Started Network Manager.
systemd: Starting Network.
systemd: Reached target Network.
systemd: Started Login and scanning of iSCSI devices.
systemd: Starting Xinetd A Powerful Replacement For Inetd...
systemd: Starting Virtualization daemon...
systemd: Starting NFS file locking service....
systemd: Starting The Apache HTTP Server...
systemd: Starting OpenSSH server daemon...
systemd: Starting The Jakarta Apache/Tomcat Server...
systemd: Starting /etc/rc.d/rc.local Compatibility...
......
httpd: (99)Cannot assign requested address: AH00072: make_sock: could not bind to address 192.168.101.101:80
httpd: no listening sockets available, shutting down
httpd: AH00015: Unable to open logs
systemd: httpd.service: main process exited, code=exited, status=1/FAILURE
systemd: Failed to start The Apache HTTP Server.
systemd: Unit httpd.service entered failed state.
-------

For the record, I searched the httpd error files and found nothing particularly interesting.

[jpollard]

The network isn't ready yet.

Have you enabled "NetworkManager-wait-online" ?
Are you using DHCP to configure a network?

[rholme]

B) I do not use DHCP, instead using hard network addresses in /etc/hosts

a) where do I enable "NetworkManager-wait-online"
(if this is part of the httpd.service script - please show example
if not, please tell me where to look and how to try this)

[jpollard]

Same way you enabled the httpd service.

systemctl enable NetworkManager-wait-online

What it does is cause systemd to wait for a notification from NetworkManager that NetworkManager has completed the network setup. Doesn't mean NetworkManager HAS, because if something causes a delay (such as DHCP negotiation) things can still fail.

But this is most unlikely with static IP numbers,

[rholme]

Thanks. Tried that and rebooted- so far OK. A couple more times to be sure.

[jpollard]

Such timing errors will always show up eventually. Not necessarily in the same place, not necessarily with the same services.

Anything that alters the timing between service startups can expose another one. It is the penalty that systemd will always have by ignoring the purpose of a service becoming a daemon.

[rholme]

Thanks for your help and explanation. Makes sense. Daemons were just fine with me, but then again I was always a Unix bigot. Give me a pipe and a daemon fork anyday.

So this fix - please clarify what it does. I am guessing that it makes the network manager service finish before any other sevices depending on it.

[jpollard]

It is a workaround. The systemd modifications had/has to change every service it starts.

With NetworkManager (which by the way, never "finishes"), the message is sent when NetworkManager has setup what it considers the "network". Is it actually ready when an IP number is set? no- there is still a bit of work done asynchronously to setting the IP, but the normal time is VERY short and doesn't necessarily matter. Once the message is sent, systemd can continue with the processing of the network of startup services... and NetworkManager can enter a monitoring state to respond to directives (from users/admins), and respond to network events (such as wireless).

The unfortunate thing with DHCP is that it can take a number of seconds to setup, yet NetworkManager cannot know if it really worked (the same problem systemd has) and it seems to send the message before dhclient completes, which allows some services to try starting before the network is ready for use.

I still haven't figured out if NetworkManager can handle multiple networks.

Is the network up if only one interface is up? or does it require all of them? or any network? This is a problem for servers as there are almost certainly going to be multiple networks. Where I worked we used 4.

One for administrative services that HAS to be up or the system is considered dead.

Another for NAS services...

A third for fibre channel disks

A fourth for general access.

We could operate with three of them up, and the NAS network down (as it was used only for backups/nearline archives). Having just the admin network up allows us to fix things remotely (well, down hall usually, but sometimes our admins had to login remotely to the admin workstations, then connect to the server). But the way NetworkManager APPEARS to work, that wouldn't be possible - either all networks come up, or none....

The problem starts with systemd assuming a service is ready to take requests as soon as it is successfully forked... The time it takes to perform the exec, for the new process to read its configuration file, and initialize itself can't be counted (this is the point the normal services fork/daemonize allowing the parent process carrying out the initialization to exit (which signals the script invoking the service).

To workaround the problem, systemd modified NetworkManager to send a message back to systemd. Hence the "NetworkManager-wait-online". It is not a real service, but a "wait for message".

Unfortunately, it fails (well, unless MORE modifications are made) when DHCP is used - ANOTHER modification would have to be made to dhclient to send a message back to NetworkManager... which would then send the message to systemd.

Of course, that doesn't work if dhclient is used by some other function...

In the case of other services (such as MySQL/MariaDB, that may have other processes such as apache waiting for them) they too have to be custom modified for systemd. In the case of something like databases it gets really tricky because now the chain of dependencies can get really twisted. Apache doesnt' always depend on a database, but may depend on some other service that depends on the database... So to get apache to wait for that other unknown service, requires that unknown service to also be modified for systemd....

Thus the complications get compounded.

For systemd to get everything right requires an extremely detailed knowledge of how every service works, and what that service needs to work... And that is just something very difficult to determine.

And the network analysis of such a network gets harder and harder - it never simplifies.

The use of round robin scheduling to start services works... but only if there are absolutely no interdependencies. Unfortunately, there always are dependencies - and not all of them are obvious.

[rholme]

Many thanks for your clear reply and for the fix which seems to work fine here (I don't use DHCP).

Personally I was happy with the old fashioned System 5 startups - SnnName ... OR even just BSD and rc.local --- It seems the systemd stuff is more elegant but perhaps (because it is elegant?) more prone to timing problems. In rc.local things were single threaded UNLESS you threw in an ampersand (this had to be done carefully) - I remember many times things had to follow some other service to work right (my stuff sure did and httpd + tomcat have always been delicate).

Sometimes things are FIPR (fixed in prior release). :=]

[jpollard]

elegant only in the eye of the authors.

It is fast... but at the penalty of being unreliable, and very difficult to debug problems.

the last prior release that was reliable was Fedora 14.