Network Manager no longer runs after upgrading from Fedora 35 to Fedora 36

[FranceBB]

Hi there,
I've upgraded from Fedora 35 to Fedora 36 with the usual procedure, namely:

sudo dnf upgrade --refresh
sudo dnf install dnf-plugin-system-upgrade
sudo dnf system-upgrade download --refresh --releasever=36 --allowerasing
sudo dnf update distribution-gpg-keys --releasever=36
sudo dnf system-upgrade reboot

however, when I ran it, it said that there wasn't enough space left, so I used the suggestions I found in this forum, namely:

sudo dnf clean all
sudo journalctl --vacuum-size 100M

after that, I've been able to run the dnf system-upgrade reboot and it installed Fedora 36.
I'm currently running Fedora 36 kernel 5.17.12-300.fc36.x86_64, but the network manager in GNOME is gone.
I tried to see whether it was running or not with:

sudo systemctl status NetworkManager
× NetworkManager.service - Network Manager
     Loaded: loaded (/usr/lib/systemd/system/NetworkManager.service; enabled; v>
     Active: failed (Result: core-dump) since Wed 2022-06-01 16:10:38 BST; 30mi>
       Docs: man:NetworkManager(8)
    Process: 3022 ExecStart=/usr/sbin/NetworkManager --no-daemon (code=dumped, >
   Main PID: 3022 (code=dumped, signal=SEGV)
        CPU: 161ms

Jun 01 16:10:38 router-localhost systemd[1]: NetworkManager.service: Main proce>
Jun 01 16:10:38 router-localhost systemd[1]: NetworkManager.service: Failed wit>
Jun 01 16:10:38 router-localhost systemd[1]: NetworkManager.service: Scheduled >
Jun 01 16:10:38 router-localhost systemd[1]: Stopped NetworkManager.service - N>
Jun 01 16:10:38 router-localhost systemd[1]: NetworkManager.service: Start requ>
Jun 01 16:10:38 router-localhost systemd[1]: NetworkManager.service: Failed wit>
Jun 01 16:10:38 router-localhost systemd[1]: Failed to start NetworkManager.ser>
lines 1-15/15 (END)

and it's not running.
I tried with:

sudo systemctl start NetworkManager

but it doesn't work:

sudo systemctl status NetworkManager
× NetworkManager.service - Network Manager
     Loaded: loaded (/usr/lib/systemd/system/NetworkManager.service; enabled; vendor preset: enabled)
     Active: failed (Result: core-dump) since Wed 2022-06-01 16:51:01 BST; 22min ago
       Docs: man:NetworkManager(8)
    Process: 18471 ExecStart=/usr/sbin/NetworkManager --no-daemon (code=dumped, signal=SEGV)
   Main PID: 18471 (code=dumped, signal=SEGV)
        CPU: 213ms

Jun 01 16:51:01 router-localhost systemd[1]: NetworkManager.service: Main process exited, code=dumped, status=11/SEGV
Jun 01 16:51:01 router-localhost systemd[1]: NetworkManager.service: Failed with result 'core-dump'.
Jun 01 16:51:01 router-localhost systemd[1]: NetworkManager.service: Scheduled restart job, restart counter is at 5.
Jun 01 16:51:01 router-localhost systemd[1]: Stopped NetworkManager.service - Network Manager.
Jun 01 16:51:01 router-localhost systemd[1]: NetworkManager.service: Start request repeated too quickly.
Jun 01 16:51:01 router-localhost systemd[1]: NetworkManager.service: Failed with result 'core-dump'.
Jun 01 16:51:01 router-localhost systemd[1]: Failed to start NetworkManager.service - Network Manager.

journalctl -xeu network.service
░░ 
░░ The process' exit code is 'exited' and its exit status is 1.
Jun 01 16:46:24 router-localhost systemd[1]: network.service: Failed with resul>
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ The unit network.service has entered the 'failed' state with result 'exit-co>
Jun 01 16:46:24 router-localhost systemd[1]: network.service: Unit process 1631>
Jun 01 16:46:24 router-localhost systemd[1]: Failed to start network.service - >
░░ Subject: A start job for unit network.service has failed
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ A start job for unit network.service has finished with a failure.
░░ 
░░ The job identifier is 11203 and the job result is failed.
Jun 01 16:46:24 router-localhost systemd[1]: network.service: Consumed 4.361s C>
░░ Subject: Resources consumed by unit runtime
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ The unit network.service completed and consumed the indicated resources.

[5]+  Stopped                 journalctl -xeu network.service

When I restart, it briefly goes to active and then inactive again:

[FranceBB@router-localhost Downloads]$ sudo systemctl start NetworkManager
You have new mail in /var/spool/mail/FranceBB
[FranceBB@router-localhost Downloads]$ sudo systemctl status NetworkManager
● NetworkManager.service - Network Manager
     Loaded: loaded (/usr/lib/systemd/system/NetworkManager.service; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2022-06-01 17:15:45 BST; 263ms ago
       Docs: man:NetworkManager(8)
   Main PID: 27231 (NetworkManager)
      Tasks: 4 (limit: 76426)
     Memory: 4.1M
        CPU: 143ms
     CGroup: /system.slice/NetworkManager.service
             └─ 27231 /usr/sbin/NetworkManager --no-daemon

Jun 01 17:15:45 router-localhost NetworkManager[27231]: <info>  [1654100145.9838] device (eth0): state change: unmanaged -> unavailable (reason 'managed', sys-iface-state: 'external')
Jun 01 17:15:45 router-localhost NetworkManager[27231]: <info>  [1654100145.9859] manager: (vboxnet0): new Ethernet device (/org/freedesktop/NetworkManager/Devices/6)
Jun 01 17:15:45 router-localhost NetworkManager[27231]: <info>  [1654100145.9869] manager: (virbr0): new Bridge device (/org/freedesktop/NetworkManager/Devices/7)
Jun 01 17:15:45 router-localhost NetworkManager[27231]: <info>  [1654100145.9898] device (virbr0): state change: unmanaged -> unavailable (reason 'connection-assumed', sys-iface-state: 'external')
Jun 01 17:15:45 router-localhost NetworkManager[27231]: <info>  [1654100145.9906] device (virbr0): state change: unavailable -> disconnected (reason 'connection-assumed', sys-iface-state: 'external')
Jun 01 17:15:45 router-localhost NetworkManager[27231]: <info>  [1654100145.9918] device (virbr0): Activation: starting connection 'virbr0' (f2c89acb-b2c4-4542-b415-7902fc8fe109)
Jun 01 17:15:45 router-localhost NetworkManager[27231]: <info>  [1654100145.9921] device (wlp2s0): driver supports Access Point (AP) mode
Jun 01 17:15:45 router-localhost NetworkManager[27231]: <info>  [1654100145.9928] manager: (wlp2s0): new 802.11 Wi-Fi device (/org/freedesktop/NetworkManager/Devices/8)
Jun 01 17:15:45 router-localhost NetworkManager[27231]: <info>  [1654100145.9930] device (wlp2s0): state change: unmanaged -> unavailable (reason 'managed', sys-iface-state: 'external')
Jun 01 17:15:46 router-localhost NetworkManager[27231]: <info>  [1654100146.0115] device (wlp2s0): set-hw-addr: set MAC address to 3E:98:79:F4:86:E4 (scanning)
[FranceBB@router-localhost Downloads]$ sudo systemctl status NetworkManager
× NetworkManager.service - Network Manager
     Loaded: loaded (/usr/lib/systemd/system/NetworkManager.service; enabled; vendor preset: enabled)
     Active: failed (Result: core-dump) since Wed 2022-06-01 17:15:47 BST; 2s ago
       Docs: man:NetworkManager(8)
    Process: 27277 ExecStart=/usr/sbin/NetworkManager --no-daemon (code=dumped, signal=SEGV)
   Main PID: 27277 (code=dumped, signal=SEGV)
        CPU: 159ms

Jun 01 17:15:47 router-localhost systemd[1]: NetworkManager.service: Main process exited, code=dumped, status=11/SEGV
Jun 01 17:15:47 router-localhost systemd[1]: NetworkManager.service: Failed with result 'core-dump'.
Jun 01 17:15:47 router-localhost systemd[1]: NetworkManager.service: Scheduled restart job, restart counter is at 5.
Jun 01 17:15:47 router-localhost systemd[1]: Stopped NetworkManager.service - Network Manager.
Jun 01 17:15:47 router-localhost systemd[1]: NetworkManager.service: Start request repeated too quickly.
Jun 01 17:15:47 router-localhost systemd[1]: NetworkManager.service: Failed with result 'core-dump'.
Jun 01 17:15:47 router-localhost systemd[1]: Failed to start NetworkManager.service - Network Manager.
[FranceBB@router-localhost Downloads]$

Any thoughts?
I'm currently without internet and I'm writing from Windows XP.


In attachment you're gonna find

journalctl -u NetworkManager
log.txt

[FranceBB]

Today is my second day without internet on Fedora and I'm reading (from Chromium 92 on XP) that Fedora 36 introduced some changes in the network stack and I found the issue!

Basically I've been using Fedora for several years and I have over 100 network saved as legacy ifcfg files.
Over the years, the NetworkManager has always been able to read both keyfiles in /etc/NetworkManager/system-connections/ AND ifcfg scripts in /etc/sysconfig/network-scripts/ but this is no longer the case with Fedora 36.
Removing all my hundred of ifcfg config solved the issue and the network manager now started successfully with a simple: sudo systemctl start NetworkManager

Now I'm trying to convert all my legacy ifcfg to keyfiles to avoid losing them.

EDIT: I managed to migrate them with

nmcli connection migrate

Happy Platinum Jubilee, everyone!

[jbkt23]

nmcli connection migrate

Did you have to run this once for each individual config or once for all configs? I guess I could read the man page but even so the explanations leave a little room for interpretation.

[FranceBB]

Did you have to run this once for each individual config or once for all configs?

It didn't work for all configs, so I went step by step to see which ones it didn't like.
Turns out there were some old Wi-Fi using WEP which it didn't like, so once I isolated them out, it worked for all configs, this is why I solved the issue.
In particular, it came down to nm-device-wifi.c in the Network Manager source code.
Thomas Haller fixed it upstream as soon as I reported it today: https://gitlab.freedesktop.org/Netwo...032dc42fe2b3e4

Cheers,
Frank

[pmoros]

This was quite useful, thanks!