How to setup a bridge for KVM virtual machines

[Kobuck]

I want to use KVM on my new F19 installation. My blocker for using Fedora's standard KVM for virtual machines was that the machines were masked from the rest of my network by NAT. Turning off NetworkManager and adding a bridge were(are) outside my comfort zone so I've waited for several releases hoping things would get simpler ( for me).

Nothing has changed so I guess I will have to. Most of the material I have found is aimed at RedHat 6 configuration with the exception of a good howto guide circa F15 by stevea.

I've assembled the following plan, but I would appreciate assistance/input either with the following plan or a good guide at F19 level of development.

My concerns with following include:
1) can I edit the ifctl-enp5s0 file and effect the P14p1 interface? (p14p1 does not exist in network-scripts)
2) is /etc/sysctl.conf still the right place for the forwarding flag? there's nothing in the file right now
3) do I need to do anything else for network.service to start on boot?

Viewers please DON'T use the attached plan without seeing feedback from any more knowledgeable members. I have low confidence in it at this at this point.

The plan:
---------------------------------------------------------
Operate as root.

su –

# Locate the NIC device to be bridged and take it down.

ifconfig
.
.
.
p14p1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.0.1.109 netmask 255.0.0.0 broadcast 10.255.255.255
inet6 fe80::xxxx:xxxx:xxxx:xxxx prefixlen 64 scopeid 0x20<link>
ether 90:2b:34:d2:ba:7a txqueuelen 1000 (Ethernet)
RX packets 140636 bytes 202161513 (192.7 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 83067 bytes 5755709 (5.4 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
.
.

ifdown p14p1


# In '/etc/sysconfig/network-scripts' find the ifcfg-*** file which contains a
# HWADDR= line that matches the ether address for p14p1. Edit it to add a line
# BRIDGE=brg0 .

# The file /etc/sysconfig/network-scripts/ifcfg-enp5s0 contains:

PEERROUTES="yes"
IPV6INIT="yes"
UUID="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" (I masked out UUID)
IPV6_PEERDNS="yes"
DEFROUTE="yes"
PEERDNS="yes"
IPV4_FAILURE_FATAL="no"
HWADDR="90:2B:34: D2:BA:7A"
BOOTPROTO="dhcp"
IPV6_DEFROUTE="yes"
IPV6_AUTOCONF="yes"
IPV6_FAILURE_FATAL="no"
IPV6_PEERROUTES="yes"
TYPE="Ethernet"
ONBOOT="yes"
NAME="enp5s0"
BRIDGE="brg0" <--- add this line


# Create a new script file to define the bridge:
#
# Add file /etc/sysconfig/network-scripts/ifcfg-brg0 to contain:

DEVICE="brg0"
TYPE="Bridge"
BOOTPROTO="dhcp"
ONBOOT="yes"
DELAY=0

# Change from 'NetworkManager' to 'network' service

systemctl disable NetworkManager
systemctl stop NetworkManager
systemctl enable network
systemctl start network


# Add a line to /etc/sysctl.conf to setup forwarding:

net.ipv4.ip_forward = 1

[yetinet]

On a fresh F19 install I have such a bridge working now. Unlike with CentOS and previous Fedoras, I am using NetworkManager, and not the network.service.

I used the Gnome applet to turn off em1, then entered 'Network Settings' (nm-connection-settings) to create the bridge (Bridge0), and added the Ethernet connection as a slave. I turned off STP.

It took a little mucking about to get the bridge active. Infact I used the nmcli command to activate it. I think the ifup command wasn't good enough. Anyway, here are the resulting ifcfg members created by NM.

tail -n 99 ifcfg-[Bbe]*
==> ifcfg-Bridge0 <==
DEVICE=bridge0
STP=no
TYPE=Bridge
BOOTPROTO=dhcp
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
NAME=Bridge0
UUID=d2d5448a-51e2-4159-bbef-c9e82b2586de
ONBOOT=yes
PEERDNS=yes
PEERROUTES=yes
IPV6_PEERDNS=yes
IPV6_PEERROUTES=yes

==> ifcfg-bridge0em1 <==
TYPE=Ethernet
NAME=bridge0em1
UUID=d4dddc92-1895-4cdc-badc-b9c094642d8f
ONBOOT=yes
BRIDGE=d2d5448a-51e2-4159-bbef-c9e82b2586de
HWADDR=00:22:19:26:EB:B7

==> ifcfg-enp4s0 <==
IPV6INIT="yes"
NAME="enp4s0"
DEFROUTE="yes"
UUID="467fe6ad-e792-47b2-b28b-8c062b1abd6d"
IPV4_FAILURE_FATAL="no"
BOOTPROTO="dhcp"
IPV6_DEFROUTE="yes"
IPV6_AUTOCONF="yes"
IPV6_FAILURE_FATAL="no"
TYPE="Ethernet"
ONBOOT=no
HWADDR=00:22:19:26:EB:B7
PEERDNS=yes
PEERROUTES=yes
IPV6_PEERDNS=yes
IPV6_PEERROUTES=yes

[yetinet]

Maybe I spoke too soon. The bridge is working fine for my host machine, but I haven't actually got a KVM guest machine to use it yet.

[yetinet]

Maybe I spoke too soon. The bridge is working fine for my host machine, but I haven't actually got a KVM guest machine to use it yet.

It turned out to be firewalld preventing the KVM guest from using the bridge. Using nm-conneciton-editor I changed the security zone of the bridge from 'public' to 'trusted' and then the network worked.

Firewalld is yet another leaning curve I'll have to climb sooner rather than later, but at least I can now do it on virtual machines.

EDIT: Don't do this!

[yetinet]

... and having spent a day or so investigating firewalld, and NetworkManager, I've backed off and reverted to virbr0 (NAT).

Putting my bridge0 into the "trusted" zone completely opens the firewall on my host which is not acceptable.

I will revisit this one day, but until then any VMs that need to be exposed to the LAN can live on a CentOS host rather than my Fedora workstation.

[yetinet]

Guess what?

Since I've upgraded to libX11-1.6.0-1.fc19.x86_64 and kernel-3.9.9-302.fc19.x86_64

Bridging works perfectly under NetworkManager.

I have had no hangs or crashes starting and stopping windows, and centos guests.

The only issue I did have was activating the bridge the first time. You need to specify DELAY=0 in /etc/sysconfig/network-scripts/ifcfg-bridge0 before it will come up.

You also have to allow traffic through the hosts firewalld. It seems that libvirtd sets things up for the NAT'd guests only. so I run this script:

#!/bin/sh
iptables -I FORWARD -i bridge0 -j ACCEPT
iptables -I FORWARD -o bridge0 -j ACCEPT

Now it is good to go. The nm-applet thingy on the top of the screen has a little cross on the network, but the network is fully operational, and unlike on F18, thunderbird recognises that the network is up without me having to tell it that is online

[Kobuck]

Well....

Looks like I needed a little more patience.

https://fedoraproject.org/wiki/Chang...anagerBridging

I appreciate your input yetinet, but I don't see how to convert the info to effective actions.
I thought there would be many people trying to make VM's visible on the local network.

Maybe I didn't ask the right question on the right forum?

[yetinet]

Hi Yetinet,

I am new to the forum. I saw your post about setting up a bridge for your guest to use without disabling NetworkManager.

Could you send me the steps, I'm kind of a noob.

Kerrell

Hi Kerrell,

Firstly I would advise you (a kind of noob), not to expect me to spend any effort answering your questions in private messages. If I did that then nobody gets a chance to improve on my feeble answers, and nobody but you could possibly get any benefit from my efforts.

Yeah, I've done this on two separate machines (home and work), both are latest F19 (currently 3.10.5-201.fc19.x86_64). The only outstanding issue is cosmetic - the applet at the top right hand corner of screen has a little 'x' on it indicating that the network is not active. It is active, and everything on the host and my VMs just works.

Steps... As I recall it went something lilke this.
- click on gnome network applet
- click network settings
- click '+' to add a new connection
- choose Bridge
- set the connection name - I called mine Bridge0 rather than the default "Bridge connection 1"; The interface name was already 'bridge0'.
-hit 'Add' to add a bridged connection
-choose Ethernet as a connection type
-Hit the pull-down on Device MAC address and choose the physical device (em1 in my case)
-I also change the connection name from "bridge0 slave 1" to simply 'Bridge0em1'

Thats about it. once I'd saved it, I reviewed /etc/sysconfig/network-scripts/ifcfg-* and in particular added "DELAY=0' after "STP=yes". I probably should have done this by setting "Forward Delay" to zero, but I used 'vi' instead. I also reviewed all of the ONBOOT= entries and then rebooted.

There was another thing too - you have to enable packets to be forwarded from the wire to your VMs through the bridge. I wrote a little bash script to do that and run it from rc.local, though there's sure to be a better way.

Cheers,
Yeti.

EDIT: Here's the script
#!/bin/sh

# If I put bridge0 in trusted zone then firewalld allows anything from
# bridg0 on both INPUT and FORWARD chains !
# So, I've put bridge0 back into the default public zone, and this script
# adds rules to allow anything to and from bridge0 to be FORWARDed but not INPUT.

BRIDGE=bridge0
iptables -I FORWARD -i $BRIDGE -j ACCEPT
iptables -I FORWARD -o $BRIDGE -j ACCEPT

[Hrumph]

In Fedora 20, after setting up the bridge, the following firewall adjustment worked for me:

firewall-cmd --permanent --direct --passthrough ipv4 -I FORWARD -m physdev 
--physdev-is-bridged -j ACCEPT

[AdamW]

the intent is simply that bridge devices are not firewalled, but there's a bug that gets in the way. See https://bugzilla.redhat.com/show_bug.cgi?id=634736 . The 'approved' recommendation is to create file /etc/udev/rules.d/99-bridge.rules with contents:

ACTION=="add", SUBSYSTEM=="module", KERNEL=="bridge", RUN+="/usr/lib/systemd/systemd-sysctl --prefix=/proc/sys/net/bridge"

which should sort the problem fairly solidly. I have a page on bridging with NM at https://www.happyassassin.net/2014/0...014-fedora-21/ .

[graemev]

OK, that's yet another day of my life I won't get back. Every time I upgrade Fedora I lose a day to getting virtual machines to run again, always it's the bridge which is the issue. This time I made lots of notes of what I did/tried.
General observations:

1: Tried a minimal mod of existing setup (simple wired Ethernet port called eno1 (that's an oh)
2: used the GUI
3: Gave up and did a literal copy (then edit) of the stuff from fedora23 PDF
4: Added back some bits from my Fedora19 setup (where I was upgrading from)

i: There is a "confusion" with NAME and DEVICE , possibly NetworkManager uses one and "everyone else uses the other
ii: The guide for using the GUI seems , just seems wrong . It ends up with too many scripts.
iii: What I believe should happen is "before" eno1 has a IP address, after it has none but br0 does have
an address. The GUI seems to leave eno1 with an IP address , can't see that will work?
iv: The documentation seems to have lots of information removed after Fedora18. The older documentation is invaluable. So what I ended up with (with redactions) is:

# pr -f *

2016-06-19 16:25                    ifcfg-br0                     Page 1
DEVICE=br0
TYPE=Bridge
IPADDR=10.xxx.xxx.xxx
NETMASK=255.255.252.0
ONBOOT=yes
BOOTPROTO=static
NM_CONTROLLED=no
DELAY=0
# -- not on RedHat PDF
DNS1=10.XX.YY.ZZ
DOMAIN=home

2016-06-19 16:25                    ifcfg-eno1                    Page 1
DEVICE=eno1
TYPE=Ethernet
HWADDR=xx:yy:80:A4:EA:zz
BOOTPROTO=none
ONBOOT=yes
NM_CONTROLLED=no
BRIDGE=br0

Which shows up as:

[root@real for-post]# ifconfig
br0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.XXX.XXX.XXX  netmask 255.255.252.0  broadcast 10.XXX.3.255
        inet6 fe80::96de:80ff:fea4:eabe  prefixlen 64  scopeid 0x20<link>
        ether 94:de:80:a4:ea:be  txqueuelen 1000  (Ethernet)
        RX packets 49639  bytes 45012757 (42.9 MiB)
        RX errors 0  dropped 4  overruns 0  frame 0
        TX packets 23394  bytes 44104151 (42.0 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eno1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::96de:80ff:fea4:eabe  prefixlen 64  scopeid 0x20<link>
        ether 94:de:80:a4:ea:be  txqueuelen 1000  (Ethernet)
        RX packets 105556  bytes 99184211 (94.5 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 82585  bytes 52226235 (49.8 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device interrupt 20  memory 0xf0400000-f0420000  

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
    ...elided...
virbr0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
    ...elided...
vnet0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
    ...elided...
    
# route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
10.XXX.0.0      0.0.0.0         255.255.252.0   U     0      0        0 br0
link-local      0.0.0.0         255.255.0.0     U     1010   0        0 br0
192.168.124.0   0.0.0.0         255.255.255.0   U     0      0        0 virbr0

# brctl show
bridge name	bridge id		STP enabled	interfaces
br0		8000.94de80a4eabe	no		eno1
							       vnet0
virbr0		8000.525400272b8e	yes	virbr0-nic

(the GUI version had STP = ON)

[graemev]

To add insult, it seems 1: fedora23 does NOT use the systemd networking startup , still using old style sysV (not a bad choice, but the manpages disagree) , the killer however is that networking is disabled in all run states:


[root@Desktop]# systemctl is-enabled network.service
network.service is not a native service, redirecting to systemd-sysv-install
Executing /usr/lib/systemd/systemd-sysv-install is-enabled network
disabled

[root@Desktop]# /usr/lib/systemd/systemd-sysv-install enable network

[root@Desktop]# systemctl is-enabled network.service
network.service is not a native service, redirecting to systemd-sysv-install
Executing /usr/lib/systemd/systemd-sysv-install is-enabled network
enabled

Man pages say:

]$ systemctl status --all | grep network
│ ├─network.service
│ │ └─3172 grep --color=auto network
● org.freedesktop.network1.busname - Network Service Bus Name
Loaded: loaded (/usr/lib/systemd/system/org.freedesktop.network1.busname; static; vendor preset: disabled)
Docs: man:systemd-networkd.service(8) ....

So lookup man:systemd-networkd.service(8) as suggested:

The configuration files are read from the files ..... and....
administration network directory /etc/systemd/network.

No they're not, still sysV style.

[stevea]

You re-lit a 2/3 yo thread.
On f23 libvirtd/virt-manager will create the bridge for you.