Bridge broken after docker install

[boscorillium]

I have a very basic setup on a fresh install of Fedora 25 Workstation for some virtual machines through virt-manager. I have a bridge (br0) setup for the VMs to use with the ethernet as its slave device. I've tried a number of different things but no matter what I do when I install docker, the bridge breaks for the VMs. The host will still be connected and reachable but then the VMs will time out trying to raise their interfaces. Remove docker and the problem goes away.

I've done a bit of diffing between with docker and without (through a LVM snapshot) and have not come up with anything obvious.

I've added the bridge after docker, I've removed docker's bridge and it seems no matter what I do docker breaks my bridge for my VMs. I would love to have them both working on this one machine and know it has to be possible (I mean the world wouldn't make sense if it wasn't possible) but I've been unable to figure out what the magic bullet is in quite a few hours of trial and error/reading.

I'm using Network Manager's cli to setup the bridge, similar to what can be found here (https://www.server-world.info/en/not...edora_23&p=kvm). But I do the ipv4.method as auto instead of static.

Does anyone have any ideas? TIA....



tim

[rexrf]

have you looked for a difference in the bridges network scripts before an after? I would use cockpit http://cockpit-project.org/ you can configure just about anything inside of it, including docker and bridges, all right out of the box.

sudo dnf -y install cockpit*
sudo systemctl enable cockpit.socket
sudo systemctl start cockpit
sudo firewall-cmd --permanent --add-port=9090/tcp
firewall-cmd --reload

Another thing you can do is follow the logs during the bridge failure. Run this when the bridge is working properly, all the way through failure. Then open it and get a cup of coffee and dig in

journalctl -f >> ~/Documents/BridgeLogs.txt

[boscorillium]

Thanks for the suggestions. I have done / am doing that and the one thing that immediately stands out is that when I install docker I get a lot of messages of this variety (about failure):

Jan 02 09:46:26 localhost.localdomain firewalld[927]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -C POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE' failed:
Jan 02 09:46:26 localhost.localdomain audit: NETFILTER_CFG table=nat family=2 entries=63
Jan 02 09:46:26 localhost.localdomain firewalld[927]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -C DOCKER -i docker0 -j RETURN' failed:
Jan 02 09:46:26 localhost.localdomain audit: NETFILTER_CFG table=nat family=2 entries=64
Jan 02 09:46:26 localhost.localdomain firewalld[927]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -D FORWARD -i docker0 -o docker0 -j DROP' failed:
Jan 02 09:46:26 localhost.localdomain firewalld[927]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -C FORWARD -i docker0 -o docker0 -j ACCEPT' failed:
Jan 02 09:46:26 localhost.localdomain audit: NETFILTER_CFG table=filter family=2 entries=105
Jan 02 09:46:26 localhost.localdomain firewalld[927]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -C FORWARD -i docker0 ! -o docker0 -j ACCEPT' failed:
Jan 02 09:46:26 localhost.localdomain audit: NETFILTER_CFG table=filter family=2 entries=106
Jan 02 09:46:26 localhost.localdomain firewalld[927]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -C FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT' failed:
Jan 02 09:46:26 localhost.localdomain audit: NETFILTER_CFG table=filter family=2 entries=107
Jan 02 09:46:26 localhost.localdomain firewalld[927]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -C FORWARD -o docker0 -j DOCKER' failed:


Not sure if this is the source of my pain or not. When I remove docker the bridge for the VMs does not work right away, I have to reboot. But then it goes back to working perfectly....



Thanks,


Tim

---------- Post added at 10:50 AM ---------- Previous post was at 10:46 AM ----------

And also, the VM doesn't have trouble raising its interface after I simply install docker, it's only after I start docker that there's an issue (all without reboot).

[rexrf]

I know very little about docker, but my guess is that docker0 (the interface it automatically creates when you install it) is itself a bridge. I'm assuming you only have a single NIC, and if both of your bridges (docker0 and br0), are trying to control that interface, it may be causing you problems. But again, I know very little about docker.

I'd refer to dockers documentation on customizing networks:

https://docs.docker.com/engine/userg...ustom-docker0/

[martdj]

Did you manage to solve this problem? I have exactly the same problem so would love to know the solution

[InviteCiel]

I'm fighting this issue too, but on Debian 8. KVM + Docker = my broken heart.

Seems that this issue happens because Docker loads

br_netfilter

module and sets

net.bridge.bridge-nf-call-arptables
net.bridge.bridge-nf-call-iptables
net.bridge.bridge-nf-call-ip6tables

kernel parameters to 1 (or it is default module config) so all bridged traffic is now subject to iptables filtering.

See following links about br_netfilter:

• Effect of br_netfilter on FORWARD behavior
• Issue about Docker and br_netfilter parameters
• net.bridge.bridge-nf-call-* and sysctl.conf

I haven't yet figured out how to fix this issue, because simply blacklisting br_netfilter won't go - Docker needs this module to filter bridged connections between containers that are created with 'link' option. Filtering bridged connections is necessary to Docker to properly handle '--icc' flag, which enables or disables inter-container communication.

I suppose that we need to figure out some Iptables rule that will ACCEPT all bridged traffic except for traffic going through Docker bridges. One way is to add such rules for each of your custom bridges:

iptables -A FORWARD -i <your-bridge> -o <your-bridge> -j ACCEPT

But I think it is not flexible solution.

Could someone propose a better approach?

[teeks99]

I just ran into this problem as well, on Ubuntu 16.04. It occurred on a random update, not a fresh install, so something about the docker package must have changed how it was working.

Anyway, I just set the following in my /etc/sysctl.conf file and it seemed to work.

net.bridge.bridge-nf-call-arptables = 0
net.bridge.bridge-nf-call-ip6tables = 0
net.brdige.bridge-nf-call-iptables = 0

Did I miss something, is there a reason this isn't good to do? My bridged VMs are back to working correctly and docker seems to be working fine as well.

[teeks99]

I just ran into this problem as well, on Ubuntu 16.04. It occurred on a random update, not a fresh install, so something about the docker package must have changed how it was working.

Anyway, I just set the following in my /etc/sysctl.conf file and it seemed to work.

net.bridge.bridge-nf-call-arptables = 0
net.bridge.bridge-nf-call-ip6tables = 0
net.brdige.bridge-nf-call-iptables = 0

Did I miss something, is there a reason this isn't good to do? My bridged VMs are back to working correctly and docker seems to be working fine as well.

This stopped working after moving to a new machine. It appears that now docker is clobbering those settings after the system runs the startup equivalent of

sysctl -p

. Instead I added a new file to /etc/network/if-pre-up.d/ called kvm_bridge_iptables, with the following content:

#!/bin/sh
iptables -A FORWARD -i br0 -o br0 -j ACCEPT

[bobx001]

one must wonder how much network performance loss in the form of latency is added with all of these layers between the customer and the server.

I will try this at home, although I don't use docker, and see what stable setup I can come up with, although I won't do it over coffee, more like a few beers.....