KVM bridged network blocks VM to VM traffic

PilotJLR · #1 29th November 2009, 05:40 PM

Hello,
I'm using KVM on Fedora 12 x86_64 to run several virtual machines. I've set up a bridged network interface, as I need these virtual machines to co-mingle on the same network as the physical machines.

The bridge is setup as per the scripts below. I'm using the default F12 sysctl.conf.

The problem is this... although virtual machines CAN access other machines on the network, including the Internet, virtual machines CANNOT communicate with each other. No pings, no file sharing, etc.

I don't know why these VM's are unable to communicate with each other. I would appreciate any ideas or suggestions!
Thanks!!

br0 interface:

Code:

# cat /etc/sysconfig/network-scripts/ifcfg-br0 
DEVICE=br0
TYPE=Bridge
BOOTPROTO=dhcp
ONBOOT=yes
DELAY=0

eth0 interface

Code:

# cat /etc/sysconfig/network-scripts/ifcfg-eth0 
# Intel Corporation 82573L Gigabit Ethernet Controller
DEVICE=eth0
HWADDR=00:19:D1:31:E9:E3
ONBOOT=yes
TYPE=Ethernet
#BOOTPROTO=dhcp
NAME="System eth0"
UUID=5fb06bd0-0bb0-7ffb-45f1-d6edd65f3e03
PEERDNS=yes
PEERROUTES=yes
BRIDGE=br0
MTU=9000

sysctl.conf

Code:

# cat /etc/sysctl.conf 
# Kernel sysctl configuration file for Red Hat Linux
#
# For binary values, 0 is disabled, 1 is enabled.  See sysctl(8) and
# sysctl.conf(5) for more details.

# Controls IP packet forwarding
net.ipv4.ip_forward = 0

# Controls source route verification
net.ipv4.conf.default.rp_filter = 1

# Do not accept source routing
net.ipv4.conf.default.accept_source_route = 0

# Controls the System Request debugging functionality of the kernel
kernel.sysrq = 0

# Controls whether core dumps will append the PID to the core filename.
# Useful for debugging multi-threaded applications.
kernel.core_uses_pid = 1

# Disable netfilter on bridges.
net.bridge.bridge-nf-call-ip6tables = 0
net.bridge.bridge-nf-call-iptables = 0
net.bridge.bridge-nf-call-arptables = 0

brctl show

Code:

# brctl show
bridge name	bridge id		STP enabled	interfaces
br0		8000.0019d131e9e3	no		eth0
							vnet0
							vnet1
virbr0		8000.000000000000	yes

---------- Post added at 10:53 AM CST ---------- Previous post was at 10:11 AM CST ----------

One more piece of info here... the bridge does see the virtual machines's MAC addresses:

Code:

# brctl showmacs br0
port no	mac addr		is local?	ageing timer
  1	00:0f:b5:23:73:05	no		 194.94
  1	00:15:af:75:9d:bb	no		 209.88
  1	00:19:d1:31:e9:e3	yes		   0.00
  1	00:22:6b:5f:1f:3d	no		   0.01
  1	00:23:32:c9:e9:f8	no		 159.23
  1	00:30:18:a9:b2:76	no		   0.00
  2	32:d2:69:48:a6:64	yes		   0.00
  2	52:54:00:07:ca:6b	no		 159.30
  3	52:54:00:6f:7a:12	no		 208.36
  3	f6:ac:62:68:f5:8b	yes		   0.00

The bold entries correspond to the virtual machine's addresses.

---------- Post added at 11:40 AM CST ---------- Previous post was at 10:53 AM CST ----------

Although this certainly may be a case of user error, I'm not positive that this is not a bug. Therefore: https://bugzilla.redhat.com/show_bug.cgi?id=542405

Of course, if anyone does have ideas, please do let me know!

David Becker · #2 30th November 2009, 12:03 PM

Maybe your firewall is in the way?

On the host, something like the following rule can be used:

Code:

iptables -I FORWARD -m physdev --physdev-is-bridged -j ACCEPT

David

PilotJLR · #3 30th November 2009, 01:03 PM

I appreciate your reply! I already had that rule in place... I'm inclined to think it's not the firewall, but I'd appreciate your input on this:

Code:

# iptables -L -v
Chain INPUT (policy ACCEPT 8566K packets, 12G bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     udp  --  virbr0 any     anywhere             anywhere            udp dpt:domain 
    0     0 ACCEPT     tcp  --  virbr0 any     anywhere             anywhere            tcp dpt:domain 
    0     0 ACCEPT     udp  --  virbr0 any     anywhere             anywhere            udp dpt:bootps 
    0     0 ACCEPT     tcp  --  virbr0 any     anywhere             anywhere            tcp dpt:bootps 

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     all  --  any    any     anywhere             anywhere            PHYSDEV match --physdev-is-bridged 
    0     0 ACCEPT     all  --  any    virbr0  anywhere             192.168.122.0/24    state RELATED,ESTABLISHED 
    0     0 ACCEPT     all  --  virbr0 any     192.168.122.0/24     anywhere            
    0     0 ACCEPT     all  --  virbr0 virbr0  anywhere             anywhere            
    0     0 REJECT     all  --  any    virbr0  anywhere             anywhere            reject-with icmp-port-unreachable 
    0     0 REJECT     all  --  virbr0 any     anywhere             anywhere            reject-with icmp-port-unreachable 

Chain OUTPUT (policy ACCEPT 4634K packets, 417M bytes)
 pkts bytes target     prot opt in     out     source               destination

I believe iptables is disabled by default for bridges in F12, anyway. My reference is the green box in the middle of this page:
http://docs.fedoraproject.org/virtua..._with_PXE.html

I have those entries in my sysctl.conf... of course, any other ideas would be appreciated! At the moment, I can't think of anything else to check.

David Becker · #4 30th November 2009, 02:59 PM

Maybe this has to be changed:

Code:

# Controls IP packet forwarding
net.ipv4.ip_forward = 0

David

PilotJLR · #5 30th November 2009, 04:20 PM

Thanks - I tried setting this to '1', and I also tried setting this to '1' and then restarting libvirtd, but the problem unfortunately remains.

beaker_ · #6 30th November 2009, 04:38 PM

Sorry, I'm away from my box so I can't confirm anything.

I believe the virtual machines are networked together through the vmnet interface but clearly is removed infavor of virbr#. So with virbr# bridged and if you suspect the bridge is the problem, why not unbridge them and try mapping (through iptables) to an alias. So with virbr0 mapped to eth0:1 and virbr1 mapped to eth0:2, can you then ping eth0:2 & eth0:1 from either virtual machine and the host?

David Becker · #7 30th November 2009, 05:25 PM

Quote:

Originally Posted by PilotJLR

Thanks - I tried setting this to '1', and I also tried setting this to '1' and then restarting libvirtd, but the problem unfortunately remains.

Did you just edit the file or did you edit the file and then restart networking or the machine?

Maybe you should do a traceroute from one guest to the other? Possibly while you're running 'tcpdump -i br0' on the host.

David

PilotJLR · #8 30th November 2009, 09:26 PM

Quote:

Originally Posted by David Becker

Did you just edit the file or did you edit the file and then restart networking or the machine?

Maybe you should do a traceroute from one guest to the other? Possibly while you're running 'tcpdump -i br0' on the host.

David

I did an "echo 1 > /proc/sys/net/ipv4/ip_forward" to turn on IP forwarding.

I will try doing the traceroute and tcpdump. I might not be able to do this until late afternoon / evening (central US), but I will report back!
Beaker - likewise, I will give that a shot and report back.

---------- Post added at 01:19 PM CST ---------- Previous post was at 11:27 AM CST ----------

Some more info...

I ran both a ping and a tracert (the guests are windows 2003 with no firewall) between 2 guests while running tcpdump on the host.
In each case, tcpdump DID list an ARP request from the guest initiating the ping. In each case, the ping target did not respond to the ARP (and did not send anything at all back).

Despite this, "brctl showmacs br0" does correctly list the MAC address of each guest.

---------- Post added at 02:16 PM CST ---------- Previous post was at 01:19 PM CST ----------

AH HA! It's not fixed, but it's getting closer.

If I manually add static ARP entries into EACH virtual machine, then they are able to communicate. I do not think this is due to blocking the ARP broadcast, because only 1 static entry (the ping source) was not successful. Once the ping ingresses to the destination, it's address should be learned... but it was not (or it was incorrect).

Which leads me to this question... why does the bridge show multiple MAC's per port?

Code:

# brctl showmacs br0
port no	mac addr		is local?	ageing timer
  1	00:15:af:75:9d:bb	no		  91.15
  1	00:19:d1:31:e9:e3	yes		   0.00
  1	00:22:6b:5f:1f:3d	no		   0.00
  1	00:23:32:c9:e9:f8	no		  63.07
  1	00:30:18:a9:b2:76	no		   0.39
  3	52:54:00:07:ca:6b	no		  62.55
  2	52:54:00:4c:c4:e0	no		  62.55
  3	ee:f2:a5:a1:58:33	yes		   0.00
  2	f6:32:32:3e:47:d8	yes		   0.00

The entries with "local" set to "no" are the correct ones, as shown inside the guest and in virt-manager.

The local "yes" entries are for vmnet0 and vmnet1... I'm not sure what these are!

---------- Post added at 03:26 PM CST ---------- Previous post was at 02:16 PM CST ----------

Problem Solved!

Thanks for the input, guys! Here is what it was.

Since I was using Windows Server 2003 guests, I chose to use a Intel e1000 virtual NIC, since that driver is included in 2003 by default and works well (oh physical hardware, at least).

I solved these problems by removing the e1000 NIC and adding a Realtek 8139 NIC using virt-manager. I created a CD ISO with the drivers from realtek's website and gave it to the guest. Once the 2003 guest had the driver, the network worked and inter-guest communication worked.

The tip off was when I kept watching "arp -a" inside a Windows Server guest. I noticed that pings received from the other guest (WITH a static arp entry) all came in as "00-00-00-00-00 invalid". After this, I installed Wireshark in the guest sending the pings and watched. Wireshark listed each source MAC as being from a Realtek NIC. I assume that the e1000 NIC was given an automatically generated Realtek address. I don't know if this is supposed to be kosher or not, but it certainly did not work. The problem was either the Realtek MAC + e1000 OR the problem was in the e1000 emulation itself.

I still think this is a bug, and I'll be updating the bugzilla entry with this info.