KVM bridged network blocks VM to VM traffic
FedoraForum.org - Fedora Support Forums and Community
Results 1 to 8 of 8
  1. #1
    Join Date
    Nov 2005
    Location
    flying a cubicle
    Posts
    465

    KVM bridged network blocks VM to VM traffic

    Hello,
    I'm using KVM on Fedora 12 x86_64 to run several virtual machines. I've set up a bridged network interface, as I need these virtual machines to co-mingle on the same network as the physical machines.

    The bridge is setup as per the scripts below. I'm using the default F12 sysctl.conf.

    The problem is this... although virtual machines CAN access other machines on the network, including the Internet, virtual machines CANNOT communicate with each other. No pings, no file sharing, etc.

    I don't know why these VM's are unable to communicate with each other. I would appreciate any ideas or suggestions!
    Thanks!!


    br0 interface:
    Code:
    # cat /etc/sysconfig/network-scripts/ifcfg-br0 
    DEVICE=br0
    TYPE=Bridge
    BOOTPROTO=dhcp
    ONBOOT=yes
    DELAY=0
    eth0 interface
    Code:
    # cat /etc/sysconfig/network-scripts/ifcfg-eth0 
    # Intel Corporation 82573L Gigabit Ethernet Controller
    DEVICE=eth0
    HWADDR=00:19:D1:31:E9:E3
    ONBOOT=yes
    TYPE=Ethernet
    #BOOTPROTO=dhcp
    NAME="System eth0"
    UUID=5fb06bd0-0bb0-7ffb-45f1-d6edd65f3e03
    PEERDNS=yes
    PEERROUTES=yes
    BRIDGE=br0
    MTU=9000
    sysctl.conf
    Code:
    # cat /etc/sysctl.conf 
    # Kernel sysctl configuration file for Red Hat Linux
    #
    # For binary values, 0 is disabled, 1 is enabled.  See sysctl(8) and
    # sysctl.conf(5) for more details.
    
    # Controls IP packet forwarding
    net.ipv4.ip_forward = 0
    
    # Controls source route verification
    net.ipv4.conf.default.rp_filter = 1
    
    # Do not accept source routing
    net.ipv4.conf.default.accept_source_route = 0
    
    # Controls the System Request debugging functionality of the kernel
    kernel.sysrq = 0
    
    # Controls whether core dumps will append the PID to the core filename.
    # Useful for debugging multi-threaded applications.
    kernel.core_uses_pid = 1
    
    # Disable netfilter on bridges.
    net.bridge.bridge-nf-call-ip6tables = 0
    net.bridge.bridge-nf-call-iptables = 0
    net.bridge.bridge-nf-call-arptables = 0
    
    brctl show
    Code:
    # brctl show
    bridge name	bridge id		STP enabled	interfaces
    br0		8000.0019d131e9e3	no		eth0
    							vnet0
    							vnet1
    virbr0		8000.000000000000	yes


    ---------- Post added at 10:53 AM CST ---------- Previous post was at 10:11 AM CST ----------

    One more piece of info here... the bridge does see the virtual machines's MAC addresses:

    Code:
    # brctl showmacs br0
    port no	mac addr		is local?	ageing timer
      1	00:0f:b5:23:73:05	no		 194.94
      1	00:15:af:75:9d:bb	no		 209.88
      1	00:19:d1:31:e9:e3	yes		   0.00
      1	00:22:6b:5f:1f:3d	no		   0.01
      1	00:23:32:c9:e9:f8	no		 159.23
      1	00:30:18:a9:b2:76	no		   0.00
      2	32:d2:69:48:a6:64	yes		   0.00
      2	52:54:00:07:ca:6b	no		 159.30
      3	52:54:00:6f:7a:12	no		 208.36
      3	f6:ac:62:68:f5:8b	yes		   0.00
    The bold entries correspond to the virtual machine's addresses.

    ---------- Post added at 11:40 AM CST ---------- Previous post was at 10:53 AM CST ----------

    Although this certainly may be a case of user error, I'm not positive that this is not a bug. Therefore: https://bugzilla.redhat.com/show_bug.cgi?id=542405

    Of course, if anyone does have ideas, please do let me know!

  2. #2
    Join Date
    Feb 2006
    Posts
    780
    Maybe your firewall is in the way?

    On the host, something like the following rule can be used:

    Code:
    iptables -I FORWARD -m physdev --physdev-is-bridged -j ACCEPT
    David

  3. #3
    Join Date
    Nov 2005
    Location
    flying a cubicle
    Posts
    465
    I appreciate your reply! I already had that rule in place... I'm inclined to think it's not the firewall, but I'd appreciate your input on this:

    Code:
    # iptables -L -v
    Chain INPUT (policy ACCEPT 8566K packets, 12G bytes)
     pkts bytes target     prot opt in     out     source               destination         
        0     0 ACCEPT     udp  --  virbr0 any     anywhere             anywhere            udp dpt:domain 
        0     0 ACCEPT     tcp  --  virbr0 any     anywhere             anywhere            tcp dpt:domain 
        0     0 ACCEPT     udp  --  virbr0 any     anywhere             anywhere            udp dpt:bootps 
        0     0 ACCEPT     tcp  --  virbr0 any     anywhere             anywhere            tcp dpt:bootps 
    
    Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
     pkts bytes target     prot opt in     out     source               destination         
        0     0 ACCEPT     all  --  any    any     anywhere             anywhere            PHYSDEV match --physdev-is-bridged 
        0     0 ACCEPT     all  --  any    virbr0  anywhere             192.168.122.0/24    state RELATED,ESTABLISHED 
        0     0 ACCEPT     all  --  virbr0 any     192.168.122.0/24     anywhere            
        0     0 ACCEPT     all  --  virbr0 virbr0  anywhere             anywhere            
        0     0 REJECT     all  --  any    virbr0  anywhere             anywhere            reject-with icmp-port-unreachable 
        0     0 REJECT     all  --  virbr0 any     anywhere             anywhere            reject-with icmp-port-unreachable 
    
    Chain OUTPUT (policy ACCEPT 4634K packets, 417M bytes)
     pkts bytes target     prot opt in     out     source               destination

    I believe iptables is disabled by default for bridges in F12, anyway. My reference is the green box in the middle of this page:
    http://docs.fedoraproject.org/virtua..._with_PXE.html

    I have those entries in my sysctl.conf... of course, any other ideas would be appreciated! At the moment, I can't think of anything else to check.

  4. #4
    Join Date
    Feb 2006
    Posts
    780
    Maybe this has to be changed:

    Code:
    # Controls IP packet forwarding
    net.ipv4.ip_forward = 0
    David

  5. #5
    Join Date
    Nov 2005
    Location
    flying a cubicle
    Posts
    465
    Thanks - I tried setting this to '1', and I also tried setting this to '1' and then restarting libvirtd, but the problem unfortunately remains.

  6. #6
    Join Date
    Nov 2008
    Location
    Canada
    Posts
    2,723
    Sorry, I'm away from my box so I can't confirm anything.

    I believe the virtual machines are networked together through the vmnet interface but clearly is removed infavor of virbr#. So with virbr# bridged and if you suspect the bridge is the problem, why not unbridge them and try mapping (through iptables) to an alias. So with virbr0 mapped to eth0:1 and virbr1 mapped to eth0:2, can you then ping eth0:2 & eth0:1 from either virtual machine and the host?

  7. #7
    Join Date
    Feb 2006
    Posts
    780
    Quote Originally Posted by PilotJLR
    Thanks - I tried setting this to '1', and I also tried setting this to '1' and then restarting libvirtd, but the problem unfortunately remains.
    Did you just edit the file or did you edit the file and then restart networking or the machine?

    Maybe you should do a traceroute from one guest to the other? Possibly while you're running 'tcpdump -i br0' on the host.

    David

  8. #8
    Join Date
    Nov 2005
    Location
    flying a cubicle
    Posts
    465
    Quote Originally Posted by David Becker
    Did you just edit the file or did you edit the file and then restart networking or the machine?

    Maybe you should do a traceroute from one guest to the other? Possibly while you're running 'tcpdump -i br0' on the host.

    David
    I did an "echo 1 > /proc/sys/net/ipv4/ip_forward" to turn on IP forwarding.

    I will try doing the traceroute and tcpdump. I might not be able to do this until late afternoon / evening (central US), but I will report back!
    Beaker - likewise, I will give that a shot and report back.

    ---------- Post added at 01:19 PM CST ---------- Previous post was at 11:27 AM CST ----------

    Some more info...

    I ran both a ping and a tracert (the guests are windows 2003 with no firewall) between 2 guests while running tcpdump on the host.
    In each case, tcpdump DID list an ARP request from the guest initiating the ping. In each case, the ping target did not respond to the ARP (and did not send anything at all back).

    Despite this, "brctl showmacs br0" does correctly list the MAC address of each guest.

    ---------- Post added at 02:16 PM CST ---------- Previous post was at 01:19 PM CST ----------

    AH HA! It's not fixed, but it's getting closer.

    If I manually add static ARP entries into EACH virtual machine, then they are able to communicate. I do not think this is due to blocking the ARP broadcast, because only 1 static entry (the ping source) was not successful. Once the ping ingresses to the destination, it's address should be learned... but it was not (or it was incorrect).

    Which leads me to this question... why does the bridge show multiple MAC's per port?
    Code:
    # brctl showmacs br0
    port no	mac addr		is local?	ageing timer
      1	00:15:af:75:9d:bb	no		  91.15
      1	00:19:d1:31:e9:e3	yes		   0.00
      1	00:22:6b:5f:1f:3d	no		   0.00
      1	00:23:32:c9:e9:f8	no		  63.07
      1	00:30:18:a9:b2:76	no		   0.39
      3	52:54:00:07:ca:6b	no		  62.55
      2	52:54:00:4c:c4:e0	no		  62.55
      3	ee:f2:a5:a1:58:33	yes		   0.00
      2	f6:32:32:3e:47:d8	yes		   0.00
    The entries with "local" set to "no" are the correct ones, as shown inside the guest and in virt-manager.

    The local "yes" entries are for vmnet0 and vmnet1... I'm not sure what these are!

    ---------- Post added at 03:26 PM CST ---------- Previous post was at 02:16 PM CST ----------

    Problem Solved!

    Thanks for the input, guys! Here is what it was.

    Since I was using Windows Server 2003 guests, I chose to use a Intel e1000 virtual NIC, since that driver is included in 2003 by default and works well (oh physical hardware, at least).

    I solved these problems by removing the e1000 NIC and adding a Realtek 8139 NIC using virt-manager. I created a CD ISO with the drivers from realtek's website and gave it to the guest. Once the 2003 guest had the driver, the network worked and inter-guest communication worked.

    The tip off was when I kept watching "arp -a" inside a Windows Server guest. I noticed that pings received from the other guest (WITH a static arp entry) all came in as "00-00-00-00-00 invalid". After this, I installed Wireshark in the guest sending the pings and watched. Wireshark listed each source MAC as being from a Realtek NIC. I assume that the e1000 NIC was given an automatically generated Realtek address. I don't know if this is supposed to be kosher or not, but it certainly did not work. The problem was either the Realtek MAC + e1000 OR the problem was in the e1000 emulation itself.

    I still think this is a bug, and I'll be updating the bugzilla entry with this info.
    Last edited by PilotJLR; 30th November 2009 at 10:32 PM.

Similar Threads

  1. Iptables blocks outgoing traffic
    By x_terminat_or_3 in forum Security and Privacy
    Replies: 8
    Last Post: 24th January 2010, 06:07 PM
  2. XEN and routed networks vs bridged network
    By chas_martel in forum Servers & Networking
    Replies: 0
    Last Post: 7th January 2008, 03:56 AM
  3. Using VMWare on Fedora, Network help [Bridged]
    By slaquer in forum Servers & Networking
    Replies: 2
    Last Post: 13th May 2006, 12:32 AM

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •