Setting up bridged network with XEN Dom0 and DomU (VM) on different network subnets

English Virtualization
1

Not a very experienced XEN user here.

I wanted to set up an Opensuse 13.1 XEN server with a SLES_11 VM running my mailserver. My idea was to have the VM running a static IP and the Dom0 machine picking up a dynamic IP from my internet provider.

I set up the system initially using YAST with both the Dom0 and DomU running a static IP address on the same netmask 255.255.255.0 subnet. Everything worked fine.

I then changed the Dom0 system so the network card and XEN bridge both got their IP address from DHCP. Because of my internet providers network configuration, that meant the dynamic IP address was completely different from the static IP address.

Specifically:

  • DHCP assigned IP for Dom0 - 97.79.x.y
  • Static IP assigned to VM - 71.40.a.b

Result:

  • Dom0 can “see” the internet fine
  • Dom0 can not ping the VM (ping command just hangs with no error message)
  • VM - can’t ping Dom0 or outside world. (ping command gives destination host unreachable)

It looks to me like network packets aren’t crossing the xen bridge and possibly the kernel routing table(s) are messed up. I tried several variations of route options without success on both the VM and Dom0.

Questions:
-Is it even possible to run XEN bridge this way?
-Are the ethernet packets “crossing” the XEN bridge affected by the Dom0 kernel routing parameters?

  • Any suggestions to fix?
2

Although I haven’t tried to setup exactly what you describe, I’ll answer on general principles…

  • Whether talking about physical networks or virtual networks, the rules to “see” each other remain the same in that both/all addresses need to have “proper” NetworkIDs as defined by the subnet masks. Note I tried to use “proper” instead of “same” because they <can> be different (review the use of subnet masks to understand what I’m referring to).

  • Most descriptions on how to set up virtual networks describe using Linux Bridge Devices (typically named for example br0) which you don’t describe. If you’re not following a guide to setting up virtual networks that describe this, do so.

  • Ordinarily, virtual networks are bound to an adapter, not the interface or IP address. So, just because your Host adapter is configured for DHCP or static address ordinarily should not make any difference if your virtual network is setup correctly.

  • Although not specifically related to your questions, remember also that it’s possible to configure multiple IP addresses on any interface. If you don’t understand subnetting, a second IP address on an interface may be a simple way to enable network connectivity.

It’s too bad that Xen doesn’t support libvirt today, I’d recommend you investigate other VMM like Vagrant to setup and configure your virtualization if Xen supports (I only dabble in Xen personally, prefer KVM when my host is openSUSE) for a small network, if you’re building a large deployment, you proabably should be looking at Puppet or Chef.

HTH,
TSU

3

question here: how many NICs do you have?

what you describing does not seam to be be Xen issue but your network issue.

if you only have one NIC and want/need to be a dynamic you might need to setup several bridges and a routing between them since if I am right it’s not only the IP gets assigned but network mask as well from your ISP.

so you might end up with

WAN(ISP) IP 97.96.x.y subnet mask 255.245.x.y
but your VM have IP 74.54.x.y subnet mask 255.255.x.y and that is different subnet hence you can not ping one from the other. you need NAT/route between them.

you might need to look into openswitch or something.

PS>> tsu2, why do you say that Xen does not support libvirt? aren’t we use libvirt in opensuse/yast to manage/create the VMs?

4

You’re right.
Muddled mind confused 9p support with libvirt support.
Of course, Xen supports libvirt.

So, to the OP
If you don’t have libvirt installed (with its GUI VM Manager, virt-install and vm-install), I highly recommend you do so. When you setup your virtual networks with VM Manager, it’ll automatically create Linux Bridge Devices for your virtual networks and optionally configure some common features(eg DHCP if you choose NAT).

TSU

5

well you know what they say :slight_smile: S#$T happens.

but OP might still have an issue if he/she only have single NIC.
my guess there is a need for a virtual network here and NAT of some sort… so the host is on the WAN network and all VMs are on the internal virtual network routed via host NAT server or something.

6

Thanks for the comment/ideas!

I have only one NIC card and when I set up the system via yast, the br0 network bridge was created.

The suggestions regarding subnet might be the issue. Within Yast/Network, I can see both the NIC card and br0 are set to pick up the IP address via DHCP. At the same time the subnet information is set. In this case, the IP is picked up as IP 97.96.x.y and the subnet mask is 255.255.254.0 (as determined via an ifconfig command).

The same ifconfig command shows no IP or mask information for br0 just a HWaddr value. As a test, I used Yast/Network to change br0 to use a static IP address (71.40.a.b) while the NIC card continued to use DHCP. Subsequently, the ifconfig command showed different IP addresses and subnets for each device - eth0 and br0.

So - does this mean that when the NIC card and br0 devices have the same IP and subnet (as I suppose happen automatically by DHCP), any ethernet packets on a different subnet are screened from the VM?

If the NIC card and br0 have DIFFERENT IP addresses (and subnets), are packets on either subnet visible to the VM or only the br0 subnet?

Is there a way to set this up so packets from ANY desired subnet are visible to the VM through the bridge? I tried using a 0.0.0.0 entry for the br0 subnet and get a yast error message (invalid value)…

I will do some research into libvirt.

7

Various documentation explains…

(From memory, you’ll likely find the actual info quoted often)
When a Linux Bridge Device is setup, then that virtual device is configured instead of the actual physical device (although personally I suspect that the physical device might also be configurable if there are no conflicts).

In general, I personally don’t recommend setting up using YAST although all official documentation describes doing it that way.

Instead, if you install libvirt then when you setup your virtual networks a different set of Linux Bridge Devices will be setup (don’t worry about pre-existing devices created by YAST, they won’t conflict and will simply be unused). The advantage of libvirt is that you’ll have a management tool which will do much more (assign network IP range, optionally configure NAT and its own DHCP). If you’re bridging (not doing NAT) then you <can> do it with YAST but you’ll need to use the “normal” tools to troubleshoot (like brctl).

Note, do not confuse “Linux Bridge Devices” with a network bridging configuration, they are completely different things.

Now, specifically your issue…
Based on what you described and especially that your Guest which should be bound to br0 did seem to pick up an IP address automatically, then if it’s contacting the same DHCP server as other Hosts on your network then the NetworkIDs should be the same for all Hosts whether virtual or physical. If they’re different, then you need to find out where/what is handing out those rogue addresses.

Unfortunately I don’t know that there is an easy way to discover rogue DHCP. I know of some free tools from many years ago, but I don’t know any that are free today.

Of course, other possibilities are

  • An incorrectly setup br0, bound to something other than the proper physical adapter
  • Although highly unlikely (my guess), it’s possible to configure a virtual DHCP but that is almost impossible to accidentally do unless it pre-exists.
  • An incorrectly setup Guest configured for who knows what.

All that guesswork is again why I recommend installing and managing with libvirt, you would/should have a much clearer picture what is likely happening.

TSU

8

Thanks All,

I finally solved the problem using the clues you provided.

It turns out the YAST configuration utility (and my ignorance) combined to mess up the interface configuration files for the NIC card and the bridge. After manually fixing the config files, the Xen Dom0 is running on a DHCP IP address and the DomU is running on a static IP (on a different subnet) with no issues.

On the Dom0 - when the bridge is properly configured - the ifconfig command will show the DCHP-assigned IP address attached to the br0 bridge, while the NIC card will show no IP address. Manually, once the bridge was created by the Xen install, I could set the network parameters on the bridge by editing /etc/sysconfig/ifcfg-br0. Make sure the NIC card is listed in the PORT= line (this was missing in my config file). In the same folder, the ifcfg- file for the NIC card should NOT assign an IP address. In my case, I deleted the file (ifcfg-enp2s0) and everything worked fine.

I also found out it was important to have the SuSE-firewall2 running. The firewall includes iptables routing instructions that send all incoming packets into the NIC card onto the bridge. Apparently, whether the NIC packets are sent through the iptables/netfilter system on the Dom0 is controlled by some kernel parameters like:
net.bridge.bridge-nf-call-ip6tables
net.bridge.bridge-nf-call-iptables
net.bridge.bridge-nf-call-arptables

Depending on these values, the Dom0 kernel sends NIC packets directly onto the bridge before or after going through iptables. These parameters can theoretically be changed via sysctl and /etc/sysctl.conf, but try as I might I could not get the kernel to reboot with the correct values. There is some chatter on the internet that this might be bug related. In any case, in OpenSuse 13.1, the default values direct packets through iptables. With the firewall on, there appropriate iptables information was there to direct packets onto the bridge. However, if I turned the firewall off, the Dom0 could not longer see the internet.

Once this is done (and the network restarted), my DomU was set up with a static IP using YAST in the usual way. Everything seemed to work.

So in a nutshell, it seems that my YAST configuration had somehow led to the NIC card being stripped from the PORT= entry in the /etc/sysconfig/ifcfg-br0 file in addition to separate IP addresses being assigned to both the bridge and the NIC card. After that, there was no hope…

Useful commands to debug:

ifconfig - to see the interface settings
brctl show - to see which interfaces are attached to the bridge
iptables --list - to see how dom0 kernel is handling interface packets
sysctl -p - to see the settings on kernel parameters
route -n - to see routing information by IP address/network

Useful references:

https://www.suse.com/communities/conversations/basic-iptables-tutorial/
http://www.thegeekstuff.com/2011/01/iptables-fundamentals/
http://www.thegeekstuff.com/2011/06/iptables-rules-examples/
http://www.linuxfoundation.org/collaborate/workgroups/networking/bridge

9

nice to know, BUT I am not convinced this is a very good/Safe setup.
what you basically doing is exposing your whole network to the internet.
not sure if it wise.

if I were you I would get a dual port card (Intel usually considered best) and use that for WAN to LAN bridging using the built in NIC for internal server LAN and management. but to each is their own.

10

Congrats on working out the solution. I’m sure you learned and now understand the Linux Bridge device better than most people.

FYI -
AFAIK the most common causes of the mystery you describe (the NIC card not identified) are

  • NIC cards are now identified by a name that identifies the manufacturer of the adapter and not generic names like eth0
  • Although not usually an issue on openSUSE but common on all other distros, the MAC address and adapter are not configured properly in the udev network adapter file if the Guest has been cloned.

Your comment about IP tables is correct. You need to verify the <bridge device> and not the physical adapter is configured in the appropriate zone.

TSU