two pysical servers - two xen hypervisor - several domU communicating

English Virtualization
#1

dear forum,
I would like to architect a bunch of SLES11 / Windows domU on two physical opensuse13.1 gnome servers interconnected with wifi cards or cross-over cable on ethernet cards, xen is 4.3
several doubts:

  1. do i need cross-over cable or regular ethernet cable would work ( no router ) ?
  2. do i need two connections on hypervisor ( xen with virt-manager ) with 2 virtual bridges, but how do I set up them in order to have all vm domU communicating ?
  3. is it easier to configure networks with ifup or with network manager ?

despite good experience with single machine setup, i feel lost especially with 2…

for one server,
eth0 was not used, virbr0 was set up: 192.168.122.0 ( range 192.168.122.1-255) and NAT
where
host xx.bla.com 192.168.122.1
gateway equal
static or dhcp domU yy.bla.com 192.168.122.2
dhcp allows to connect to internet via firefox, static wouldn’t not allow

Any help is welcome !

#2

what exactly are you trying to build out?

why 2 servers?
what are they used for?
what is the network topology?
why are you using opensuse13.1+xen as main host?
why not XenServer build? I mean I understand that VMWare ESXi have it limitations for free setup, but XenServer is free and very robust platform, and unless you plan to run something else on the main host hardware/OS XenServer is a good choice.

I might not be a good help here as I am somewhat noob my self who tries to build out a single VM server on OpenSuse 13.1 + Xen base but very interesting in your progress…
I have a reason why my choice is OpenSuse13.1+Xen over the XenServer, and that reason is
who I want to manage my hard drives storage. what’s yours?

#3

Your option with XenServer is totally valid, but because of xenserver will run 4.1 xen , I won’t be able to set up opensuse 13.1 with 4.3 and will have to downgrade ( and it is eventually tricky, still worth trying https://en.opensuse.org/SDB:How_to_install_openSUSE_13.1_as_PV_guest_on_XenServer ). Such a configuration would make me use XenCenter within Windows PV or OpenStack Havana within opensuse 13.1…
I want to set up 2 physical Servers to cumulate ( cluster) their memory and disks. This would reach 48G RAM and 2 Tb disks. In the meantime, I would simply use Xen an hypervisor, but as described in previous link, above PVM and HVM, xen will give us containers as a new (?) virtualization method with less i/o to dom0. So I guess, I stick with OpenSuse 13.1 for the good reason I am not too well yet with network config, filesystems and xenserver comands ( must be similar to xl and libvirt ), and have a mix of Yast and other peace of software ( R ) running in Gnome, and other opensource goodies. I 'll keep you posted on the progress.

Cheers

#4

Here are some basic principles…

You can build on two physical servers.
However you network the two physical machines does not matter (cross-over(many machines have auto-crossover), hub/switch/other, whatever). So, before you even start to configure Guests, make sure your physical machines can communicate with each other.
When you configure your virtual networking, you only need to configure the same virtual networking networkid on both physical machines, and they will communicate with each other.

Of course, do not configure or insert a router between your two physical machines (virtual, physical or otherwise) or you will then need to configure appropriate routing tables (assuming Guests on both machines will want to communicate with each other).

You configure a Linux Bridge Device (ie br0 or virbr0) for each virtual network you want to support. There is no limit to the number you might want to configure depending on what you want to do but you should configure at least one.

For network management in the Guests, you can use ifup or network manager, it doesn’t make any difference. Use whatever you’re more comfortable with. Personally I use network manager mainly because I’m used to it, but for anyone YMMV.

You describe how your first machine is set up.
Build your second physical machine identical to the first (of course it must have unique machine name but same workgroup name, and a unique IP address). The virbr0 should be the same networked (the same masked network octets) but a unique hosted (octet number to the right of the mask).

Then everything should “just work.”

TSU

#5

thx Tsu, clear up to the point i’ll make it work, will keep you posted,

found this about bridging ( option 2 shared physical device )

http://wiki.libvirt.org/page/Networking#Forwarding_Incoming_Connections

#6

Finally, I got eventually my homework .done…As decribed by TSU, we are in a Linux bridge scenario

Scenario :

  • Two physical hosts running openSuse 13.1 and
  • two hypervisors Xen or KVM with
  • all vm’s and hosts communicating (inbound-outbound)
  • on the same local (isolated from outside) network 192.168.0.0 restricted to 255.255.255.192 /26
  • One Dlink router ( apparently without LAN bridging capabilities)
  • static dhcp definition / or dynamic
  • Using NetworkManager or Not, but NetWork Manager does a better job to switch easily to your wifi network (after breaking off local ethernet network)
  • No Firewall or Wifi bridging

Reading : on your opensuse 13.1 server /usr/share/doc/packages/sysconfig
README.virtualization
Network and netconfig.png
http://www.freedesktop.org/wiki/Software/systemd/PredictableNetworkInterfaceNames/

3 Steps :

  1. Create Physical connections

  2. Create Linux Bridge

  3. Create Virtual Network using Linux Bridge

  4. Plug your router and make sure, it is set to factory standard, for example with a local
    ip 192.168.0.1, theoretically comes with DHCP on, meaning that any host connecting via NIC card will get assigned
    automatically a sequenced/randow ip address defined in the net defined on the router

In FireFox http://192.168.0.1 > username password > Network Settings : IP and netmask 192.168.0.0 restricted to 255.255.255.192 /26
Range 192.168.0.2 - 192.168.0.62

Static is easier in such a small network for this exercise…

  1. on host A and B : Yast > Goto Services Manager > Disable Network Manager

Reason: Network Manager will end up override your settings but when you are confident enough
you can do the same with NM

  1. on host A and B Yast > Goto Network Settings > Delete All Interfaces >
    they appear now as Not Configured and with command line look @

ls /etc/sysconfig/network

> notice that no ifcfg-br0, ifcfg-virbr0, ifcfg-eth0 or ifcfg-enp5s0 files exist
> if some do exist
mkdir -p /etc/sysconfig/network/bak
mv ifcfg-br bak/*

those configuration files are created when you use Yast > NetWork Settings
by the traditional way with ifup and not NetWork manager

ifconfig

shows only lo interface

2.1. on host A and B :Create the linux bridge with Yast > NetWork Settings > Add > Device Type Bridge > Configuration Name ( will concatenate the number choosen in the dropdown menu to predefined conventiomnal string “br” , ie. br0 in case of a first bridge created).
Note: You are not tight to this convention anymore with NetWork Manager

Fill in the ip address of the router ( when you had the chance to write down its ip)
and fill 192.168.0.1 255.255.255.0 or /24 myhost.mysite.com
Then
Goto Bridged Device Tab > Select enp5s0 or other ethernet NIC device and finish

ifconfig

> Doesn’t show the bridge and interface ?

service network stop

service network start

or

Go back to Network Settings > Edit Bridge Br0 > Global View > Change the Option StartUp On Boot to HotPlug

or

ifconfig enp5s0 down

ifconfig br0 down

cd /etc/sysconfig/network

cat cfcfg-br0

and you will notice that BRIDGE_PORTS is blank,

vim /etc/sysconfig/network/ifcfg-br0

edit according to’enp5s0’ or your ehernet NIC if, save and exit
BRIDGE_PORTS=’’ >> BRIDGE_PORTS=‘enp5s0’

ifconfig enp5s0 up

ifconfig br0 up

brctl show

bridge name bridge id STP enabled interfaces
br0 8000.14dae9413349 no enp5s0

reboot now

reboot in Xen Domain0

2.2
on host A ( and B : br0 with different HwAddr linked differently to enp5s0f2)

ifconfig

shows properly
br0 Link encap:Ethernet HWaddr 14:DA:E9:41:33:49
inet addr:192.168.0.1 Bcast:192.168.0.255 Mask:255.255.255.0
inet6 addr: fe80::16da:e9ff:fe41:3349/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:95 errors:0 dropped:0 overruns:0 frame:0
TX packets:97 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:5448 (5.3 Kb) TX bytes:13255 (12.9 Kb)

enp5s0 Link encap:Ethernet HWaddr 14:DA:E9:41:33:49
inet6 addr: fe80::16da:e9ff:fe41:3349/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:142 errors:0 dropped:0 overruns:0 frame:0
TX packets:131 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:44052 (43.0 Kb) TX bytes:18555 (18.1 Kb)

  1. On host A:

3.1 Create the Virtual Network with virsh and xml

cd /usr/share/libvirt/networks

cat > enp5s0_br0.xml <<EOF

<network>
<name>enp5s0_br0</name>
<forward mode=“bridge”/>
<bridge name=“br0”/>
</network>
EOF

On Host B:

cd /usr/share/libvirt/networks

cat > enp5s0f2_br0.xml <<EOF

<network>
<name>enp5s0f2_br0</name>
<forward mode=“bridge”/>
<bridge name=“br0”/>
</network>
EOF

virsh net-list

virsh net-destroy default

in case virsh net-list shows other network running

host A

vish net-create enp5s0_br0.xml

host B

vish net-create enp5s0f2_br0.xml

virsh net-list

net-list                       list networks

net-autostart                  autostart a network

3.2. Change the vm definition files in /etc/xen/vm
native and xml file definition for the ds virtual machine

name=“ds”
description=“ds”
uuid=“c23e07d0-5e6a-76cc-26da-6c6bae394a4c”
memory=2465
maxmem=2465
…]
>> vif= ‘mac=00:16:3e:23:a4:66,script=vif-bridge,bridge=br0’, ]
vfb=‘type=vnc,vncunused=1’]

3.3. Start the Virtual Machines on both hosts

virt-manager

3.4 Adjust Yast>Hostnames with ip that ifconfig gives you in local vm’s (static or dynamic)

Simple but requires to practice because wrong sequence with beginners make systems unbehaved…
I used cross-over cables (I didn’t have regular available) to connect router to hosts and it didn’t cause any issue:.

Enjoy.

#7

KVM bridging is a bit more tricky …:(.and for the price to pay for complexity, why not adding openvswitch in the picture. But probably some of you noticed that for few concurrent hosts on same host. Kvm is very efficient…still not as userfriendly as Virtual Box, but straight to boot and install (not along with xen ).

https://www.suse.com/documentation/sles11/singlehtml/book_kvm/book_kvm.html#cha.qemu.running.networking.bridge

next steps: 1. reformulate libvirt ( virt-manager) with tap / linux bridge device for Kvm
2.adding OpenSwitch layer
3. Lxc ?