Virtual Networking -some Basics - All OS

English Virtualization
1

I am posting this as a completely new thread although it’s written as a continuation of a couple recent other threads titled “Bridging is broken on virt-manager” and “Beginning with KVM” because this topic about virtualization networking is an independent topic and although this is written mainly for libvirt KVM/Xen/LXC Users, is almost completely applicable to Users of VBox, VMware, Hyper-V and many other virtualization technologies as well.

As I mentioned in one of my links to my earlier criticisms of the current community openSUSE virtualization documentation, this is one of the areas which IMO was lacking. And, most current SLES documentation which describes virtual networking is rather brief and generally not complete… something that might be helpful for someone with prior experience with other virtualization but not likely enough for someone brand new. The following is all off the top of my head and is not double-checked against other sources for accuracy, so as usual “Use at your own Risk” and maybe people will submit succeeding posts that will correct, clarify or ask for more about what follows.

Let me first fill in some gaps in the** KVM reference**(https://www.linux-kvm.org/page/Networking) since it’s the only one mentioned in a post in this thread up to this point.
The reference describes several types of networking, but doesn’t do so in simple, common terms that people might recognize.

First is SLIRP.
Although listed first, it’s not often used.
This is the primary means of configuring networking if you were running User Mode Linux (UML). For those who want to explore basics, I encourage looking into this. It’s very simple to set up and execute, but I probably wouldn’t run it for more than educational purposes… You don’t need to install anything new, but you may want to download a kernel you want to use for your “machine.”
http://user-mode-linux.sourceforge.net/

The Private Virtual Bridge and Public Virtual Bridge
These are used in probably over 99% of all virtualization technologies granting Guests networking functionality. The “Private Virtual Bridge” is probably more easily recognized as “Host Only” networking in VBox, VMware and Hyper-V. The “Public Virtual Bridge” can be configured a number of ways, as bridging(The traditional definition where the supported ip network is the same as the existing interface) and NAT (possible variations included). Note the warning about the Public Virtual Bridge and wireless interfaces in the KVM documentation! That is where the MacVtap can be implemented instead.

VDE
Is a relatively new type of virtual networking.
But since today the above “virtual networking bridge” aka Linux Bridge Devices aks ?? (you might find this described many ways in different documentation) is almost always used, VDE should not be important unless you have a special need.

An additional useful general concept is that you can usually recognize and determine the origin of networking devices you’re displaying or connecting to by its name… Note that no matter how a virtual networking bridge device is name and by what it was created, it’s the <same thing>

Physical Devices - Names like eth0, wlan0, eppl0
Bridging Devices created by YaST, brctl, other standard tools - Names like br0, br2
Bridging Devices created by libvirt - Names like virbr0, virbr2
Bridging Devices created by VMware - Names like VMnet0, VMnet2
Bridging Devices created by VBox - Unknown, on my personal machine VBox looks like it’s running virtual networking bridge devices supporting IPv4 networking on top of HostOS Ipv6 interfaces

OK…
Now that the above has been gotten out of the way, for those who are experienced virtualization Users with experience in other technologies should have a leg up on understanding what is typically configured and might be able to just open vm manager, create a virtual network and recognize familiar stuff without needing more explanation.

For the newbies and people who haven’t paid any attention to virtual networks and virtualization bridge devices, read on. The following might serve as a first draft for something with more thought later on…

The Network Bridge Device (aka virtualization bridge device, bridge device, etc)
Goes by many names, but is the way you set up networking in practically every known type of virtualization and isolation on top of every major OS including Windows and Unix. Notable exceptions are UML and Docker although I suspect that those can also be configured this way if you dig beyond the standard documentation.

Although to the new User this might seem like an extra complication, it’s actually not because it solves a number of problems and gives you powerful options compared to accessing the network interface directly (which you would do if you configured SLIRP or MacVtap).

And, it should be recognized that libvirt like practically everywhere else will automatically set up and manage your Guests configured to use virtual networks that implement the described virtual networking bridge devices.

Network bridge devices should not be confused with configuring a network interface in bridging mode…

  • A Network bridge device is a networking component that stands on its own and is bound to at least one networking interface.
    . A network bridging mode configuration is not a separate component, it’s a standard configuration of a network interface (which can be physical or virtual) which results in the device or machine becomes invisible on the network. So, for instance a multi-homed machine’s network interfaces connecting to two differnt physical networks in which the NetworkID is the same, so the machine is acting as a “network bridge” instead of a router which would require the networks to have different NetworkIDs. In the case of virtual bridge devices, this would mean that the virtual machine is seen and communicates with other remote machines like any ordinary physical or non-physical machine on the network.

Common ways a network bridge device is configured
As a completely separate and independent component, it can optionally define an entire virtual network internally, and possibly provide network services like DHCP to that virtual network without having to install and configure a full blown DHCP server.
Typical and common configurations include
Bridging - Seen and communicates on the regular physical network with physical and virtual machines both on the same HostOS and remotes on other physical machines.
Host-Only- A virtual network in which only virtual machines on the same physical machine can communicate with each other.
NAT - A private network. Essentially similar to a Host-Only network but with a defined DG to access the physical network. Optionally, DHCP services can be provided to the virtual network.

When a network bridge device is bound to the network interface, depending on how it’s created and configured, the HostOS may lose access to the physical interface. The simple solution in this case is to re-configure the Host to use a bridging device configured for network bridging functionality (The typical default configuration for br0).

Creating a virtual network bridging device
Libvirt
This is the best way to do this, because unlike others you are walked through a graphical “wizard” of configuration steps to complete your task. There are very few ways you can make a mistake doing this and there is almost no way to create a conflicting device.

YaST
Just create a new device, specifying a “bridge” type.
If you’re trying to replicate the bridge device normally set up when using YaST to install virtualization, configure it as a DHCP client.
Do not remove the physical interface as recommended in a prior post in this thread…

brctl
Sorry, I don’t generally do this… Maybe if I was on some other distro than openSUSE, but one of the nice things about using openSUSE is that we can ignore a lot of things that are required. I do run brctl often to display bridge device information though and recommend Users become familiar with this use.

Configuring Networking for a new Install
When running virt-install which is part of libvirt, on the last screen before the install configuration ends, you will be presented with a dropdown populated with choices for any virtual networks that are available(ie configured virtual networking bridge devices), MacVtap or custom other…

If you configured your virtual networking first, you’ll have more options so I highly recommend reviewing your existing virtual networks before you create your first VM, and if necessary create any new networks you may need.

But, there is no requirement for creating virtual networks first.
It’s easy to switch virtual networks later and can even be done while the Guest is running.

HTH and as stated in the beginning…
Corrections and suggested rewording, and questions about what is described in this post is welcomed and requested.

TSU

2

A loose end not mentioned in the original document which might clarify for some…

A major reason for defining a virtual network is to enable and support multiple virtualized Guests with their own IP addresses communicating with each other.

I don’t know if something similar can be configured easily on a physical interface without a virtual networking bridge device, but even if it might be possible would have to be plenty messy with all sorts of network addresses and networks defined on that one interface. Enabling all your Guests communicating through a common virtual networking interface is much neater, where all your virtual networking configuration and functionality is separated from your physical networking.

TSU

3

I think this is my case. Just like you suggested in one of my previous threads, I started by installing KVM server by using Yast. When Yast finished downloading and installing the packages it asked if I wanted to setup a bridging device, for which I answered “yes”. Out of my 2 physical devices, all the entire time eth0 has been on and being used -obviously-, while wlan0 totally unused ever since openSUSE installation.

But, if Yast really configured the bridge device, and in this case you suggest not to disable eth0 -and clearly I have never done-, why is it that virt-installer’s last step, in network, cannot see the bridge device? It is not being listed…

4

Run the following command that displays all bridge devices that exist on your machine

brctl show

If it exists, then it’s something to investigate.
If br0 doesn’t exist, then no big deal.
Just create a new bridging device (I recommend creating a virbr0 device using vm-manager to create the device).

However you create your bridging device, it will then show up when you create a Guest.
If you already have a Guest created, then you can configure the Guest Properties to use the bridging device.

TSU

5

Is a virbr device only created from virt-manager, or can it be created directly from virt-installer as well (i.e., while creating the virtual machine)?

6

On Mon 21 Aug 2017 01:36:01 AM CDT, F style wrote:

tsu2;2835000 Wrote:
> YaST
> Just create a new device, specifying a “bridge” type.
> If you’re trying to replicate the bridge device normally set up when
> using YaST to install virtualization, configure it as a DHCP client.
> Do not remove the physical interface as recommended in a prior post in
> this thread…
I think this is my case. Just like you suggested in one of my previous
threads, I started by installing KVM server by using Yast. When Yast
finished downloading and installing the packages it asked if I wanted to
setup a bridging device, for which I answered “yes”. Out of my 2
physical devices, all the entire time eth0 has been on and being used
-obviously-, while wlan0 totally unused ever since openSUSE
installation.

But, if Yast really configured the bridge device, and in this case you
suggest not to disable eth0 -and clearly I have never done-, why is it
that virt-installer’s last step, in network, cannot see the bridge
device? It is not being listed…

Hi
So I guess the question that hasn’t been asked, is you are using wicked
and not NetworkManager?

–
Cheers Malcolm °¿° SUSE Knowledge Partner (Linux Counter #276890)
openSUSE Leap 42.2|GNOME 3.20.2|4.4.79-18.26-default
If you find this post helpful and are logged into the web interface,
please show your appreciation and click on the star below… Thanks!

7

Hi
During the VM install… it’s forwarded to the bridge (br0)

http://thumbs.imagebam.com/fc/07/3a/f14b80582353753.jpg](ImageBam)

8

No, this is an incorrect configuration.
To avoid messy setups, don’t configure a bridge device to forward to another bridge device without good reason, it should be "bound’ to a physical interface like eth0 or wlan0.

So, in this case if br0 is bound to eth0, then virbr0 should have been configured to bind directly to eth0.

TSU

9

Adding an important tip…

Generally bridging devices created by any other method than using libvirt (vm manager) are basic, and once created you really don’t know what is in the bridging device except by opening up its configuration file for inspection.

On the other hand, any bridging devices created by libvrit (vm manager) can always be inspected later in vm-manager,
So, is another good reason to use vm manager to create your bridging devices.

On this topic,
Answering a question posted elsewhere, it <does not matter> what created these bridging devices, they are and can be used by any virtualization once created. This is because once created, they are system networking objects and therefor become available to any application running on the system.

For example, in the past I’ve had KVM, VMware and Virtualbox installed on the same system.
I’ve used the same br0 with all of them.
Although VMware isn’t normally managed by libvirt, I’ve configured VMware guests to use vmnet bridge objects.
And, I’ve configured KVM guests to use VMware created vmnet objects.
(If anyone tries this kind of stuff, remember you should never run multiple virtualization apps simultaneously. Stop all before starting one).

TSU

10

Had to -shamefully- ask a pal to briefly borrow his openSUSE box, while I broght an old Linux liveCD, because doubts and stress arouse after re-reading all the related threads…

I sent Yast and Wicked totally to !"#$ off. I realized a bridge can be created just from Settings, the Network option (guess these options are network manager’s anyway…). But I preferred the pure virt-manager way, as tsu2 always advises.

I reviewed this article.

It says “libvirt ships with NAT virtual networking working out of the box, one can see the bridge already existent”. Except that on openSUSE this is not true. Installing KVM through Yast or zypper does not explicitly create “virbrx” devices, only the Yast’s d**n br0 and br1 if one accepts.

So one has to create the bridge (the “virtual network”?) from virt-manager. I realized I got the “Error: network is already in use by eth0” at the end of wizard if I specified the same network address as the one being used by eth0 itself. So if eth0 is in 192.168.5.0/24 I must set a different one, say 192.168.6.0/24. This way I no longer had problems with wizard, even leaving “nat forwarding” to “any physical device”.

This created 2 devices instead of one! Being them virbr0 and virbr0-nic. Output of brctl looks like

# brctl show
bridge name ----- bridge id ---------- STP enabled ------ interfaces
virbr0 -------- 8000.000000000000 ---- yes ------------- virbr0-nic

Nevertheless in guest’s settings I set its network source to the newly created “virtual network” (shouldn’t it be a bridge!?)
Decided to give it a try by using the Linux liveCD for the guest. Booted, started, and tried pinging. It was unexpected: guest had internet access and was able to ping openSUSE host itself and any PC in the physical LAN. As well, host and any PC in LAN was able to ping virtual guest. All even when guest in in different network!

BTW, when guest was up brctl looked like

# brctl show
bridge name ----- bridge id ---------- STP enabled ------ interfaces
virbr0 -------- 8000.000000000000 ---- yes ------------- virbr0-nic
-----------------------------------------------------------vnet0

Mr tsu2 mentioned to ask if after testing some more I still didn’t understand things.

WTH with these 2 virbr0 devices? Why still 2 anyways?

How is this “virtual NAT” thing or whatever actually working? Is it “routing” or the like? Only concept about NAT I have is the one “opening ports” in physical routers.

When configuring the bridge in virt-manager I left NAT forward to “any physical device”. Is it actually taking any working physical device that it detects, or what?

How do I disable STP for the bridge? I already read very well about it and I really don’t want it. I could not find any option within virt-manager to disable it…

11

Update:
This doc is better:
https://wiki.libvirt.org/page/VirtualNetworking
I must correct myself: I did not try pinging from other PCs in physical LAN to virtual guest, only from openSUSE host.

Still doc insist in the fact libvirt creates a “default” virtual network (in NAT mode) when installed and started. This was clearly not my case. Could it be because I accepted Yast’s advise of creating bridges back when installing KVM stuff?

Yet, still in need of help for the other doubts:

Why still 2 virbr0 devices?
When configuring the bridge in virt-manager I left forward to “any physical device”. Is it actually taking any working physical device that it detects, or what?
Is brctl really the only way to disable STP on bridges?

12

I haven’t deployed a brand new 42.3 KVM Host as of yet to know for sure if anything has changed (intentionally or not).
In any case, if all you notice awry is a missing NAT network, I’d consider that minor, and practically trivial to resolve although keep in mind that the NetworkID you set up may not be the same as what should have been configured by default. In your case, unless you re-built from scratch after your admitted mis-steps installing, your system shouldn’t be considered typical. Creating bridge devices during the YaST setup no matter how and what you did would have <no effect> whatsoever on libvirt created bridge devices. At least, I can’t see any possible conflict.

In what you posted, virbr0 is mapped to multiple interface objects.
In your specific case(not necessarily applicable to others) I don’t see that you should see any unusual functionality, and your multiple entries are related to how the STP (spanning tree protocol) works.
In general, you <never> want to disable implementing the STP, because it provides safeguards against various misconfigurations. You can read various articles about the STP, but as long as you don’t have a special need to inrwnrionLLY configure spanning multiple network interfaces and keep your bridge devices accordingly very simple(eg bind to physical interfaces and never or rarely other bridge devices), then you don’t have to learn very much about STP, just accept how it’s configured automatically.

TSU

13

Got a garbage PC only for a day, just to get rid of certain doubts no one here has ever been able to solve.

Clean-installed Leap 42.3 3 DED times*:

–Installed KVM again through Yast and accepting its nasty bridges. No mistake: it always creates 2 br’s. So anyway had to switch to Wicked, and immediately did the test of deleting br0 -the one with nothing- and leaving br1 -the one with dhcp and eth0 slaved-. Again used the Linux liveCD for a test virtual guest, for which I certainly used br1.
Result: worked pretty good just like Proxmox bridged style.
Conclusion: only one br interface is needed, not two! D**n Yast…

–Installed KVM yet again through Yast, but rejecting bridge offering.

–Installed KVM with “zypper in -t pattern kvm_server” and enabled libvirtd service.

Result of last 2 cases: no virtual networks at all created out of the box; I had to manually create one.
This piece of… text is false:
“A standard libvirt installation on openSUSE Leap already comes with a predefined virtual network providing DHCP server and network address translation (NAT) named “default”.”
Unfortunately totally false…

During last case I did some few last tests, first of all going back to Network manager.

A bridge can also be created through Settings > Network > wired network properties. Problem is virt-installer/virt-manager never recognizes it. Perhaps it’s true what was mentioned in a past thread: virt-manager is designed to only work with Wicked regarding bridges.

Seemingly virt-manager’s virtual networks are designed to never work in same network segment as host’s currently used interface; virtual guests get external access either through NAT or routed modes, but always in a different network segment.
In NAT mode virtual guests can see everything outside, but can only be seen by host itself and other guests under same virtual network. No PCs in physical LAN can see virtual guests due to iptables rules in virtual network.

Just curiosity, does anyone have idea of how one could make an “external” PC in LAN communicate with virtual guests? Would it be through iptable rule modifying?
What’s depicted here certainly looks more lika a “hack”…

14

Hi
Maybe it creates two bridges due to having one wireless and one etherent. Now if you look at the bridges created via YaST -> Network Settings, select bridges and edit, you will see they may not be associated to an interface. Again you can go in and set this up how you want (which is as it should be).

The system I used (running Tumbleweed at the moment, no vm’s) for vm’s has three (3) ethernet interfaces and bridged (br0, br1 and br2) and connected direct to the guest as a physical interface (static ip’s), the wireless interface is used for management so no issues with external pc’s in local network contacting guest vm’s.

You might want to try gnome boxes as AFAIK this interfaces direct with NetworkManager, but has limited options (but does use spice).

15

Since time immemorial,
br0 has always been created by YaST in a “bridging mode” configuration that exposes the physical LAN to the Guest(and vice versa).
This means for example that if you configure the Guest network settings as a DHCP client, it will contact the same DHCP Server physical machines in the LAN also use, and the Guest will be assigned an IP address with the same NetworkID as every other physical machine in the LAN. This also means that remote machines can find and communicate with the Guest, so the Guest can itself be configured to provide network services… For example, being configured as a Network File Server serving network shares.

If a br1 is created by YaST, that would be new and worth investigating. As of today, it’s not possible to use YaST to manually create a bridging device that supports NAT, you’d have to first create it, then modify its configuration likely with a text editor. Of course, something like this could be pre-configured for special installation when YaST installs virtualization, but I haven’t seen it prior to 42.3 (and I haven’t verified this happens in 42.3).

On the other hand, as the documentation says when you install libvirt (typically using YaST virtuallization install) you’re supposed to have a virbr (not br) named device that is set up as a NAT device.

If you install KVM without libvirt, then you should know that no current openSUSE or SUSE documentation will describe your setup or how to use and manage your virtualization. You can still use an alternative to libvirt(eg Vagrant) or use KVM or QEMU command lines to do everything you need to do, though I’d recommend the official KVM documentation.

I’ve never been bothered by missing bridge devices whenever it happened in previous versions of openSUSE, partly because I hardly ever used default bridging devices(virtual networks) anyway. I’ve always used only bridging devices I created either because of necessity or preference. I subnet and supernet my networks carefully which requires design and that isn’t supported by defaults. Also, I’m always suspicious of using defaults which can be a gross mistake… One primary example is the county library system where I live… It set up as one enormous Class A private network using the default subnet mask(Yes, it’s a very big mistake obvious only to those who design networks. And, as usual once set up you can’t really change it).

As for your reference for forwarding to NAT networks
https://wiki.libvirt.org/page/Networking#Forwarding_Incoming_Connections

As usual, when you reference something that is very, very old only someone experienced can determine what is still relevant or not.
So, a primary general recommendation is that if the material is new to you, DO SEARCH ONLY for recent documentation, the more recent the better.
In general, the provided configuration should be able to forward to a NAT virtual network.
But, as you probably suspect most people won’t do that or prefer that method since the alternative which is to instead use a bridge device that is configured for bridging(ie shared) mode in the next section immediately below what you referenced is simple and straight-forward. But, don’t follow what is described in the “Bridged networking (aka “shared physical device”)” section which describes creating interface files manually… Just use YaST or use the “brctl addbr” command to create the device(default configuration) or use libvirt(configured correctly).

TSU

16

@malcolmlewis:
or any other moderator/administrator:

With all respect, I think tsu2 is either NOT using openSUSE (instead using perhaps SLES) or his pride is slowly raising lately.
He seems to only check the links he likes, thus ignored the openSUSE virtualization docs link I clearly mentioned in my last post, the one telling that a default virtual network was supposed to be created.
It’s clearly the openSUSE docs, so only way to install KVM in openSUSE is installing qemu and libvirt along! And if libvirt is also installed it should also have created the aforementioned default virtual network, which as I already said it doesn’t!

If it’s a big bug I’m no longer eager to attempt to report it since last report about this has never been touched. Guess it’s definitely better for a moderator/administrator to report bugs.

Again, this is not being very helpful…

17

On Fri 01 Sep 2017 03:36:01 PM CDT, F style wrote:

@malcolmlewis:
or any other moderator/administrator:

With all respect, I think tsu2 is either NOT using openSUSE (instead
using perhaps SLES) or his pride is slowly raising lately.
He seems to only check the links he likes, thus ignored the openSUSE
virtualization docs link I clearly mentioned in my last post, the one
telling that a default virtual network was supposed to be created.
It’s clearly the openSUSE docs, so only way to install KVM in openSUSE
is installing qemu and libvirt along! And if libvirt is also installed
it should also have created the aforementioned default virtual network,
which as I already said it doesn’t!

If it’s a big bug I’m no longer eager to attempt to report it since
‘last report about this’ (Variety wallpaper changer, some few suggestions - Applications - openSUSE Forums) has never been
touched. Guess it’s definitely better for a moderator/administrator to
report bugs.

Again, this is not being very helpful…

Hi
Well I use SLES and SLED (and other SUSE products) as well as
openSUSE :wink: The whole point of Leap is better synergy, both from end
use, documentation, package maintenance.

Again, with openSUSE documentation if something isn’t right, then edit
it accordingly if’s it’s wrong/not working etc so the next
person to not hit the bumps you seem to have. The whole point of
docs/wiki etc.

I’m a command line junky, so old habits die hard and what I do/use
works for setting up libvirt/kvm/qemu in either SLE or openSUSE.

Why not give gnome-boxes a whirl as an interface to libvirt/qemu as
this maybe what you need (and works with NetworkManager).

–
Cheers Malcolm °¿° SUSE Knowledge Partner (Linux Counter #276890)
openSUSE Leap 42.2|GNOME 3.20.2|4.4.79-18.26-default
If you find this post helpful and are logged into the web interface,
please show your appreciation and click on the star below… Thanks!

18

It’s possible to install only KVM without libvirt on openSUSE, it’s not advisable or default but possible.
If you do that, then openSUSE/SUSE documentation does not apply.

Your insistence not to use YaST to install KVM means that your situation is not guaranteed to be the same “default” everyone else begins with and highly inadvisable.
Any comments related to your setup relates only to your own and no one else, and is OK as long as what you’re doing is for educational purposes but is highly inadvisable for anyone/everyone who wants to start with a known stable setup from the beginning and just wants something that works, instead of exploring little seen corners.

TSU

19

And your insistence of playing ignorant or blind starts to be a thing suitable for reporting to moderators.

What part of “last 2 cases” are you not able to get?
The “2 cases” include the one installing through Yast. And even the zypper attempt using the exact command above installs recommended packages, which also includes libvirt itself. If what you said was true I’d have used the --no-recommends option in zypper (and I cannot remember if libvirt is even listed as a “recommended” package anyways!)

What I’m seeing here now is, with permission of moderators to express, someone calling liar to someone else. I don’t think this is appropriate…

20

Seems to have developed into a confusion here, and not a very harmonious confusion either. I think perhaps we close the thread.