KVM + Cisco Router CSR1000v

Regarding to this topic: https://forums.opensuse.org/showthread.php/519474-KVM-trunk-interface-on-gues

I reinstalled Opensuse to 42.1. I think that I found simple sollution.
I tried configuratio via virt-manager: passthrough, macvtap, virtio and trunking is working for router.

I also found sollution for multicast:

ip link set dev macvtap1 allmulticast on

but how can I run this scripts when VMs are booted?

With systemd after.local is now working, right? I was reading some pages in internet but tests didn’t work for me.
I also tried hook script for KVM (/etc/libvirt/hooks/qemu) but it is also not wokring.

Any ideas how can I run this allmulticast command when system is booted and virtual routers started (they are starting automaticaly).

First,
I really wouldn’t have approached your scenario as you did, doing any hardware pass-through (and therefor anything related like MacVtap)… You should understand the consequences which would be an atypical setup hardly anyone else is likely going to use.

Instead, the likely more logical approach which would result in a <common configuration> others will recognize is to <follow available documentation> which in this case would be the RHEL link I provided you in the other thread. And then <only if necessary> you bind the vNiCs in the article to physical interfaces. But, even forcibly binding may not be necessary, that’s often automatically configured based on existing devices and routing tables.

Aside from resulting in a configuration others can assist you with (if you even have difficulties and I doubt that too many would assist if you go off the reservation and build something non-standard), your current home-cook solution doesn’t support shared I/O because hardware pass-through is a monopolistic method… When you assign that hardware device to any OS virtual or real, only that OS has access and no other Guest or even the Host itself can monitor, manage or otherwise share functionality.

As for the your question about getting your newly created functionality to start on boot, there are probably several ways to do it, but I’d recommend…

  1. Create a custom systemd Unit file in /etc/systemd/system (copy an existing Unit file from /usr/lib/systemd/system/ for use as a template)
  2. In your newly created Unit file, I’d guess there should be a WantedBy=multi-user.target and Requires=network.service, plus directives to start and possibly stop and restart (ExecStart=).
  3. Point the ExecStart directive to your script.

You may need to experiment a bit to make sure you have your timing and dependencies correct…

TSU

Aside from resulting in a configuration others can assist you with (if you even have difficulties and I doubt that too many would assist if you go off the reservation and build something non-standard), your current home-cook solution doesn’t support shared I/O because hardware pass-through is a monopolistic method… When you assign that hardware device to any OS virtual or real, only that OS has access and no other Guest or even the Host itself can monitor, manage or otherwise share functionality.

I think this is exactly what I need. I want to give routers (instaled on virtual machine) physical interfaces to use. Not touched by operating system, not monitored, not filtered (vlan tagged or broadcast/multicast packets). I have 10 physical interfaces in server and every router will have one.
Anyway, if you are convincing me that it is not correct, then I can try something different. Which exactly redhat instruction should I try?

  1. Create a custom systemd Unit file in /etc/systemd/system (copy an existing Unit file from /usr/lib/systemd/system/ for use as a template)
  2. In your newly created Unit file, I’d guess there should be a WantedBy=multi-user.target and Requires=network.service, plus directives to start and possibly stop and restart (ExecStart=).
  3. Point the ExecStart directive to your script.

You may need to experiment a bit to make sure you have your timing and dependencies correct…

Systemd… !(@#&!@#^!@%# >:(

I don’t know what I did, but now I see:

linux-m797:/home/kamil # rclibvirtd status
libvirtd.service - Virtualization daemon
Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled)
Active: activating (start-post) since Sat 2016-09-10 13:57:41 CEST; 43s ago
Docs: man:libvirtd(8)
http://libvirt.org
Main PID: 2549 (libvirtd); : 2567 (sh)
CGroup: /system.slice/libvirtd.service
|-2549 /usr/sbin/libvirtd --listen
-control |-2567 /bin/sh -c /usr/sbin/vmmulticast -2569 sleep 120

Sep 10 13:57:42 linux-m797 libvirtd[2549]: error from service: CreateMachine: Machine ‘qemu-Router7’ already exists
Sep 10 13:57:42 linux-m797 libvirtd[2549]: Failed to autostart VM ‘Router7’: error from service: CreateMachine: Machine ‘qemu-Router7’ already exists
Sep 10 13:57:43 linux-m797 libvirtd[2549]: error from service: CreateMachine: Machine ‘qemu-Router5’ already exists
Sep 10 13:57:43 linux-m797 libvirtd[2549]: Failed to autostart VM ‘Router5’: error from service: CreateMachine: Machine ‘qemu-Router5’ already exists
Sep 10 13:57:43 linux-m797 libvirtd[2549]: error from service: CreateMachine: Machine ‘qemu-Router2’ already exists
Sep 10 13:57:43 linux-m797 libvirtd[2549]: Failed to autostart VM ‘Router2’: error from service: CreateMachine: Machine ‘qemu-Router2’ already exists
Sep 10 13:57:43 linux-m797 libvirtd[2549]: error from service: CreateMachine: Machine ‘qemu-Router6’ already exists
Sep 10 13:57:44 linux-m797 libvirtd[2549]: Failed to autostart VM ‘Router6’: error from service: CreateMachine: Machine ‘qemu-Router6’ already exists
Sep 10 13:57:44 linux-m797 libvirtd[2549]: error from service: CreateMachine: Machine ‘qemu-Router1’ already exists
Sep 10 13:57:44 linux-m797 libvirtd[2549]: Failed to autostart VM ‘Router1’: error from service: CreateMachine: Machine ‘qemu-Router1’ already exists

Virtual machines are not starting, I can’t run them also via virt-manager:

Błąd podczas uruchamiania domeny (means in english somethink like - error in starting domain): error from service: CreateMachine: Machine ‘qemu-Router1’ already exists

Traceback (most recent call last):
File “/usr/share/virt-manager/virtManager/asyncjob.py”, line 89, in cb_wrapper
callback(asyncjob, *args, **kwargs)
File “/usr/share/virt-manager/virtManager/asyncjob.py”, line 125, in tmpcb
callback(*args, **kwargs)
File “/usr/share/virt-manager/virtManager/libvirtobject.py”, line 83, in newfn
ret = fn(self, *args, **kwargs)
File “/usr/share/virt-manager/virtManager/domain.py”, line 1433, in startup
self._backend.create()
File “/usr/lib64/python2.7/site-packages/libvirt.py”, line 1029, in create
if ret == -1: raise libvirtError (‘virDomainCreate() failed’, dom=self)
libvirtError: error from service: CreateMachine: Machine ‘qemu-Router1’ already exists

The times for using real physical interfaces are rare and nearly non-existent.
I stated some of the reasons (IMO significant) for treading the road most used and following guides which will practically never set up any kind of configuration that configures real, physical objects. Ignore at your own peril, and the principles are fundamental to learning and using any computing technology. For one, if I were running a business and perhaps head of IT, I would never authorize anyone going “off the reservation” to unknown areas where what you create can’t be inspected, serviced, maintained or upgraded by anyone else but you. You will “own” your own creation and the deeper you delve into your custom setup, you will be further removed from a setup anyone else can understand and help.

At the moment, I’m going to guess your libvirt daemon isn’t running.

  • You can query the status of the service with the following
systemctl status libvirtd.service

Your command “rclibvirtd status” returned similar, but only stated “activating” and not “running” with a series of errors which suggests your uniquely and questionable approach to configuring physical objects is preventing the service from running.

These aren’t systemd problems, they’re typical for doing something no one has ever likely done before.
You <might> be successful eventually getting something to work, but it will be far longer than following the RHEL guide but soon enough your setup will be so unique no one will likely be able to offer too much help if something breaks.

TSU

Your answers sound like a part of this article: https://www.mauras.ch/systemd-run-it-last.html

you: Hello, I’m wondering what would be the best way to paint my car red?
them: Why don’t you first start telling us why you think you need to paint your car red
you: ??? I want it red that’s all ???
them: Well usually people think they want their car red, but that’s wrong, they just need it black
you: Well I don’t need it black I want it red
them: Yep, you want it black

I’m saying - the best for me is to attach physical interface to virtual machine because I don’t care about any trafic inspection and I need to send ALL trafic from switch to virtual machine where Cisco Router is installed.
And you say me - no, you don’t need it :slight_smile:
This is my test LAB to learn Cisco for certyfication. I I’ll not touch Linux host if it will start running :wink: And there is no boss, no production system :slight_smile:

But I understand that you can help me only in cases which you tested/understant/are well documented.
So, now I’ll reinstall again my OpenSUSE and try to use this RedHat instruction. Can you please paste it to be sure that I’m using correct instruction?
And we will see if it will work and what problems will appear.

Is that OK 4U? :slight_smile:

Whether you’re building something in a learning lab or production,
The same fundamentals apply, which includes following documentation.

Believe the links to setting up your Cisco device in KVM were in this post
https://forums.opensuse.org/showthread.php/519474-KVM-trunk-interface-on-gues?p=2789348#post2789348

You’ll find that the closer your learning lab approximates what is needed for Production (and by your Cisco testing), the more accurate will be your lessons learned.

And yes,
it’s not just the systemd mentioned in your reference,
Students learning technology suffer from

  • They don’t know enough yet to make comprehensive, “design” decisions to arrive at an optimum result
  • They work in piece meal fashion, addressing one step at a time whereas an experienced architect <knows> how the immediate piece fits into the bigger picture
  • They typically don’t have any knowledge of subsystems and how components interact.
  • They don’t truly understand the full features and functions necessary for the end result to work, indeed they typically are building to learn

All of which makes it that much more important to follow an authoritative guide or bend the ear of someone who has mastered that technology.

TSU

OK, I’ll install fresh OpenSUSE and try this: http://www.cisco.com/c/en/us/td/docs/routers/csr1000/software/configuration/csr1000Vswcfg/installkvm.html

I’m learning Cisco many years. Doing something “right way” is not good idea. You should test everything deeply, try all cases, do thing untypical way. Then you are ready to solve every problem in productive netowrk where you can find many different configurations. Then you are learning good desing but you know all options befor :slight_smile:
Anyway, lets end this offtopic :smiley: I’ll be back with results soon. :slight_smile:

:slight_smile:
Break things only after you know how things should work…

TSU

On 09/14/2016 12:16 PM, tsu2 wrote:
>
> :slight_smile:
> Break things only after you know how things should work…
>
> TSU
>
>

That takes all the fun out of it. :slight_smile:


Ken
linux since 1994
S.u.S.E./openSUSE since 1996

So true :slight_smile:

I checked this instuction and I see two ways:

  1. virt-manager: http://www.cisco.com/c/en/us/td/docs/routers/csr1000/software/configuration/csr1000Vswcfg/installkvm.html#pgfId-1339510
  2. virt-install: http://www.cisco.com/c/en/us/td/docs/routers/csr1000/software/configuration/csr1000Vswcfg/installkvm.html#pgfId-1339719

but there are no details how this bridge should be created? How network interface in virt-manager should be created.
I should create bridge in yast or brctl/network files? Without any special options i suppose?
In virt-manages I should chose virtio to created by me bridge interface?

This is what I have now:

  • I reinstalled OpenSUSE and installed KVM.
  • One interface is for server management and is configured staticaly:

linux-dfua:~ # cat /etc/sysconfig/network/ifcfg-eth1
BOOTPROTO=‘static’
STARTMODE=‘auto’
IPADDR=‘192.168.1.252/24’
NAME=‘hadziuga’

  • via Yast I configured br0 which is connected to eth0. This will be network for managing routers.

linux-dfua:~ # cat /etc/sysconfig/network/ifcfg-eth0
BOOTPROTO=‘none’
STARTMODE=‘auto’
IPADDR=‘’
NAME=‘testLab’
NETMASK=‘’
PREFIXLEN=‘’
linux-dfua:~ # cat /etc/sysconfig/network/ifcfg-br0
BOOTPROTO=‘static’
BRIDGE=‘yes’
BRIDGE_FORWARDDELAY=‘0’
BRIDGE_PORTS=‘eth0’
BRIDGE_STP=‘off’
BROADCAST=‘’
ETHTOOL_OPTIONS=‘’
IPADDR=‘172.16.0.252/24’
MTU=‘’
NAME=‘’
NETWORK=‘’
REMOTE_IPADDR=‘’
STARTMODE=‘auto’

  • I created 3 routers via virt-manager and added them interfaces which are connected to bridge br0 (bridged to eth0), with device model rtl8139. This is working fine. If I configure virtual Cisco router interface then they can ping another routers, ping host, ping another network devices connected to physical interface eth0.
  • Then I created second interfaces. In two routers I created macvtatp, passthrough, virtio to physical interface - if I configure subinterfaces (tagged interfaces/vlan interfaces) on this two virtual routers then they can reach each other, reach another network devices (switches) which have interfaces configured in this V-LAN.
  • http://zapisz.net/images/409_r3vm.png
    ](http://zapisz.net/view.php?filename=409_r3vm.png) - To make it like in RHEL instruction, I also created third virtual machine/virtual router. In this router I added second interface as a bridge to br3 (br3 is bridge to eth4 created in yast). In virt-manager I configured it as virtio. This way configured virtual router is not able to communicate via this bridge interface (configured the same way like two previous routers - I configured one vlan subinterface) with another devices.

http://zapisz.net/images/134_r4vm.png

linux-dfua:~ # cat /etc/sysconfig/network/ifcfg-br3
BOOTPROTO=‘none’
BRIDGE=‘yes’
BRIDGE_FORWARDDELAY=‘0’
BRIDGE_PORTS=‘eth4’
BRIDGE_STP=‘off’
BROADCAST=‘’
ETHTOOL_OPTIONS=‘’
IPADDR=‘’
MTU=‘’
NAME=‘’
NETMASK=‘’
NETWORK=‘’
REMOTE_IPADDR=‘’
STARTMODE=‘auto’

l

linux-dfua:~ # brctl show
bridge name bridge id STP enabled interfaces
br0 8000.80c16e723b64 no eth0
vnet0
vnet1
vnet2
br3 8000.ac162d84152a no eth4
vnet3

br0 is for management. Routers interfaces connected to this bridge are working because there is simple configuration of interfaces - without v-lan.
br3 is to make my configuration “correct way”, like in instruction. Connection via this bridge to router is not workwarding vlan tagged notwork trafic I think.

Bridge configuration started working with v-lans. I found stupid mistake on network side.
I’ll now test multicasts and connections speed (pasthrough should work faster?).
I’ll be back with more results soon :slight_smile:

Congrats on getting this far…

Some FYI that might help make life easier…

Bridge devices (like br0) can be created a number of different ways. If you use certain tools, the bridge device might be named slightly differently, but they are essentially the same and will always be seen the same way.

brctl is the command line tool to view and manage your bridge devices. I commonly use brctl to inspect my device spanning attribute.

If you create a bridge device using libvirt by creating a new “Virtual Network” then the bridge device will be named “virbr” instead of “br” Creating bridge devices using the libvirt virt-manager app also easily exposes a number of options that are harder to configure manually… like configuring the device as a Host-Only or NAT networking, enabling an internal DHCP service (with specified address range) and default gateway(no, you don’t actually install a full-blown DHCP server).

As you’ve discovered, bridge devices can be utilized and shared by multiple Guests and the Host. If you had done pass-through, this would not be possible because hardware pass through always makes one machine (virtual or physical) the sole owner removing access from all others.

There is a tiny bit of overhead using virtualized I/O vs real mode hardware access(often estimated to be about 2%), but it’s usually considered insignificant compared to the benefits.

What you’re learning about bridge devices can be leveraged if in the future you work with other virtualization, almost all common virtualization uses bridge devices, in fact if you had more than one technology running on your machine you’d find that they all can use the same bridge devices no matter what created them (Just don’t run more than one at a time). Some virtualization that use bridge devices… KVM, Xen, Virtualbox, VMware, Hyper-V, LXC. Only major option I know of that doesn’t use bridge devices (but probably can if you really want to) is Docker.

Good Luck,
TSU