firewalld and libvirt zone

English Network/Internet
1

I have a problem with a XEN VM. But I think that the problem is related to the dom0 and firewalld rules.

In 2016 I created a VM (fully virtualized) on my server with this network typology:


 ┌────────────────────────┐                        ┌───────┐
 │ Tumbleweed Server with │ eno1 (192.168.1.120)   │ CABLE │
 │ DHCP + DNS + firewalld ├────────────────────────│ Modem ├─── Internet
 │                        │enslaved in br0 (for VM)│       │
 │                        │                        └───────┘
 └────────────────────────┘                    Gateway (192.168.1.1)

the bridge br0 was created when installing the xen tools. From the VM I could connect to internet and to the dom0.

Some month ago I added a new internet connection via ppoe and some vlan and I needed to set firewalld.
Thereafter I could not more start the VM . Not sure if problem is related to the new connections.
Now I try to recreate the VM on my server.
The server has 3 ethernet ports configured (eno1 for VM, eno2 for vlans, eno3 for ppoe)

My new network topology look like this


 ┌────────────────────────┐                      ┌───────┐
 │ Tumbleweed Server with │ eno3 (no IP)         │ CABLE │
 │ DHCP + DNS + firewalld ├───────────ppp0───────│ Modem ├─── Internet
 │                        │                      │       │
 │ do intervlan routing   │                      └────┬──┘
 └───┬────────────────┬───┘                    Gateway (192.168.1.1)
 eno2 (No IP)     eno1 (192.168.1.120)                
     │                │enslaved in br0 (for VM)       
     │                │                                    
trunk│ port           │                               
 ┌───┴────────────────┴───────────────────────────────┴───┐
 │         TL─SG3216         Swithch Level 2              │
 │                                                        │
 │                         VLAN  ID                       │
 │       2                  3                    4        │
 │(192.168.20.0/24) (192.168.30.0/24)   (192.168.4.0/24)  │
 └───────┬──────────────────┬─────────────────────┬───────┘
         │                  │                     │
        PCs                 PCs                  Printer
  192.168.20.100─     192.168.30.100─      192.168.4.50
  192.168.20.199      192.168.30.199

when I create a VM with virt-manager or Yast the network cannot be found but the creation is succesfully using a local iso.
I configured the network connection in the VM setting a static IP address. (192.168.90.2) and updated the local DNS for this range of address

After this I’m able to connect to the dom0 but not to the internet from the VM.

Before starting the VM the network configuration of the dom0 is like this


hpprol2:~ # ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master br0 state UP group default qlen 1000
    link/ether 9c:8e:99:5b:48:12 brd ff:ff:ff:ff:ff:ff
    altname enp2s0f0
3: eno2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 9c:8e:99:5b:48:13 brd ff:ff:ff:ff:ff:ff
    altname enp2s0f1
4: eno3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 9c:8e:99:5b:48:14 brd ff:ff:ff:ff:ff:ff
    altname enp2s0f2
5: eno4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 9c:8e:99:5b:48:15 brd ff:ff:ff:ff:ff:ff
    altname enp2s0f3
6: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether ae:02:30:64:2e:32 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.100/24 brd 192.168.1.255 scope global br0:100
       valid_lft forever preferred_lft forever
    inet 192.168.1.101/24 brd 192.168.1.255 scope global secondary br0:101
       valid_lft forever preferred_lft forever
    inet 192.168.1.120/24 brd 192.168.1.255 scope global secondary br0
       valid_lft forever preferred_lft forever
7: vlan3@eno2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 9c:8e:99:5b:48:13 brd ff:ff:ff:ff:ff:ff
    inet 192.168.30.1/24 brd 192.168.30.255 scope global vlan3
       valid_lft forever preferred_lft forever
8: vlan4@eno2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 9c:8e:99:5b:48:13 brd ff:ff:ff:ff:ff:ff
    inet 192.168.4.1/24 brd 192.168.4.255 scope global vlan4
       valid_lft forever preferred_lft forever
9: vlan2@eno2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 9c:8e:99:5b:48:13 brd ff:ff:ff:ff:ff:ff
    inet 192.168.20.1/24 brd 192.168.20.255 scope global vlan2
       valid_lft forever preferred_lft forever
10: ppp0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1492 qdisc pfifo_fast state UNKNOWN group default qlen 3
    link/ppp 
    inet xx.xx.xx.xx peer xx.xx.xx.xx/32 scope global ppp0
       valid_lft forever preferred_lft forever
11: virbr1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 52:54:00:3c:cc:26 brd ff:ff:ff:ff:ff:ff
    inet 192.168.90.1/24 brd 192.168.90.255 scope global virbr1
       valid_lft forever preferred_lft forever
hpprol2:~ # ip route show
default dev ppp0 scope link 
10.24.97.36 dev ppp0 proto kernel scope link src 81.241.102.90 
192.168.1.0/24 dev br0 proto kernel scope link src 192.168.1.100 
192.168.4.0/24 dev vlan4 proto kernel scope link src 192.168.4.1 
192.168.20.0/24 dev vlan2 proto kernel scope link src 192.168.20.1 
192.168.30.0/24 dev vlan3 proto kernel scope link src 192.168.30.1 
192.168.90.0/24 dev virbr1 proto kernel scope link src 192.168.90.1 linkdown

The virbr1 was created during creation of the VM.
After starting the VM the network the network configuration in dom0 is (change only)


[FONT=fixedsys]hpprol2:~ # ip a
...
11: virbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 52:54:00:3c:cc:26 brd ff:ff:ff:ff:ff:ff
    inet 192.168.90.1/24 brd 192.168.90.255 scope global virbr1
       valid_lft forever preferred_lft forever
13: vif2.0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master virbr1 state UP group default qlen 32
    link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
[/FONT]

firewalld configuration:
In firewalld I have a new zone libvirt where I activated services dhcp, dns, http, https, ftp, tftp
I can reach the dom0 using ftp client
I cannot ping an internet address
When I start the browser in the VM I receive “Connection failed”
Using wiresharlk in dom0 I see the next lines


No.    Time    Source    Destination    Protocol    Length    Info
68    27.184922774    192.168.90.2    192.168.1.120    DNS    79    Standard query 0x5ec0 A search.opensuse.org
69    27.185321284    192.168.1.120    192.168.90.2    DNS    139    Standard query response 0x5ec0 A search.opensuse.org CNAME proxy.opensuse.org CNAME proxy-nue.opensuse.org A 195.135.221.140
70    27.185670715    192.168.90.2    195.135.221.140    TCP    74    59392 → 443 [SYN] Seq=0 Win=65340 Len=0 MSS=1452 SACK_PERM=1 TSval=689059451 TSecr=0 WS=128
71    27.185713311    192.168.90.1    192.168.90.2    ICMP    102    Destination unreachable (Port unreachable)

In firewalld port 443 is allowed in zone libvirt.

I have some direct rules in firewalld in the default zone that are needed to reach internet from the vlan.

# cat /etc/firewalld/direct.xml
<?xml version="1.0" encoding="utf-8"?>
<direct>
  <rule ipv="ipv4" table="filter" chain="FORWARD" priority="0">-i ppp0 -o vlan2 -m state --state RELATED,ESTABLISHED -j ACCEPT</rule>
  <rule ipv="ipv4" table="filter" chain="FORWARD" priority="0">-i ppp0 -o vlan3 -m state --state RELATED,ESTABLISHED -j ACCEPT</rule>
  <rule ipv="ipv4" table="filter" chain="FORWARD" priority="1">-i vlan2 -o vlan3 -j REJECT</rule>
  <rule ipv="ipv4" table="filter" chain="FORWARD" priority="1">-i vlan3 -o vlan2 -j REJECT</rule>
  <rule ipv="ipv4" table="filter" chain="FORWARD" priority="0">-i ppp0 -o br0 -m state --state RELATED,ESTABLISHED -j ACCEPT</rule>
  <rule ipv="ipv4" table="nat" chain="POSTROUTING" priority="0">-o ppp0 -j MASQUERADE</rule>
  <passthrough ipv="ipv4">-I FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu</passthrough>
</direct>

So I added two rules for the virbr1


  <rule ipv="ipv4" table="filter" chain="FORWARD" priority="0">-i virbr1 -o ppp0 -j ACCEPT</rule>
  <rule ipv="ipv4" table="filter" chain="FORWARD" priority="0">-i ppp0 -o virbr1 -m state --state RELATED,ESTABLISHED -j ACCEPT</rule>

but still no internet connection.

I think that I missed something in the firewalld configuration but I’m unable to find a solution to have vm connection with the internet

Any idea?
Many thanks in advance
Philippe

2

Really nice ASCII drawings :slight_smile:
Can you add a rule to log dropped packets? Maybe it will give you some hints on what is failing.
There was an option to enable that in the old firewall there must be a way to add it with firewalld since I think it’s using iptables under the hood.

3

In Tumbleweed firewalld is based on nftables by default; it is impossible to allow anything using direct rules that is not allowed by main firewalld configuration.

You may consider exploring firewalld policies which offer support to configure forwarding between zones or switching backend back to iptables which should enable you to use direct rules to overrule firewalld (pun intended).

4

Hello,

Thanks for your remark about the limits of direct rules.
I reread the xen network installation (see https://doc.opensuse.org/documentation/leap/virtualization/single-html/book.virt/#cha-libvirt-networks) and I recreated the xen VM and choosing a network definition using an existing bridge (https://doc.opensuse.org/documentation/leap/virtualization/single-html/book.virt/#id-1.8.4.7.6.7.6.11).
This solved the problem and I have now internet connection in the VM. :slight_smile:

This avoid also the creation of a virbr1 bridge and the activation of a libvirt zone in firewalld. The xen VM is using the default zone where br0 is present.
I removed the direct rules that I created in firewalld for the virbr1 device.

As far as I understand the xen network using libvirt connection is a Layer 3 forwarding while using directly the bridge is a Layer 2
See https://doc.opensuse.org/documentation/leap/virtualization/single-html/book.virt/#libvirt-networks-bridged where they explain:

The network bridge configuration provides a Layer 2 switch for VM Guests, switching Layer 2 Ethernet packets between ports on the bridge based on MAC addresses associated with the ports. This gives the VM Guest Layer 2 access to the VM Host Server's network. This configuration is analogous to connecting the VM Guest's virtual Ethernet cable into a hub that is shared with the host and other VM Guests running on the host. The configuration is often referred to as *shared physical device*.

Many thanks
Philippe

5

Oh I didn’t know this. I thought it was iptables. Guess I learned something new today. Thanks :slight_smile:

6

Thank you for posting the solution. I’m really glad it works now.

7

Bringing this back like it’s 2021 because it’s busted again.

Tumbleweed. Virtual Machine Manager 4.1.0.

Attempting to start virtual machine results in error, “Error starting domain: Requested operation is not valid: network ‘default’ is not active.”

Attempting to start the virtual network results in error, “Error starting network ‘default’: internal error: firewalld is set to use the nftables backend, but the required firewalld ‘libvirt’ zone is missing. Either set the firewalld backend to ‘iptables’, or ensure that firewalld has a ‘libvert’ zone by upgrading to firewalld to a version support rule priorities (0.7.0+) and/or rebuilding libvirt with --with-firewalld-zone.”

At first the solution seems as simple as creating a libvirt.xml zone file in an appropriate place. Since that seems too easy, I assume that’s not the solution.

8

Actually, it is that easy.