firewalld blocks inter vlan ICMP communication

phil524 · January 20, 2021, 10:35pm

Hello,

I have a server with dhcp, dns and a ppoe connection for internet were firewalld is running.
This server acts as a router for a switch (TL-SG3216 only layer2) where multiple PC and a printer are connected to 3 vlan (vlan2 for gamers, vlan3 for home and vlan4 for a printer)
Firewalld is used for internet access for the server and the different PC. This is working.
I can print from the server but not from the PC.

If I stop firewalld (systemctl stop firewalld) I can print from the PC but I loss the internet connection and the firewall protection.
Also with firewald running I cannot reach the printer web interface using the printer IP address (192.168.4.50) from the PC. It works from the server.
The server ethernet interface eno2 is connected to a trunk port of the switch.
tcpdump shows this error when firewalld is running

hpprol2:~ # tcpdump -i eno2 -n -s0 icmp  -v 
tcpdump: listening on eno2, link-type EN10MB (Ethernet), capture size 262144 bytes
22:21:10.044783 IP (tos 0xc0, ttl 64, id 442, offset 0, flags [none], proto ICMP (1), length 80)
    192.168.30.1 > 192.168.30.100: ICMP host 192.168.4.50 unreachable - admin prohibited filter, length 60
        IP (tos 0x0, ttl 63, id 62540, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.30.100.57613 > 192.168.4.50.80: Flags [S], cksum 0x8756 (correct), seq 3197479150, win 64240, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
22:21:10.305078 IP (tos 0xc0, ttl 64, id 480, offset 0, flags [none], proto ICMP (1), length 80)
    192.168.30.1 > 192.168.30.100: ICMP host 192.168.4.50 unreachable - admin prohibited filter, length 60
        IP (tos 0x0, ttl 63, id 62541, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.30.100.57614 > 192.168.4.50.80: Flags [S], cksum 0x60a9 (correct), seq 3757157950, win 64240, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0

If I look at the content of nftables I have the feeling that there are some rules missing for icmp

hpprol2:~ # nft list ruleset
table inet firewalld {
......
chain filter_INPUT {
                type filter hook input priority filter + 10; policy accept;
                ct state { established, related } accept
                ct status dnat accept
                iifname "lo" accept
                jump filter_INPUT_POLICIES_pre
                jump filter_INPUT_ZONES
                jump filter_INPUT_POLICIES_post
                ct state { invalid } drop
                reject with icmpx type admin-prohibited
        }

        chain filter_FORWARD {
                type filter hook forward priority filter + 10; policy accept;
                ct state { established, related } accept
                ct status dnat accept
                iifname "lo" accept
                ip6 daddr { ::/96, ::ffff:0.0.0.0/96, 2002::/24, 2002:a00::/24, 2002:7f00::/24, 2002:a9fe::/32, 2002:ac10::/28, 2002:c0a8::/32, 2002:e000::/19 } reject with icmpv6 type addr-unreachable
                jump filter_FORWARD_POLICIES_pre
                jump filter_FORWARD_IN_ZONES
                jump filter_FORWARD_OUT_ZONES
                jump filter_FORWARD_POLICIES_post
                ct state { invalid } drop
                reject with icmpx type admin-prohibited
        }
......

I think that I need direct rules allowing no rejection of icmp between the vlan but which?

Many thanks in advance
Philippe

arvidjaar · January 21, 2021, 6:34am

You told in title “ICMP communication” and now it turns out much more is blocked.
Without knowing in which VLAN and to which interfaces with which zones your PC and printer are connected it is impossible to say anything.

But if they are in different VLAN reachable via different interfaces which are configured with different zones this is normal behavior - firewalld blocks forwarding except when masquerading is enabled for a zone. Starting with 0.9.0 firewalld supports forwarding between interfaces in the same zone (it is even default), but not between different zones.

I think that I need direct rules allowing no rejection of icmp between the vlan but which?

When using nftables backend you cannot use direct rules to allow anything that is blocked by main firewalld configuration. You will need to switch to iptables backend (or carefully study rule chains created by firewalld and inject your rules there - which is rather fragile).

phil524 · January 21, 2021, 8:19am

Hello,

There are 3 vlan connected to the same interface (eno2 defined as parent of the 3 vlan) and in the same zone.
I try to use the printer (vlan 4 and IP=192.168.4.50) from PC (vlan 3, IP = 192.168.30.100 and gateway 192.168.30.1)
When I do a tcdump on this interface I receive the following dump and the only error seems to be for ICMP
for printing:

# tcpdump -i eno2 -n -s0 net 192.168.30.0/24  -v 
07:25:21.601298 IP (tos 0x0, ttl 64, id 62628, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.30.100.58327 > 192.168.4.50.9100: Flags [S], cksum 0x0973 (correct), seq 3040414248, win 64240, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
07:25:21.601419 IP (tos 0xc0, ttl 64, id 42280, offset 0, flags [none], proto ICMP (1), length 80)
    192.168.30.1 > 192.168.30.100: ICMP host 192.168.4.50 unreachable - admin prohibited filter, length 60
        IP (tos 0x0, ttl 63, id 62628, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.30.100.58327 > 192.168.4.50.9100: Flags [S], cksum 0x0973 (correct), seq 3040414248, win 64240, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0

Trying to reach the printer web interface same error

07:28:14.709800 IP (tos 0x0, ttl 64, id 62660, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.30.100.58349 > 192.168.4.50.80: Flags [S], cksum 0xdd56 (correct), seq 1090964893, win 64240, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
07:28:14.709953 IP (tos 0xc0, ttl 64, id 64355, offset 0, flags [none], proto ICMP (1), length 80)
    192.168.30.1 > 192.168.30.100: ICMP host 192.168.4.50 unreachable - admin prohibited filter, length 60
        IP (tos 0x0, ttl 63, id 62660, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.30.100.58349 > 192.168.4.50.80: Flags [S], cksum 0xdd56 (correct), seq 1090964893, win 64240, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0

I tried adding this direct rule in firewalld

firewall-cmd --direct --add-rule ipv4 filter INPUT 0 -p icmp -s 0.0.0.0/0 -d 0.0.0.0/0 -j ACCEPT

but this doesn’t help.

I have one more question: I don’t see in the firewall-config how to define masquerade for a zone? In the direct rule I can add masquerade for an interface.
Currently my direct rules are:

<?xml version="1.0" encoding="utf-8"?>
<direct>
  <rule ipv="ipv4" table="filter" chain="FORWARD" priority="0">-i vlan2 -j ACCEPT</rule>
  <rule ipv="ipv4" table="filter" chain="FORWARD" priority="0">-i vlan3 -j ACCEPT</rule>
  <rule ipv="ipv4" table="filter" chain="FORWARD" priority="0">-i ppp0 -o vlan2 -m state --state RELATED,ESTABLISHED -j ACCEPT</rule>
  <rule ipv="ipv4" table="filter" chain="FORWARD" priority="0">-i ppp0 -o vlan3 -m state --state RELATED,ESTABLISHED -j ACCEPT</rule>
  <rule ipv="ipv4" table="filter" chain="FORWARD" priority="0">-i vlan4 -j ACCEPT</rule>
  <rule ipv="ipv4" table="filter" chain="FORWARD" priority="1">-i vlan2 -o vlan3 -j REJECT</rule>
  <rule ipv="ipv4" table="filter" chain="FORWARD" priority="1">-i vlan3 -o vlan2 -j REJECT</rule>
  <rule ipv="ipv4" table="nat" chain="POSTROUTING" priority="0">-o ppp0 -j MASQUERADE</rule>
  <passthrough ipv="ipv4">-I FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu</passthrough>
</direct>

When using nftables backend you cannot use direct rules to allow anything that is blocked by main firewalld configuration.

AFAIK with nftables as backend the direct rules are evaluated first?
In tumbleweed nftables version is 0.9.8
Many thanks for your answer
Philippe

arvidjaar · January 21, 2021, 8:47am

This means three different interfaces from OS point of view.

and in the same zone.

Try

firewall-cmd --permanent] --zone=*zone*] --add-forward

AFAIK with nftables as backend the direct rules are evaluated first?

And? How exactly does it change what I said? It is even documented in firewalld manuals.

phil524 · January 21, 2021, 3:01pm

Hello,

Many thanks this solved the problem.

Can I ask you one more question?
This printer worked before with backend iptables.
I shifted to nftables backend and discover the printing problem some days later. Trying to go back to iptables backend didn’t solve the problem.
Did I forgot something with the change of backend or do you have another explanation?

Many thanks in advance
Philippe

arvidjaar · January 22, 2021, 6:26am

Sorry, no idea.

tsu2 · January 22, 2021, 8:01am

The provided command opens up everything between your VLANs, I don’t know if that’s consistent with your reason for creating VLANS in the first place.
If opening everything within your VLANs don’t violate your objectives and purpose, then don’t do anything more.

I haven’t tried to resolve your problem before, but wonder if you might want to instead use firewall-cmd to create ICMP rules that set either the redirect or network-redirect parameters.

I notice you can also edit the nftables directly using the nft command, there seem to be many good references including the following
Simple examples to get started
https://newfivefour.com/nftables-accept-icmp-ping-stop-floods-rate-limiting.html
Full reference(note is also everything you can execute in firewall-cmd)
https://wiki.nftables.org/wiki-nftables/index.php/Quick_reference-nftables_in_10_minutes#Icmp

Just spitballing,
TSU

phil524 · January 22, 2021, 12:15pm

Hello Tsu,

It seems that the isolation between vlan2 and vlan3 is still working. I cannot ping between the pc in 2 different vlan and the network shows only the PC on the same vlan (all these devices are windows machine)
I have two direct rules that block any transfert between these 2 vlan

<rule ipv="ipv4" table="filter" chain="FORWARD" priority="1">-i vlan2 -o vlan3 -j REJECT</rule>
 <rule ipv="ipv4" table="filter" chain="FORWARD" priority="1">-i vlan3 -o vlan2 -j REJECT</rule>

These rules seem still working:)
Many thanks for your references about nftables I’ll read this very carefully.
Just for info here my current network toplogy

    [FONT=courier new]------------------------               ---------
    | Tumbleweed Server with| eno3         | CABLE |
    | DHCP + DNS + firewalld|-----ppp0-----| Modem |--- Internet
    |                       |              |       |
    | do intervlan routing  |              ---------
    -------------------------           
    eno2                   | eno1 enslaved in br0 (for VM)    
     |                     |
trunk| port                |
    --------------------------------------------------------------
    |         L-SG3216    Swithch Level 2                        |
    |               VLAN  ID                                     |
    | 2(192.168.20.0/24 )  3(192.168.30.0/24 )  4(192.168.4.0/24)|
    --------------------------------------------------------------
      |                    |                    |
      PCs                  PCs                  Printer
192.168.20.100-            192.168.30.100-      192.168.4.50
192.168.20.199             192.168.30.199

Regards
Philippe
[/FONT]

arvidjaar · January 22, 2021, 1:09pm

Yes, it is possible to use direct rules with firewalld nftables backend to block something.

Just for info here my current network toplogy

That is what you should have started with.