Docker network is blocked by firewalld

Hello,

docker (version 19.03.15, build 99e3ed89195c) doesn’t start on fresh openSuse 15.2 installation.

 
sudo dockerd 
WARN[2021-02-13T10:14:57.722335384+01:00] failed to load plugin io.containerd.snapshotter.v1.devmapper  error="devmapper not configured"
WARN[2021-02-13T10:14:57.723689078+01:00] could not use snapshotter devmapper in metadata plugin  error="devmapper not configured"
WARN[2021-02-13T10:14:58.034228793+01:00] Your kernel does not support swap memory limit 
WARN[2021-02-13T10:14:58.034282072+01:00] Your kernel does not support cgroup rt period 
WARN[2021-02-13T10:14:58.034288675+01:00] Your kernel does not support cgroup rt runtime 
WARN[2021-02-13T10:14:58.034294045+01:00] Your kernel does not support cgroup blkio weight 
WARN[2021-02-13T10:14:58.034299084+01:00] Your kernel does not support cgroup blkio weight_device 
WARN[2021-02-13T10:14:58.199764957+01:00] could not create bridge network for id 4054dc3a36c8c2ad0f41a64a716e7ea90f572b1f2d9251d570a4e7a39602dcca bridge name docker0 while booting up from persistent state: Failed to program NAT chain: INVALID_ZONE: docker 
failed to start daemon: Error initializing network controller: Error creating default "bridge" network: Failed to program NAT chain: INVALID_ZONE: docker

The problem is caused by firewalld. Docker starts ok, when is the firewalld disbaled ( or if I use UFW).

The firewalld settings are (almost default conf.):


sudo firewall-cmd --list-all
  home (active)
  target: default
  icmp-block-inversion: no
  interfaces: eth0
  sources: 
  services: ssh mdns samba-client dhcpv6-client docker-registry docker-swarm
  ports: 
  protocols: 
  masquerade: yes
  forward-ports: 
  source-ports: 
  icmp-blocks: 
  rich rules:

and the network configuration is:


1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 52:54:00:d4:97:b2 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.205/24 brd 192.168.122.255 scope global dynamic noprefixroute eth0
       valid_lft 3252sec preferred_lft 3252sec
    inet6 fe80::417d:4237:a06b:fff6/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 02:42:12:90:48:fb brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever

Could you please help me to proeprly set up the Firewalld to enable the docker start?

Did you try to create zone docker in firewalld?

Just hit this too

Part of the docker init is that it creates and manages the DOCKER and DOCKER-USER zones and manual changes to that can cause chaos.

From /usr/lib/systemd/system/docker.service

[Unit]
Description=Docker Application Container Engine
Documentation=http://docs.docker.com
After=network.target lvm2-monitor.service SuSEfirewall2.service


[Service]
EnvironmentFile=/etc/sysconfig/docker


# While Docker has support for socket activation (-H fd://), this is not
# enabled by default because enabling socket activation means that on boot your
# containers won't start until someone tries to administer the Docker daemon.
Type=notify
ExecStart=/usr/bin/dockerd --add-runtime oci=/usr/sbin/docker-runc $DOCKER_NETWORK_OPTIONS $DOCKER_OPTS
ExecReload=/bin/kill -s HUP $MAINPID

You can see the $DOCKER_NETWORK_OPTIONS on the exec line

The part that worries me is the after statement at the top still including SuSEfirewall2.

Any help would be welcome. I’m guessing that as they released this for SLEs on Friday that there will be some changes there very shortly

Tim

Just to highlight the problem with the docker insertion into iptables

Feb 13 17:52:22 t-web2 firewalld[961]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -D PREROUTING -m addrtype --dst-type LOCAL -j DOCKER' failed: iptables: No chain/target/match by th>
Feb 13 17:52:22 t-web2 firewalld[961]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -D OUTPUT -m addrtype --dst-type LOCAL ! --dst 127.0.0.0/8 -j DOCKER' failed: iptables: No chain/ta>
Feb 13 17:52:22 t-web2 firewalld[961]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -D OUTPUT -m addrtype --dst-type LOCAL -j DOCKER' failed: iptables: No chain/target/match by that n>
Feb 13 17:52:22 t-web2 firewalld[961]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -D PREROUTING' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Feb 13 17:52:22 t-web2 firewalld[961]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -D OUTPUT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Feb 13 17:52:22 t-web2 firewalld[961]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -F DOCKER-ISOLATION' failed: iptables: No chain/target/match by that name.
Feb 13 17:52:22 t-web2 firewalld[961]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -X DOCKER-ISOLATION' failed: iptables: No chain/target/match by that name.
Feb 13 17:52:23 t-web2 firewalld[961]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -D FORWARD -i docker0 -o docker0 -j DROP' failed: iptables: Bad rule (does a matching rule exist in that c>
Feb 13 17:52:23 t-web2 firewalld[961]: ERROR: INVALID_ZONE: docker
Feb 13 17:52:23 t-web2 dockerd[19029]: time="2021-02-13T17:52:23.088594096Z" level=warning msg="could not create bridge network for id 615284ea44da1493914ec8209a35e5210b2bf4a9ce265b429e37c4ea3d5>
Feb 13 17:52:23 t-web2 firewalld[961]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -D FORWARD -i docker0 -o docker0 -j DROP' failed: iptables: Bad rule (does a matching rule exist in that c>
Feb 13 17:52:23 t-web2 firewalld[961]: ERROR: INVALID_ZONE: docker
Feb 13 17:52:24 t-web2 dockerd[19029]: failed to start daemon: Error initializing network controller: Error creating default "bridge" network: Failed to program NAT chain: INVALID_ZONE: docker
Feb 13 17:52:24 t-web2 systemd[1]: docker.service: Main process exited, code=exited, status=1/FAILURE
Feb 13 17:52:24 t-web2 systemd[1]: Failed to start Docker Application Container Engine.
-- Subject: Unit docker.service has failed
-- Defined-By: systemd

I have 2 LEAP 15.2 servers with docker doing simple containers - one test and one live. The live is unpatched and working.

Looking at the history of /var/log/zypp you cans see:
# 2021-02-13 13:26:07 docker-19.03.15_ce-lp152.2.3.1.x86_64.rpm installed ok
# Additional rpm output:
# Updating /etc/sysconfig/docker ...
# Job for docker.service failed because the control process exited with error code.

and yet zypper continued without displaying any issues.

More as I find it

Tim

See bugzilla:
https://bugzilla.opensuse.org/show_bug.cgi?id=1182168

Thanks for the update

Looks like I can stand down for a bit :slight_smile:

Take care

T

Just for completeness I created a new server directly patched. Installed without docker.

Added docker with

zypper in docker

which added 16 packages

Docker then failed to start

-- Unit docker.service has begun starting up.
Feb 13 18:33:40 install dockerd[6496]: time="2021-02-13T18:33:40.903821639Z" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.devmapper" error="devmapper not configured"
Feb 13 18:33:40 install dockerd[6496]: time="2021-02-13T18:33:40.907439006Z" level=warning msg="could not use snapshotter devmapper in metadata plugin" error="devmapper not configured"
Feb 13 18:33:40 install dockerd[6496]: time="2021-02-13T18:33:40.982355588Z" level=warning msg="Your kernel does not support swap memory limit"
Feb 13 18:33:40 install dockerd[6496]: time="2021-02-13T18:33:40.982444356Z" level=warning msg="Your kernel does not support cgroup rt period"
Feb 13 18:33:40 install dockerd[6496]: time="2021-02-13T18:33:40.982459207Z" level=warning msg="Your kernel does not support cgroup rt runtime"
Feb 13 18:33:40 install dockerd[6496]: time="2021-02-13T18:33:40.982471883Z" level=warning msg="Your kernel does not support cgroup blkio weight"
Feb 13 18:33:40 install dockerd[6496]: time="2021-02-13T18:33:40.982484587Z" level=warning msg="Your kernel does not support cgroup blkio weight_device"
Feb 13 18:33:41 install firewalld[1198]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -D PREROUTING -m addrtype --dst-type LOCAL -j DOCKER' failed: iptables: No chain/target/match by t>
Feb 13 18:33:41 install firewalld[1198]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -D OUTPUT -m addrtype --dst-type LOCAL ! --dst 127.0.0.0/8 -j DOCKER' failed: iptables: No chain/t>
Feb 13 18:33:41 install firewalld[1198]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -D OUTPUT -m addrtype --dst-type LOCAL -j DOCKER' failed: iptables: No chain/target/match by that >
Feb 13 18:33:41 install firewalld[1198]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -D PREROUTING' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Feb 13 18:33:41 install firewalld[1198]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -D OUTPUT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Feb 13 18:33:41 install firewalld[1198]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -F DOCKER-ISOLATION' failed: iptables: No chain/target/match by that name.
Feb 13 18:33:41 install firewalld[1198]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -X DOCKER-ISOLATION' failed: iptables: No chain/target/match by that name.
Feb 13 18:33:41 install firewalld[1198]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -D FORWARD -i docker0 -o docker0 -j DROP' failed: iptables: Bad rule (does a matching rule exist in that >
Feb 13 18:33:41 install firewalld[1198]: ERROR: INVALID_ZONE: docker
Feb 13 18:33:42 install dockerd[6496]: failed to start daemon: Error initializing network controller: Error creating default "bridge" network: Failed to program NAT chain: INVALID_ZONE: docker
Feb 13 18:33:42 install systemd[1]: docker.service: Main process exited, code=exited, status=1/FAILURE
Feb 13 18:33:42 install systemd[1]: Failed to start Docker Application Container Engine.
-- Subject: Unit docker.service has failed
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit docker.service has failed.



So confirmed that it doesn’t matter if this is a patch or new install. It’s broken.

Have a grand weekend

T

From the error that was thrown, my first suspicion is that a bridge device hasn’t been created.
I don’t know if this is really a packaging error, there are a few ways for Docker networking to be implemented, and it’s only the past few years that Docker has been recommending using a Linux bridge device like other virtualization… Originally Docker networking was done differently, was implemented its own unique way without the missing bridge device your configuration requires.

There are probably a zillion ways to create a bridge device, and a bridge device can be created using any technology… once created a bridge device can be used by that or any other multi-tenant technology on the machine.

Although I could describe how to do this (or is probably found in numerous places on the Internet using the ip command),
Probably the easiest way is to have YaST create it for you installing another virtualization technology (KVM, Xen or LXC). You can leave the other virtualization technology in place or uninstall it, leaving the bridge device, docker can be run safely and easily side by side with other technologies without a problem.
Or, although I haven’t done it recently, it can be created using YaST Network Settings.

If you have problems with any of the above solutions to creating a network bridge device on your machine, post again and I or someone else can post more detailed steps.

TSU

@tsu

Do you read the bugreport?

rotfl!rotfl!rotfl!rotfl!

Took a closer look this morning.
After updating the 15.2 system, fired up a container, is not having a problem connecting to a network.
For anyone who observed this problem, do you still see it?

If so,
I’ll have to take a closer look at whether my networking is set up to test the problem.

TSU

Hello,

thank you for the help. The docker works without problems after the update.