Rootless podman containers fail to start with different permission errors

English Applications
1

Hi there,

I just upgraded to snapshot 20231005-0 and kernel 6.5.4-1-default (I avoided the kernel upgrade until now because of zfs).

Unfortunately now the exact same containers that were working before the upgrade refuse to start with different permission errors.

For example docker.io/library/mariadb:10.6:

/bin/bash: error while loading shared libraries: /lib/x86_64-linux-gnu/libc.so.6: cannot apply additional memory protection after relocation: Permission denied

Or docker.io/pymedusa/medusa:latest:

Error relocating /lib/ld-musl-x86_64.so.1: RELRO protection failed: Permission denied
Error relocating runscripts/init.docker: RELRO protection failed: Permission denied

I searched around a bit and found this bug report but the suggested solution (running restorecon -R /var/lib/containers/storage-1000/overlay*) did not help.

Running ausearch -ts boot -m avc I can see denied requests:

----
time->Sat Oct  7 15:47:36 2023
type=AVC msg=audit(1696686456.805:482): avc:  denied  { read } for  pid=4712 comm="sh" path="/lib/ld-musl-x86_64.so.1" dev="dm-1" ino=78031 scontext=system_u:system_r:container_t:s0:c357,c799 tcontext=unconfined_u:object_r:container_var_lib_t:s0 tclass=file permissive=0
----
time->Sat Oct  7 15:47:36 2023
type=AVC msg=audit(1696686456.805:483): avc:  denied  { read } for  pid=4712 comm="sh" path="/bin/busybox" dev="dm-1" ino=77868 scontext=system_u:system_r:container_t:s0:c357,c799 tcontext=unconfined_u:object_r:container_var_lib_t:s0 tclass=file permissive=0
----
time->Sat Oct  7 15:50:12 2023
type=AVC msg=audit(1696686612.246:485): avc:  denied  { read } for  pid=6130 comm="init.docker" path="/lib/ld-musl-x86_64.so.1" dev="dm-1" ino=578840 scontext=system_u:system_r:container_t:s0:c74,c178 tcontext=unconfined_u:object_r:container_var_lib_t:s0 tclass=file permissive=0
----
time->Sat Oct  7 15:50:12 2023
type=AVC msg=audit(1696686612.246:486): avc:  denied  { read } for  pid=6130 comm="init.docker" path="/bin/busybox" dev="dm-1" ino=578677 scontext=system_u:system_r:container_t:s0:c74,c178 tcontext=unconfined_u:object_r:container_var_lib_t:s0 tclass=file permissive=0

I didn’t change anything in the selinux config though.

Any ideas how I can fix this besides setting permissive mode for selinux?

Thanks

2

Most search hits point to SELinux. Does it work after

setenforce 0

(you may need to install selinux-tools package, not sure if it is present by default).

3

The containers start if I set --security-opt label=disable so it is definitely selinux.

I’d rather not disable the extra security though and it worked before without issue. So something must have changed that broke the containers. I have no idea what though or where to start looking.

4

Looks like I figured out the problem. The root cause seems to be that I changed the location of rootless_storage_path in /etc/containers/storage.conf without correctly relabeling the target folder.

Doing the following fixed my issue:

$ semanage fcontext -a -e ~/.local/share/containers/storage /NEWSTORAGEPATH
$ restorecon -R -v /NEWSTORAGEPATH
1 Like