Bubblewrap in apptainer: Tumbleweed works, Leap16.0 does not

repoer · September 11, 2026, 9:10pm

I packaged an application inside an apptainer to isolate it from my host. I did this on my tumbleweed running laptop. I then transferred the apptainer to a machine running Leap16.0, where it did not run as expected.
Some testing showed that it boils down to the following: the ‘bwrap’ command inside the arch-based apptainer image works on a tumbleweed host, but fails on a Leap16.0 host.

To recreate, I executed the following in one Tumbleweed/Leap16.0 VM each:

##################################################################
# Prep work - done beforehand
##################################################################
## install leap16 / tumbleweed with default settings            
#sudo zypper up
#sudo zypper in apptainer
#sudo reboot

##################################################################
# Create simple apptainer def for arch with bubblewrap installed
##################################################################
echo "BootStrap: docker"                   > bwrap.def
echo "From: archlinux"                     >> bwrap.def
echo "%post"                               >> bwrap.def
echo "  pacman-key --init"                 >> bwrap.def
echo "  pacman -Syyu --noconfirm"          >> bwrap.def
echo "  pacman -S --noconfirm bubblewrap"  >> bwrap.def
echo "  pacman -Scc --noconfirm"           >> bwrap.def

##################################################################
# Build just defined apptainer
##################################################################
apptainer build bwrap.sif bwrap.def

##################################################################
# Try to execute bash through bubblewrap inside container
##################################################################
apptainer exec --containall bwrap.sif bwrap --bind / / bash

On Tumbleweed, I do get a bash shell inside apptainer.
On Leap16.0, I do get the following error:

user@leap16vm:~/test_script> apptainer exec --containall bwrap.sif bwrap --bind / / bash
bwrap: Can't bind mount / on /: Unable to mount source on destination: Invalid argument

Things I have already tried:
Disabling SELinux - has no effect
Installing the apptainer RPM from Tumbleweed on Leap16.0 - has no effect.

So it seems to be caused by some difference between the two operating systems. Does anyone have an idea what might be causing this or how I can debug this?

deano_ferrari · September 11, 2026, 10:22pm

I’m not very familiar with using apptainer or bubblewrap, but it might be useful to try bwrap --bind / / bash directly on the Leap 16 system (outside of apptainer).

This will start a bubblewrap sandbox, make the existing root filesystem available as /, and present a bash shell inside the sandbox. If that fails, it points to a problem with bubblewrap itself, or the Leap 16 host’s kernel, or namespace/mount handling perhaps. Report back with your findings.

If I get time I’ll check for myself.

Manpage reference:
https://manpages.debian.org/trixie/bubblewrap/bwrap.1.en.html
Archwiki reference:
https://wiki.archlinux.org/title/Bubblewrap/Examples

FWIW, an interesting Medium article on using it.

repoer · September 12, 2026, 7:32pm

Thanks for the suggestion! When I run

bwrap --bind / / bash

I do get the same output on leap / tumbleweed:

Despite the “bash: /dev/null: Permission denied”, it seems to work as expected - a new bash shell is spawned (where grep output isn’t colored) and after an ‘exit’ I’m not logged out, but end up where I started. So bubblewrap itself seems to run fine on both systems.

deano_ferrari · September 12, 2026, 9:15pm

Ok, so bwrap works directly on both hosts. That is good to know. So now we need to understand how Apptainer is configured/isolates the container on Leap vs Tumbleweed.

Compare the apptainer config global and apptainer version output between Leap and Tumbleweed hosts.

repoer · September 13, 2026, 1:46pm

As I already mentioned, even with the tumbleweed version of apptainer installed on leap16, the behaviour is the same.

apptainer config global

gave me the following instead of showing config values:

user@leap16vm:~> apptainer config global
Error for command "global": accepts between 1 and 2 arg(s), received 0

Usage:
  apptainer [global options...] config global <option> <directive> [value,...]

Run 'apptainer --help' for more detailed usage information.

So, instead I mounted the /etc/apptainer/ directory from the leap vm in tumbleweed through sshfs into ‘remote_apptainer’ and compared the two:

user@tumbleweedvm:~> sudo diff -rs remote_apptainer /etc/apptainer
Files remote_apptainer/apptainer.conf and /etc/apptainer/apptainer.conf are identical
Files remote_apptainer/capability.json and /etc/apptainer/capability.json are identical
Files remote_apptainer/cgroups/cgroups.toml and /etc/apptainer/cgroups/cgroups.toml are identical
Files remote_apptainer/dmtcp-conf.yaml and /etc/apptainer/dmtcp-conf.yaml are identical
Files remote_apptainer/ecl.toml and /etc/apptainer/ecl.toml are identical
Files remote_apptainer/global-pgp-public and /etc/apptainer/global-pgp-public are identical
Files remote_apptainer/network/00_bridge.conflist and /etc/apptainer/network/00_bridge.conflist are identical
Files remote_apptainer/network/10_ptp.conflist and /etc/apptainer/network/10_ptp.conflist are identical
Files remote_apptainer/network/20_ipvlan.conflist and /etc/apptainer/network/20_ipvlan.conflist are identical
Files remote_apptainer/network/30_macvlan.conflist and /etc/apptainer/network/30_macvlan.conflist are identical
Files remote_apptainer/network/40_fakeroot.conflist and /etc/apptainer/network/40_fakeroot.conflist are identical
Files remote_apptainer/nvliblist.conf and /etc/apptainer/nvliblist.conf are identical
Files remote_apptainer/remote.yaml and /etc/apptainer/remote.yaml are identical
Files remote_apptainer/rocmliblist.conf and /etc/apptainer/rocmliblist.conf are identical
Files remote_apptainer/seccomp-profiles/default.json and /etc/apptainer/seccomp-profiles/default.json are identical

As far as I can tell, the configurations are identical and I haven’t touched any defaults.

To be complete, here are the apptainer versions for Leap16:

user@leap16vm:~> apptainer --version
apptainer version 1.5.3-bp160.3.1

And for tumbleweed:

user@tumbleweedvm:~> apptainer --version
apptainer version 1.5.3-3.1

Since the version is very close, the config is identical and even with the tumbleweed version on leap I get the same behaviour, I believe the difference has to be somewhere else.

deano_ferrari · September 13, 2026, 6:01pm

Ok, no further ideas about this from me. Let’s see what others can suggest.

dza · September 14, 2026, 4:17am

try

strace -f – $failing_command_here

Compare on both, it might reveal the diff.

repoer · September 17, 2026, 9:28pm

Thanks for the idea - I have the full log files here:
Tumbleweed: https://justpaste.it/gs2ue
Leap 16: https://justpaste.it/gd24l

Since these are 10K+ lines each and I’m not used to reading these, there is not all that much that I can see from them. I at least found the section that ultimately throws the error, which looks like this in Leap:

...
[pid  1834] mount(NULL, "/", NULL, MS_REC|MS_SILENT|MS_SLAVE, NULL) = 0
[pid  1834] mount("tmpfs", "/tmp", "tmpfs", MS_NOSUID|MS_NODEV, NULL) = 0
[pid  1834] newfstatat(AT_FDCWD, ".", {st_mode=S_IFDIR|0755, st_size=40, ...}, 0) = 0
[pid  1834] newfstatat(AT_FDCWD, "/home/user", {st_mode=S_IFDIR|0755, st_size=40, ...}, 0) = 0
[pid  1834] chdir("/tmp")               = 0
[pid  1834] mkdir("newroot", 0755)      = 0
[pid  1834] mount("newroot", "newroot", NULL, MS_MGC_VAL|MS_BIND|MS_REC|MS_SILENT, NULL) = 0
[pid  1834] mkdir("oldroot", 0755)      = 0
[pid  1834] pivot_root("/tmp", "oldroot") = 0
[pid  1834] chdir("/")                  = 0
[pid  1834] mkdir("proc", 0755)         = 0
[pid  1834] mount("oldroot/proc", "proc", NULL, MS_BIND|MS_REC|MS_SILENT, NULL) = 0
[pid  1834] openat(AT_FDCWD, "/oldroot", O_RDONLY|O_NOCTTY|O_CLOEXEC|O_PATH) = 4
[pid  1834] openat2(4, "/", {flags=O_RDONLY|O_PATH, resolve=RESOLVE_NO_MAGICLINKS|RESOLVE_IN_ROOT}, 24) = 5
[pid  1834] close(4)                    = 0
[pid  1834] fstat(5, {st_mode=S_IFDIR|0755, st_size=80, ...}) = 0
[pid  1834] openat(AT_FDCWD, "/newroot", O_RDONLY|O_NOCTTY|O_CLOEXEC|O_PATH) = 4
[pid  1834] openat2(4, "/", {flags=O_RDONLY|O_DIRECTORY, resolve=RESOLVE_NO_MAGICLINKS|RESOLVE_IN_ROOT}, 24) = 6
[pid  1834] close(4)                    = 0
[pid  1834] mkdirat(6, ".", 0755)       = -1 EEXIST (File exists)
[pid  1834] openat(6, ".", O_RDONLY|O_NOFOLLOW|O_CLOEXEC|O_PATH) = 4
[pid  1834] fstat(4, {st_mode=S_IFDIR|0755, st_size=40, ...}) = 0
[pid  1834] readlink("/proc/self/fd/4", "/newroot", 99) = 8
[pid  1834] mount("/proc/self/fd/5", "/proc/self/fd/4", NULL, MS_BIND|MS_REC|MS_SILENT, NULL) = -1 EINVAL (Invalid argument)
[pid  1821] <... read resumed>, "P\0\0\0\17\0\0\0\16\1\0\0\0\0\0\0b\34T\7\1\0\0\0\350\3\0\0\350\3\0\0"..., 135168) = 80
[pid  1821] writev(5, [{iov_base="\20@\1\0\0\0\0\0\16\1\0\0\0\0\0\0", iov_len=16}, {iov_base="\0\0\351\372\342\377\377\307\205\200\373\377\377\10\0\0\0\213\205P\373\377\377\211\205\204\373\377\377\351\300\366"..., iov_len=81920}], 2) = 81936
[pid  1834] write(2, "bwrap: ", 7 <unfinished ...>
bwrap: [pid  1821] read(5 <unfinished ...>
[pid  1834] <... write resumed>)        = 7
[pid  1834] write(2, "Can't bind mount / on /", 23Can't bind mount / on /) = 23
[pid  1834] write(2, ": Unable to mount source on dest"..., 39: Unable to mount source on destination) = 39
[pid  1834] write(2, ": Invalid argument", 18: Invalid argument) = 18
...

On Tumbleweed, the comparable section does not contain the “EINVAL” return value and continues with several "mount"s - that would then probably be bubblewrap working its magic?

...
[pid  1698] mount(NULL, "/", NULL, MS_REC|MS_SILENT|MS_SLAVE, NULL) = 0
[pid  1696] <... signalfd4 resumed>)    = 3
[pid  1698] mount("tmpfs", "/tmp", "tmpfs", MS_NOSUID|MS_NODEV, NULL) = 0
[pid  1696] poll([{fd=3, events=POLLIN}], 1, -1 <unfinished ...>
[pid  1698] newfstatat(AT_FDCWD, ".", {st_mode=S_IFDIR|0755, st_size=40, ...}, 0) = 0
[pid  1698] newfstatat(AT_FDCWD, "/home/user", {st_mode=S_IFDIR|0755, st_size=40, ...}, 0) = 0
[pid  1698] chdir("/tmp")               = 0
[pid  1698] mkdir("newroot", 0755)      = 0
[pid  1698] mount("newroot", "newroot", NULL, MS_MGC_VAL|MS_BIND|MS_REC|MS_SILENT, NULL) = 0
[pid  1698] mkdir("oldroot", 0755)      = 0
[pid  1698] pivot_root("/tmp", "oldroot") = 0
[pid  1698] chdir("/")                  = 0
[pid  1698] mkdir("proc", 0755)         = 0
[pid  1698] mount("oldroot/proc", "proc", NULL, MS_BIND|MS_REC|MS_SILENT, NULL) = 0
[pid  1698] openat(AT_FDCWD, "/oldroot", O_RDONLY|O_NOCTTY|O_CLOEXEC|O_PATH) = 4
[pid  1698] openat2(4, "/", {flags=O_RDONLY|O_PATH, resolve=RESOLVE_NO_MAGICLINKS|RESOLVE_IN_ROOT}, 24) = 5
[pid  1698] close(4)                    = 0
[pid  1698] fstat(5, {st_mode=S_IFDIR|0755, st_size=100, ...}) = 0
[pid  1698] openat(AT_FDCWD, "/newroot", O_RDONLY|O_NOCTTY|O_CLOEXEC|O_PATH) = 4
[pid  1698] openat2(4, "/", {flags=O_RDONLY|O_DIRECTORY, resolve=RESOLVE_NO_MAGICLINKS|RESOLVE_IN_ROOT}, 24) = 6
[pid  1698] close(4)                    = 0
[pid  1698] mkdirat(6, ".", 0755)       = -1 EEXIST (File exists)
[pid  1698] openat(6, ".", O_RDONLY|O_NOFOLLOW|O_CLOEXEC|O_PATH) = 4
[pid  1698] fstat(4, {st_mode=S_IFDIR|0755, st_size=40, ...}) = 0
[pid  1698] readlink("/proc/self/fd/4", "/newroot", 99) = 8
[pid  1698] mount("/proc/self/fd/5", "/proc/self/fd/4", NULL, MS_BIND|MS_REC|MS_SILENT, NULL) = 0
[pid  1698] openat(3, "self/mountinfo", O_RDONLY|O_CLOEXEC) = 7
[pid  1698] read(7, "361 474 0:72 / /oldroot rw,nodev"..., 4079) = 3996
[pid  1698] read(7, "475 474 0:93 /newroot /newroot r"..., 83) = 83
[pid  1698] read(7, "id=1000,gid=1000,inode64\n543 474"..., 4080) = 3988
[pid  1698] read(7, "", 92)             = 0
[pid  1698] close(7)                    = 0
[pid  1698] mount("none", "/newroot", NULL, MS_NOSUID|MS_NODEV|MS_REMOUNT|MS_BIND|MS_SILENT|MS_RELATIME, NULL) = 0
[pid  1698] mount("none", "/newroot/dev", NULL, MS_NOSUID|MS_NODEV|MS_REMOUNT|MS_BIND|MS_SILENT|MS_RELATIME, NULL) = 0
[pid  1698] mount("none", "/newroot/dev/pts", NULL, MS_NOSUID|MS_NODEV|MS_NOEXEC|MS_REMOUNT|MS_BIND|MS_SILENT|MS_RELATIME, NULL) = 0
[pid  1698] mount("none", "/newroot/dev/console", NULL, MS_NOSUID|MS_NODEV|MS_NOEXEC|MS_REMOUNT|MS_BIND|MS_SILENT|MS_RELATIME, NULL) = 0
[pid  1698] mount("none", "/newroot/dev/tty", NULL, MS_NOSUID|MS_NODEV|MS_REMOUNT|MS_BIND|MS_SILENT, NULL) = 0
[pid  1698] mount("none", "/newroot/dev/null", NULL, MS_NOSUID|MS_NODEV|MS_REMOUNT|MS_BIND|MS_SILENT, NULL) = 0
[pid  1698] mount("none", "/newroot/dev/zero", NULL, MS_NOSUID|MS_NODEV|MS_REMOUNT|MS_BIND|MS_SILENT, NULL) = 0
[pid  1698] mount("none", "/newroot/dev/random", NULL, MS_NOSUID|MS_NODEV|MS_REMOUNT|MS_BIND|MS_SILENT, NULL) = 0
[pid  1698] mount("none", "/newroot/dev/urandom", NULL, MS_NOSUID|MS_NODEV|MS_REMOUNT|MS_BIND|MS_SILENT, NULL) = 0
[pid  1698] mount("none", "/newroot/sys/fs/selinux", NULL, MS_NOSUID|MS_NODEV|MS_NOEXEC|MS_REMOUNT|MS_BIND|MS_SILENT|MS_RELATIME, NULL) = 0
[pid  1698] mount("none", "/newroot/etc/passwd", NULL, MS_NOSUID|MS_NODEV|MS_REMOUNT|MS_BIND|MS_SILENT|MS_RELATIME, NULL) = 0
[pid  1698] mount("none", "/newroot/etc/group", NULL, MS_NOSUID|MS_NODEV|MS_REMOUNT|MS_BIND|MS_SILENT|MS_RELATIME, NULL) = 0
...

Since I can’t reasonably expect anyone to go through 25K lines of logs for me, do you maybe have some tips what I could look out for?

Also, I did an additional experiment. To rule out the arch userland as a cause for problems / incompatablities with leap, I built an apptainer image with latest leap & bubblewrap inside. The behaviour is the same: It works as expected on the tumbleweed host and fails on the leap host.

repoer · October 5, 2026, 7:23pm

In the meantime, I did not gain any meaningful insight from strace output. So I still don’t know what exactly causes this issue.
I did, however, find a workaround. For me, following “SDB:InstallNewerKernel” (https://en.opensuse.org/SDB:InstallNewerKernel) did solve the issues - with everything else unchanged and just the kernel updated, everything works as expected.
If anyone knows what exactly caused this or why changing the kernel made such a difference, I would be happy to hear it.

GrandDixence2 · October 5, 2026, 9:38pm

[pid  1834] mount("/proc/self/fd/5", "/proc/self/fd/4", NULL, MS_BIND|MS_REC|MS_SILENT, NULL) = -1 EINVAL (Invalid argument)

# man 2 mount

EINVAL
source had an invalid superblock.
EINVAL
A bind operation (MS_BIND) was requested where source referred a mount namespace magic link (i.e., a /proc/pid/ns/mnt magic link or a bind mount to such a link) and the propagation type of the parent mount of target was MS_SHARED, but propagation of the requested bind mount could lead to a circular dependency that might prevent the mount namespace from ever being freed.
EINVAL
An attempt was made to bind mount an unbindable mount.

Failed pivot dance:
https://www.linkedin.com/pulse/nullfs-empty-filesystem-reshapes-container-security-vishnu-santhosh-46bzc

GrandDixence2 · October 5, 2026, 10:03pm

try bwrap --bind / / bash directly on the Leap 16 system (outside of apptainer).

Better and more secure:

# bwrap --unshare-ipc --unshare-pid --unshare-net --unshare-user --disable-userns --cap-drop ALL --new-session --clearenv --die-with-parent \
  --ro-bind /usr/bin/ /usr/bin --ro-bind /lib64/ /lib64 --ro-bind /usr/lib64/ /usr/lib64 --ro-bind /usr/sbin/ /usr/sbin \
  --proc /proc --dev /dev --remount-ro /dev --remount-ro /proc --remount-ro /dev/random --remount-ro /dev/urandom \
  --tmpfs /tmp --tmpfs /var/tmp --perms 0700 --tmpfs /root --remount-ro / /usr/bin/bash

Compare:

# mount

deano_ferrari · October 5, 2026, 10:07pm

Sorry not following this discussion now. My initial suggestion was only for a quick diagnostic test at the time. Feel free to engage with the OP if you want to offer any further insights.