KVM: Cannot create Virtual Machine (libvirt error - cannot delete directory '/run/libvirt/qemu/2-Test.shm': Device or resource busy)

Leap 15.5 with KVM, DRBD, Pacemaker installed. When trying to create a virtual machine, whether from virt-manager or virsh create w/XML, I receive the follow error condition:

In the messages below, “Test” is the name of the VM…

virt-manager: Unable to complete install: ‘internal error: Process exited prior to exec: libvirt: error : Cannot delete directory ‘/run/libvirt/qemu/2-Test.shm’: Device or resource busy’

virsh create XML: error: internal error: Process exited prior to exec: libvirt: error : Cannot delete directory ‘/run/libvirt/qemu/2-Test.shm’: Device or resource busy

The hangup seems to be something in /run/libvirt/qemu. I have checked the permissions on the content, and they match another Leap15.5 system that is working. I suspect that there is either a timing issue (requests out of order) or an obvious configuration issue. I’m just not sure where to look. Below are the relevant files I currently have installed.

qemu packages:

qemu-ui-opengl-7.1.0-150500.49.15.1.x86_64
qemu-ui-curses-7.1.0-150500.49.15.1.x86_64
qemu-microvm-7.1.0-150500.49.15.1.noarch
qemu-ui-gtk-7.1.0-150500.49.15.1.x86_64
qemu-block-curl-7.1.0-150500.49.15.1.x86_64
qemu-skiboot-7.1.0-150500.49.15.1.noarch
qemu-hw-display-virtio-vga-7.1.0-150500.49.15.1.x86_64
qemu-hw-usb-host-7.1.0-150500.49.15.1.x86_64
system-user-qemu-20170617-150400.24.2.1.noarch
qemu-block-nfs-7.1.0-150500.49.15.1.x86_64
qemu-chardev-baum-7.1.0-150500.49.15.1.x86_64
qemu-ipxe-1.0.0±150500.49.15.1.noarch
qemu-audio-spice-7.1.0-150500.49.15.1.x86_64
qemu-ui-spice-app-7.1.0-150500.49.15.1.x86_64
qemu-hw-usb-redirect-7.1.0-150500.49.15.1.x86_64
qemu-ksm-7.1.0-150500.49.15.1.x86_64
qemu-sgabios-8-150500.49.15.1.noarch
qemu-tools-7.1.0-150500.49.15.1.x86_64
qemu-hw-display-qxl-7.1.0-150500.49.15.1.x86_64
qemu-hw-s390x-virtio-gpu-ccw-7.1.0-150500.49.15.1.x86_64
qemu-accel-tcg-x86-7.1.0-150500.49.15.1.x86_64
libvirt-daemon-qemu-9.0.0-150500.6.23.1.x86_64
qemu-block-rbd-7.1.0-150500.49.15.1.x86_64
qemu-seabios-1.16.0_0_gd239552-150500.49.15.1.noarch
qemu-hw-display-virtio-gpu-7.1.0-150500.49.15.1.x86_64
qemu-vgabios-1.16.0_0_gd239552-150500.49.15.1.noarch
qemu-ui-spice-core-7.1.0-150500.49.15.1.x86_64
qemu-chardev-spice-7.1.0-150500.49.15.1.x86_64
qemu-hw-usb-smartcard-7.1.0-150500.49.15.1.x86_64
qemu-7.1.0-150500.49.15.1.x86_64
qemu-block-ssh-7.1.0-150500.49.15.1.x86_64
libvirt-daemon-driver-qemu-9.0.0-150500.6.23.1.x86_64
qemu-hw-display-virtio-gpu-pci-7.1.0-150500.49.15.1.x86_64
qemu-ovmf-x86_64-202208-150500.4.1.noarch
qemu-x86-7.1.0-150500.49.15.1.x86_64
qemu-ivshmem-tools-7.1.0-150500.49.15.1.x86_64

libvirt packages:

libvirt-daemon-driver-storage-logical-9.0.0-150500.6.23.1.x86_64
libvirt-glib-1_0-0-4.0.0-150400.1.10.x86_64
libvirt-daemon-driver-storage-scsi-9.0.0-150500.6.23.1.x86_64
libvirt-daemon-driver-network-9.0.0-150500.6.23.1.x86_64
libvirt-daemon-config-network-9.0.0-150500.6.23.1.x86_64
system-group-libvirt-20170617-150400.24.2.1.noarch
libvirt-daemon-9.0.0-150500.6.23.1.x86_64
libvirt-daemon-driver-storage-gluster-9.0.0-150500.6.23.1.x86_64
python3-libvirt-python-9.0.0-150500.1.4.x86_64
libvirt-daemon-driver-secret-9.0.0-150500.6.23.1.x86_64
libvirt-9.0.0-150500.6.23.1.x86_64
libvirt-daemon-driver-nwfilter-9.0.0-150500.6.23.1.x86_64
libvirt-libs-9.0.0-150500.6.23.1.x86_64
libvirt-daemon-driver-storage-iscsi-9.0.0-150500.6.23.1.x86_64
libvirt-daemon-driver-storage-mpath-9.0.0-150500.6.23.1.x86_64
libvirt-daemon-driver-nodedev-9.0.0-150500.6.23.1.x86_64
libvirt-daemon-driver-storage-9.0.0-150500.6.23.1.x86_64
libvirt-daemon-driver-libxl-9.0.0-150500.6.23.1.x86_64
libvirt-daemon-config-nwfilter-9.0.0-150500.6.23.1.x86_64
libvirt-daemon-driver-storage-rbd-9.0.0-150500.6.23.1.x86_64
libvirt-daemon-driver-storage-core-9.0.0-150500.6.23.1.x86_64
libvirt-daemon-qemu-9.0.0-150500.6.23.1.x86_64
libvirt-daemon-driver-interface-9.0.0-150500.6.23.1.x86_64
libvirt-daemon-driver-storage-disk-9.0.0-150500.6.23.1.x86_64
libvirt-daemon-driver-storage-iscsi-direct-9.0.0-150500.6.23.1.x86_64
libvirt-daemon-driver-qemu-9.0.0-150500.6.23.1.x86_64
libvirt-daemon-driver-lxc-9.0.0-150500.6.23.1.x86_64
libvirt-client-9.0.0-150500.6.23.1.x86_64

KVM:

system-group-kvm-20170617-150400.24.2.1.noarch
patterns-server-kvm_tools-20180302-lp155.2.4.x86_64
kvm_stat-5.14.21-150400.22.5.noarch
patterns-server-kvm_server-20180302-lp155.2.4.x86_64

Just a quick update: I’m still struggling with making any headway on this. I’ve looked in the libvirt log files, but they’re not giving me any detail on the failure. The --debug feature on virsh is not showing anything definitive either. Looking for specifics on where I can look that will help to shed some light on what is going on. Any suggestions will be appreciated.

I attached an strace to libvirtd, and at least found why it’s happening. Here’s a chunk of the relevant strace output. The unlink is failing because /dev/shm is mounted there at the time the unlink is called, and the filesystem remains mounted because the unmount call fails. There’s a lazy umount /dev in there which is a little confusing as well, but maybe we overlook that for now. :sweat_smile:. It’s curious that it’s trying to move the mount instead of doing a bind/rbind, but I’m not a deep enough expert in what libvirt is doing there to say it’s wrong. :slight_smile:

I haven’t gotten far enough in to see why the mount succeeds while the umount fails, or why the filesystem is ultimately unmounted after the process exits. But I thought this much was interesting, and wanted to share that it’s not just one person seeing this. I suspect apparmor, but need to prove that. Leap 15.6, read-only root, this happens on all formerly working VMs, etc.

12979 mount("/dev/shm", "/run/libvirt/qemu/2-cerberus.shm", 0x7f9e7093cf1b, MS_MOVE, NULL <unfinished ...>
12979 <... mount resumed>)              = 0
...
12979 unlink("/run/libvirt/qemu/2-cerberus.pts/3") = -1 EPERM (Operation not per
mitted)
12979 close(3)                          = 0
12979 umount2("/run/libvirt/qemu/2-cerberus.shm", 0) = -1 EACCES (Permission denied)
12979 newfstatat(AT_FDCWD, "/run/libvirt/qemu/2-cerberus.shm", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=40, ...}, 0) = 0
12979 access("/run/libvirt/qemu/2-cerberus.shm", F_OK) = 0
12979 openat(AT_FDCWD, "/run/libvirt/qemu/2-cerberus.shm", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 3
12979 fstat(3, {st_mode=S_IFDIR|S_ISVTX|0777, st_size=40, ...}) = 0
12979 getdents64(3, 0x7f9e4c09c080 /* 2 entries */, 32768) = 48
12979 getdents64(3, 0x7f9e4c09c080 /* 0 entries */, 32768) = 0
12979 rmdir("/run/libvirt/qemu/2-cerberus.shm") = -1 EBUSY (Device or resource busy)
12979 close(3)                          = 0
12979 write(2, "libvirt:  error : Cannot delete "..., 102) = 102
12979 exit_group(-1)                    = ?
1 Like

Yeah, there we go. The apparmor rules are missing a umount corresponding to the final mount rule. Here’s the relevant snippet from /etc/apparmor.d/usr.sbin.libvirtd.

  mount options=(rw,rslave)  -> /,
  mount options=(rw, nosuid) -> /{var/,}run/libvirt/qemu/*.dev/,
  umount /{var/,}run/libvirt/qemu/*.dev/,
  umount /dev/,

  # libvirt provides any mounts under /dev to qemu namespaces
  mount options=(rw, move) /dev/ -> /{,var/}run/libvirt/qemu/*.dev/,
  mount options=(rw, move) /dev/** -> /{,var/}run/libvirt/qemu/*{,/},
  mount options=(rw, move) /{,var/}run/libvirt/qemu/*.dev/ -> /dev/,
  mount options=(rw, move) /{,var/}run/libvirt/qemu/*{,/} -> /dev/**,

Adding a line to the umount section like this resolves the issue and allows VMs to start again.

  umount /{var/,}run/libvirt/qemu/*{,/},

That file needs some other cleanup, like making where the “or nothing” comma goes consistent at the beginning or end of the braces. I guess there’s probably a proper place to open a bug report for this. Off to figure out which OBS package is the “real” upstream.

1 Like

My OpenSUSE Bugzilla account seems to have been disabled (I formerly worked for Suse, and several of my open source accounts broke when I left), so fixing that to report a bug will take some time. Digging around in the libvirt package source, though, it looks like that apparmor profile might actually come from upstream anyway. Specifically, at https://github.com/openSUSE/libvirt/blob/master/src/security/apparmor/usr.sbin.libvirtd.in, which is a direct mirror of upstream. So I guess this is a libvirt bug.

1 Like

Thanks for validating my sanity! :rofl:

Thanks, Danny! I’ll try out the snippet as a quick test. As for the bug report, do you need anything from me to help with the submission?

If it works for you too, you’re welcome to open a suse bug report referencing the admittedly quick merge request I just made at Allow unmounting some things libvirt mounted (!375) · Merge requests · libvirt / libvirt · GitLab. Since this is on Leap 15.5 and still present in 15.6, I think the bug report is supposed to be opened against SLES. I’ll open one up after I figure out the Bugzilla account issue if you don’t get around to it. :wink:

@dannysauer: Thanks so much for showing me where to look. Just as a reminder, I have a two node Pacemaker/DRBD/KVM cluster, both built at the same time, starting with Leap 15.3… I found something VERY interesting… you’re gonna love this!

Both nodes have been upgraded to Leap 15.5, using the off-line method as recommended, from 15.3 to 15.4 to 15.5; this was done over time, upgrading other systems first to determine stability of each of the platforms before upgrading this cluster. Prior to 15.5, each node would successfully start and maintain the guests, as part of the natural fail-over process that I’ve configured. With the 15.5 upgrade, “Node 1”, let’s call it, still successfully mounted VM’s. Node 2, on the other hand, would no longer do that.

Looking at the /etc/apparmor.d/usr.sbin.libvirtd file you mentioned, I have rather drastic differences. Here is the content of the working node:

#include <tunables/global>
@{LIBVIRT}="libvirt"

profile libvirtd /usr/sbin/libvirtd flags=(attach_disconnected) {
  #include <abstractions/base>
  #include <abstractions/dbus>

  capability kill,
  capability net_admin,
  capability net_raw,
  capability setgid,
  capability sys_admin,
  capability sys_module,
  capability sys_ptrace,
  capability sys_pacct,
  capability sys_nice,
  capability sys_chroot,
  capability setuid,
  capability dac_override,
  capability dac_read_search,
  capability fowner,
  capability chown,
  capability setpcap,
  capability mknod,
  capability fsetid,
  capability audit_write,
  capability ipc_lock,
  capability sys_rawio,
  capability bpf,
  capability perfmon,

  # Needed for vfio
  capability sys_resource,

  mount options=(rw,rslave)  -> /,
  mount options=(rw, nosuid) -> /{var/,}run/libvirt/qemu/*.dev/,
  umount /{var/,}run/libvirt/qemu/*.dev/,
  umount /dev/,

  # libvirt provides any mounts under /dev to qemu namespaces
  mount options=(rw, move) /dev/ -> /{,var/}run/libvirt/qemu/*.dev/,
  mount options=(rw, move) /dev/** -> /{,var/}run/libvirt/qemu/*{,/},
  mount options=(rw, move) /{,var/}run/libvirt/qemu/*.dev/ -> /dev/,
  mount options=(rw, move) /{,var/}run/libvirt/qemu/*{,/} -> /dev/**,

  network inet stream,
  network inet dgram,
  network inet6 stream,
  network inet6 dgram,
  network netlink raw,
  network packet dgram,
  network packet raw,

  # for --p2p migrations
  unix (send, receive) type=stream addr=none peer=(label=unconfined addr=none),

  ptrace (read,trace) peer=unconfined,
  ptrace (read,trace) peer=@{profile_name},
  ptrace (read,trace) peer=dnsmasq,
  ptrace (read,trace) peer=/usr/sbin/dnsmasq,
  ptrace (read,trace) peer=libvirt-*,
  ptrace (read,trace) peer=swtpm,

  signal (send) peer=dnsmasq,
  signal (send) peer=/usr/sbin/dnsmasq,
  signal (read, send) peer=libvirt-*,
  signal (send) set=("kill", "term") peer=unconfined,

  # For communication/control to qemu-bridge-helper
  unix (send, receive) type=stream addr=none peer=(label=libvirtd//qemu_bridge_helper),
  signal (send) set=("term") peer=libvirtd//qemu_bridge_helper,

  # allow connect with openGraphicsFD, direction reversed in newer versions
  unix (send, receive) type=stream addr=none peer=(label=libvirt-[0-9a-f]*-[0-9a-f]*-[0-9a-f]*-[0-9a-f]*-[0-9a-f]*),
  # unconfined also required if guests run without security module
  unix (send, receive) type=stream addr=none peer=(label=unconfined),

  # required if guests run unconfined seclabel type='none' but libvirtd is confined
  signal (read, send) peer=unconfined,

  # Very lenient profile for libvirtd since we want to first focus on confining
  # the guests. Guests will have a very restricted profile.
  / r,
  /** rwmkl,

  /bin/* PUx,
  /sbin/* PUx,
  /usr/bin/* PUx,
  /usr/sbin/virtlogd pix,
  /usr/sbin/* PUx,
  /{usr/,}lib/udev/scsi_id PUx,
  /usr/{lib,lib64}/xen-common/bin/xen-toolstack PUx,
  /usr/{lib,lib64,libexec}/xen/bin/* Ux,
  /usr/{lib,libexec}/xen-*/bin/libxl-save-helper PUx,
  /usr/{lib,libexec}/xen-*/bin/pygrub PUx,
  /usr/{lib,lib64,lib/qemu,libexec}/vhost-user-gpu PUx,
  /usr/{lib,lib64,lib/qemu,libexec}/virtiofsd PUx,

  # Required by nwfilter_ebiptables_driver.c:ebiptablesWriteToTempFile() to
  # read and run an ebtables script.
  /var/lib/libvirt/virtd* ixr,

  # force the use of virt-aa-helper
  audit deny /{usr/,}sbin/apparmor_parser rwxl,
  audit deny /etc/apparmor.d/libvirt/** wxl,
  audit deny /sys/kernel/security/apparmor/features rwxl,
  audit deny /sys/kernel/security/apparmor/matching rwxl,
  audit deny /sys/kernel/security/apparmor/.* rwxl,
  /sys/kernel/security/apparmor/profiles r,
  /usr/lib64/libvirt/* PUxr,
  /usr/lib64/libvirt/libvirt_parthelper ix,
  /usr/lib64/libvirt/libvirt_iohelper ix,
  /etc/libvirt/hooks/** rmix,
  /etc/xen/scripts/** rmix,

  # allow changing to our UUID-based named profiles
  change_profile -> @{LIBVIRT}-[0-9a-f]*-[0-9a-f]*-[0-9a-f]*-[0-9a-f]*-[0-9a-f]*,

  /usr/{lib,lib64,lib/qemu,libexec}/qemu-bridge-helper Cx -> qemu_bridge_helper,
  # child profile for bridge helper process
  profile qemu_bridge_helper {
   #include <abstractions/base>

   capability setuid,
   capability setgid,
   capability setpcap,
   capability net_admin,

   network inet stream,

   # For communication/control from libvirtd
   unix (send, receive) type=stream addr=none peer=(label=libvirtd),
   signal (receive) set=("term") peer=/usr/sbin/libvirtd,
   signal (receive) set=("term") peer=libvirtd,

   /dev/net/tun rw,
   /etc/qemu/** r,
   owner @{PROC}/*/status r,

   /usr/{lib,lib64,lib/qemu,libexec}/qemu-bridge-helper rmix,
  }

  # Site-specific additions and overrides. See local/README for details.
  include if exists <local/usr.sbin.libvirtd>
}

Notice that the mount and umount commands match what you pointed out, and yet this unit works! Very strange… The date stamp of this file is 5/28/24; the upgrade was performed 6/7/24, so the creation date matches that.

Now, the failing node looks like this:

# Last Modified: Mon Apr  5 15:03:58 2010
#include <tunables/global>
@{LIBVIRT}="libvirt"

profile libvirtd /usr/sbin/libvirtd flags=(attach_disconnected) {
  #include <abstractions/base>
  #include <abstractions/dbus>

  capability kill,
  capability net_admin,
  capability net_raw,
  capability setgid,
  capability sys_admin,
  capability sys_module,
  capability sys_ptrace,
  capability sys_pacct,
  capability sys_nice,
  capability sys_chroot,
  capability setuid,
  capability dac_override,
  capability dac_read_search,
  capability fowner,
  capability chown,
  capability setpcap,
  capability mknod,
  capability fsetid,
  capability audit_write,
  capability ipc_lock,

  # Needed for vfio
  capability sys_resource,

  mount options=(rw,rslave)  -> /,
  mount options=(rw, nosuid) -> /{var/,}run/libvirt/qemu/*.dev/,

  # libvirt provides any mounts under /dev to qemu namespaces
  mount options=(rw, move) /dev/ -> /{,var/}run/libvirt/qemu/*.dev/,
  mount options=(rw, move) /dev/** -> /{,var/}run/libvirt/qemu/*{,/},
  mount options=(rw, move) /{,var/}run/libvirt/qemu/*.dev/ -> /dev/,
  mount options=(rw, move) /{,var/}run/libvirt/qemu/*{,/} -> /dev/**,

  network inet stream,
  network inet dgram,
  network inet6 stream,
  network inet6 dgram,
  network netlink raw,
  network packet dgram,
  network packet raw,

  # for --p2p migrations
  unix (send, receive) type=stream addr=none peer=(label=unconfined addr=none),

  ptrace (read,trace) peer=unconfined,
  ptrace (read,trace) peer=@{profile_name},
  ptrace (read,trace) peer=dnsmasq,
  ptrace (read,trace) peer=/usr/sbin/dnsmasq,
  ptrace (read,trace) peer=libvirt-*,

  signal (send) peer=dnsmasq,
  signal (send) peer=/usr/sbin/dnsmasq,
  signal (read, send) peer=libvirt-*,
  signal (send) set=("kill", "term") peer=unconfined,

  # For communication/control to qemu-bridge-helper
  unix (send, receive) type=stream addr=none peer=(label=/usr/sbin/libvirtd//qemu_bridge_helper),
  signal (send) set=("term") peer=/usr/sbin/libvirtd//qemu_bridge_helper,

  # allow connect with openGraphicsFD, direction reversed in newer versions
  unix (send, receive) type=stream addr=none peer=(label=libvirt-[0-9a-f]*-[0-9a-f]*-[0-9a-f]*-[0-9a-f]*-[0-9a-f]*),
  # unconfined also required if guests run without security module
  unix (send, receive) type=stream addr=none peer=(label=unconfined),

  # required if guests run unconfined seclabel type='none' but libvirtd is confined
  signal (read, send) peer=unconfined,

  # Very lenient profile for libvirtd since we want to first focus on confining
  # the guests. Guests will have a very restricted profile.
  / r,
  /** rwmkl,

  /bin/* PUx,
  /sbin/* PUx,
  /usr/bin/* PUx,
  /usr/sbin/virtlogd pix,
  /usr/sbin/* PUx,
  /{usr/,}lib/udev/scsi_id PUx,
  /usr/{lib,lib64}/xen-common/bin/xen-toolstack PUx,
  /usr/{lib,lib64}/xen/bin/* Ux,
  /usr/lib/xen-*/bin/libxl-save-helper PUx,

  # Required by nwfilter_ebiptables_driver.c:ebiptablesWriteToTempFile() to
  # read and run an ebtables script.
  /var/lib/libvirt/virtd* ixr,

  # force the use of virt-aa-helper
  audit deny /{usr/,}sbin/apparmor_parser rwxl,
  audit deny /etc/apparmor.d/libvirt/** wxl,
  audit deny /sys/kernel/security/apparmor/features rwxl,
  audit deny /sys/kernel/security/apparmor/matching rwxl,
  audit deny /sys/kernel/security/apparmor/.* rwxl,
  /sys/kernel/security/apparmor/profiles r,
  /usr/{lib,lib64}/libvirt/* PUxr,
  /usr/{lib,lib64}/libvirt/libvirt_parthelper ix,
  /usr/{lib,lib64}/libvirt/libvirt_iohelper ix,
  /etc/libvirt/hooks/** rmix,
  /etc/xen/scripts/** rmix,

  # allow changing to our UUID-based named profiles
  change_profile -> @{LIBVIRT}-[0-9a-f]*-[0-9a-f]*-[0-9a-f]*-[0-9a-f]*-[0-9a-f]*,

  /usr/{lib,lib64,lib/qemu,libexec}/qemu-bridge-helper Cx -> qemu_bridge_helper,
  # child profile for bridge helper process
  profile qemu_bridge_helper {
   #include <abstractions/base>

   capability setuid,
   capability setgid,
   capability setpcap,
   capability net_admin,

   network inet stream,

   # For communication/control from libvirtd
   unix (send, receive) type=stream addr=none peer=(label=/usr/sbin/libvirtd),
   signal (receive) set=("term") peer=/usr/sbin/libvirtd,
   signal (receive) set=("term") peer=libvirtd,

   /dev/net/tun rw,
   /etc/qemu/** r,
   owner @{PROC}/*/status r,

   /usr/{lib,lib64,lib/qemu,libexec}/qemu-bridge-helper rmix,
  }
}

Notice that there are NO umount commands! There are other subtle differences as well, references to 32-bit mode vs. 64-bit mode, and some other components are missing that were introduced over time by the developers (my educated guess). The date stamp of this file is 10/29/2021. The creation date is 3/29/21, obviously from an older release of Leap. I plan to copy the file to the failing node to see what happens. Of course, I have to call into question what other files may be out of date that have produced this environment, and potentially rebuild this node from scratch. I will follow up shortly with my test results.

So when I replaced the /etc/apparmor.d/usr.sbin.libvirtd, apparmor would not load unless I also replaced /etc/apparmor.d/usr.lib.libvirt.virt-aa-helper. The original file was dated the same as libvirtd. I’ve included both files here for reference:

Original:

# Last Modified: Mon Apr  5 15:10:27 2010
#include <tunables/global>

profile virt-aa-helper /usr/{lib,lib64}/libvirt/virt-aa-helper {
  #include <abstractions/base>

  # needed for searching directories
  capability dac_override,
  capability dac_read_search,

  # needed for when disk is on a network filesystem
  network inet,
  network inet6,

  deny @{PROC}/[0-9]*/mounts r,
  @{PROC}/[0-9]*/net/psched r,
  owner @{PROC}/[0-9]*/status r,
  @{PROC}/filesystems r,

  # Used when internally running another command (namely apparmor_parser)
  @{PROC}/@{pid}/fd/ r,

  /etc/libnl*/classid r,

  # for gl enabled graphics
  /dev/dri/{,*} r,

  # for hostdev
  /sys/devices/ r,
  /sys/devices/** r,
  /sys/bus/usb/devices/ r,
  deny /dev/sd* r,
  deny /dev/vd* r,
  deny /dev/dm-* r,
  deny /dev/drbd[0-9]* r,
  deny /dev/dasd* r,
  deny /dev/nvme* r,
  deny /dev/zd[0-9]* r,
  deny /dev/mapper/ r,
  deny /dev/mapper/* r,

  /usr/{lib,lib64}/libvirt/virt-aa-helper mr,
  /{usr/,}sbin/apparmor_parser Ux,

  /etc/apparmor.d/libvirt/* r,
  /etc/apparmor.d/libvirt/libvirt-[0-9a-f]*-[0-9a-f]*-[0-9a-f]*-[0-9a-f]*-[0-9a-f]* rw,

  # for backingstore -- allow access to non-hidden files in @{HOME} as well
  # as storage pools
  audit deny @{HOME}/.* mrwkl,
  audit deny @{HOME}/.*/ rw,
  audit deny @{HOME}/.*/** mrwkl,
  audit deny @{HOME}/bin/ rw,
  audit deny @{HOME}/bin/** mrwkl,
  @{HOME}/ r,
  @{HOME}/** r,
  /var/lib/libvirt/images/ r,
  /var/lib/libvirt/images/** r,
  /var/lib/nova/instances/_base/* r,
  /{media,mnt,opt,srv}/** r,
  # For virt-sandbox
  /{,var/}run/libvirt/**/[sv]d[a-z] r,

  /**.img r,
  /**.raw r,
  /**.qcow{,2} r,
  /**.qed r,
  /**.vmdk r,
  /**.vhd r,
  /**.[iI][sS][oO] r,
  /**/disk{,.*} r,

  #include <local/usr.lib.libvirt.virt-aa-helper>
}

journalctl pointed out line 73 in the above file as the culprit for the failure to start.

Here is the replacement file:

#include <tunables/global>

profile virt-aa-helper /usr/lib64/libvirt/virt-aa-helper {
  #include <abstractions/base>
  #include <abstractions/openssl>

  # needed for searching directories
  capability dac_override,
  capability dac_read_search,

  # needed for when disk is on a network filesystem
  network inet,
  network inet6,

  deny @{PROC}/[0-9]*/mounts r,
  @{PROC}/[0-9]*/net/psched r,
  owner @{PROC}/[0-9]*/status r,
  @{PROC}/filesystems r,

  # Used when internally running another command (namely apparmor_parser)
  @{PROC}/@{pid}/fd/ r,

  # allow reading libnl's classid file
  /etc/libnl{,-3}/classid r,

  # for gl enabled graphics
  /dev/dri/{,*} r,

  # for hostdev
  /sys/devices/ r,
  /sys/devices/** r,
  /sys/bus/usb/devices/ r,
  deny /dev/sd* r,
  deny /dev/vd* r,
  deny /dev/dm-* r,
  deny /dev/drbd[0-9]* r,
  deny /dev/dasd* r,
  deny /dev/nvme* r,
  deny /dev/zd[0-9]* r,
  deny /dev/mapper/ r,
  deny /dev/mapper/* r,

  /usr/lib64/libvirt/virt-aa-helper mr,
  /{usr/,}sbin/apparmor_parser Ux,

  /etc/apparmor.d/libvirt/* r,
  /etc/apparmor.d/libvirt/libvirt-[0-9a-f]*-[0-9a-f]*-[0-9a-f]*-[0-9a-f]*-[0-9a-f]* rw,

  # for backingstore -- allow access to non-hidden files in @{HOME} as well
  # as storage pools
  audit deny @{HOME}/.* mrwkl,
  audit deny @{HOME}/.*/ rw,
  audit deny @{HOME}/.*/** mrwkl,
  audit deny @{HOME}/bin/ rw,
  audit deny @{HOME}/bin/** mrwkl,
  @{HOME}/ r,
  @{HOME}/** r,
  /var/lib/libvirt/images/ r,
  /var/lib/libvirt/images/** r,
  /var/lib/nova/instances/_base/* r,
  /{media,mnt,opt,srv}/** r,
  # For virt-sandbox
  /{,var/}run/libvirt/**/[sv]d[a-z] r,

  /**.img r,
  /**.raw r,
  /**.qcow{,2} r,
  /**.qed r,
  /**.vmdk r,
  /**.vhd r,
  /**.[iI][sS][oO] r,
  /**/disk{,.*} r,

  include if exists <local/usr.lib.libvirt.virt-aa-helper>
}

At this point, I’ll try the fail-over to see if it works.

Sorry for the delayed response. So making those two changes gave me successful results. I’ll leave this post open until the changes @dannysauer has suggested are made for Leap 15.6; I can upload those files to the case if you (Danny) were successful in authenticating and opening it. Please let me know, and I’ll contribute in whatever way needed, including opening the case as well.