Updating kernel on upgraded 16.0: Missing bash in initrd?

The system originates from a Leap 15.5 installation, upgraded to 15.6 without issues and then to 16.0.

After updating the kernel (with zypper up on 16.0) now, booting does not work and I need to rollback with snapper.
The error messages show on boot are:

Failed to start Virtual Console Setup

Entering the emergency console gives:

sulogin: /bin/bash cannot be executed: Permission denied
sulogin: /bin/sh cannot be executed: Permission denied

and the console exists.

The system also has a NVIDIA GeForce MX350 which has a driver from openSUSE’s nvidica repository. Uninstalling all packages from that repo resulted in a similar boot issue:

Failed to start User Manager for UID 1000
Failed to start User Manager for UID 463

Running

dracut -f --regenerate-all

after the installation of the new kernel also leads to the same problem.

To me it seems like dracut is missing some parts?

Okay, the failure of the Virtual Console Setup was unrelated. I replaced KEYMAP=de-latin1-nodeadkeys with KEYMAP=de-nodeadkeys in /etc/vconsole.conf and the service works now.

When starting normally with the NVIDIA drivers, without the emergency shell, the error messages are:

Failed to start NVIDIA Persistence Daemon.
Failed to start NVIDIA Persistence Daemon.
Failed to start NVIDIA Persistence Daemon.
Failed to start NVIDIA Persistence Daemon.
User Manager for UID 1000.
Failed to start NVIDIA Persistence Daemon.
User Manager for UID 1000.

with all NVIDIA drivers uninstalled, it’s just the User Manager failures.

Getting confused. Maybe you better make this one Solved and start a new one with a title that expresses your problem.

Well, I don’t know the cause of the problem is.

Just updating the kernel leads to these error messages:

Failed to start NVIDIA Persistence Daemon.
Failed to start NVIDIA Persistence Daemon.
Failed to start NVIDIA Persistence Daemon.
Failed to start NVIDIA Persistence Daemon.
User Manager for UID 1000.
Failed to start NVIDIA Persistence Daemon.
User Manager for UID 463.

and the boot stops.

removing the NVIDIA drivers before updating the kernels, results in just

User Manager for UID 1000.
User Manager for UID 463.

The error messages from the emergency shell could indicate the cause, but I’m not sure on that.

In the first post, I wrote that the boot screen also showed the error message Failed to start Virtual Console Setup. As I can’t change that post, I made a new one stating that this was unrelated.

@sebix So your using the G06 series for that GPU, switch to a tty, log in as root user. Then systemctl isolate multi-user.target then force the install of the G06 kmp so it rebuilds, if no errors, reboot and should be ok again.

As logging in in that system after the update is not possible, I reverted to the last snapshot and then exexuted the re-installation of the package. Unfortunately I didn’t start a capture of the output, so it is truncated a bit:

re-installation
$ sudo zypper in -f nvidia-driver-G06-kmp-default 2>&1 | tee /run/media/zeit/SEBIUSB31/kmpreinstall.txt
...
/usr/src/kernel-modules/nvidia-580.159.03-default/nvidia.o: warning: objtool: _nv048678rm+0x13: 'naked' return found in MITIGATION_RETHUNK build
/usr/src/kernel-modules/nvidia-580.159.03-default/nvidia.o: warning: objtool: _nv048688rm+0x48: 'naked' return found in MITIGATION_RETHUNK build
...
/usr/src/kernel-modules/nvidia-580.159.03-default/nvidia.o: warning: objtool: _nv014301rm+0x5d: 'naked' return found in MITIGATION_RETHUNK build
/usr/src/kernel-modules/nvidia-580.159.03-default/nvidia.o: warning: objtool: _nv048865rm+0x12f: 'naked' return found in MITIGATION_RETHUNK build
  MODPOST /usr/src/kernel-modules/nvidia-580.159.03-default/Module.symvers
  CC [M]  /usr/src/kernel-modules/nvidia-580.159.03-default/nvidia.mod.o
  CC [M]  /usr/src/kernel-modules/nvidia-580.159.03-default/.module-common.o
  CC [M]  /usr/src/kernel-modules/nvidia-580.159.03-default/nvidia-uvm.mod.o
  CC [M]  /usr/src/kernel-modules/nvidia-580.159.03-default/nvidia-modeset.mod.o
  CC [M]  /usr/src/kernel-modules/nvidia-580.159.03-default/nvidia-drm.mod.o
  CC [M]  /usr/src/kernel-modules/nvidia-580.159.03-default/nvidia-peermem.mod.o
  LD [M]  /usr/src/kernel-modules/nvidia-580.159.03-default/nvidia-modeset.ko
  BTF [M] /usr/src/kernel-modules/nvidia-580.159.03-default/nvidia-modeset.ko
Skipping BTF generation for /usr/src/kernel-modules/nvidia-580.159.03-default/nvidia-modeset.ko due to unavailability of vmlinux
  LD [M]  /usr/src/kernel-modules/nvidia-580.159.03-default/nvidia-peermem.ko
  LD [M]  /usr/src/kernel-modules/nvidia-580.159.03-default/nvidia.ko
  LD [M]  /usr/src/kernel-modules/nvidia-580.159.03-default/nvidia-drm.ko
  LD [M]  /usr/src/kernel-modules/nvidia-580.159.03-default/nvidia-uvm.ko
  BTF [M] /usr/src/kernel-modules/nvidia-580.159.03-default/nvidia-peermem.ko
Skipping BTF generation for /usr/src/kernel-modules/nvidia-580.159.03-default/nvidia-peermem.ko due to unavailability of vmlinux
  BTF [M] /usr/src/kernel-modules/nvidia-580.159.03-default/nvidia-drm.ko
Skipping BTF generation for /usr/src/kernel-modules/nvidia-580.159.03-default/nvidia-drm.ko due to unavailability of vmlinux
  BTF [M] /usr/src/kernel-modules/nvidia-580.159.03-default/nvidia-uvm.ko
Skipping BTF generation for /usr/src/kernel-modules/nvidia-580.159.03-default/nvidia-uvm.ko due to unavailability of vmlinux
  BTF [M] /usr/src/kernel-modules/nvidia-580.159.03-default/nvidia.ko
Skipping BTF generation for /usr/src/kernel-modules/nvidia-580.159.03-default/nvidia.ko due to unavailability of vmlinux
make[2]: Verzeichnis „/usr/src/linux-6.12.0-160000.28-obj/x86_64/default“ wird verlassen
make[1]: Leaving directory '/usr/src/linux-6.12.0-160000.28'
ld.bfd  -T /usr/src/linux-obj/x86_64/default/scripts/module.lds -r -o nv-linux.o \
  nvidia.mod.o nvidia/nv-interface.o
/
rm -f -r conftest
make[1]: Entering directory '/usr/src/linux-6.12.0-160000.28'
make[2]: Verzeichnis „/usr/src/linux-6.12.0-160000.28-obj/x86_64/default“ wird betreten
  CLEAN   /usr/src/kernel-modules/nvidia-580.159.03-default
  CLEAN   /usr/src/kernel-modules/nvidia-580.159.03-default/Module.symvers
make[2]: Verzeichnis „/usr/src/linux-6.12.0-160000.28-obj/x86_64/default“ wird verlassen
make[1]: Leaving directory '/usr/src/linux-6.12.0-160000.28'
/
SKIP: /usr/share/nvidia-pubkeys/MOK-nvidia-driver-G06-580.159.03-lp160.48.1-default.der.delete is not in MokList
.......+...+++++++++++++++++++++++++++++++++++++++*.+.+.....+.......+........+...+.+......+..+...+...+.........+......+.......+..+.......+.....+.+..+++++++++++++++++++++++++++++++++++++++*.........+........+...+......+......+.......+..+..........+...+..+...+...+....+......+.........+...+..................+........+.......+..+................+.........+..+...+.....................+....+........+...+.+.....+...+................+..+.........+..........+...+..+.+..............+.+..+.......+........+.+.....+.+...+.....+...+...+......+.+..............+...+............+.............+......++++++
...+++++++++++++++++++++++++++++++++++++++*.+...+++++++++++++++++++++++++++++++++++++++*..+..+.+..+...+............+.+.........+...+..+......+.......+...+...........+.+...+..............+......+.+.....+....+..+...+.......+.................+.............+..+.....................+.........+.+.........+..+...+.......+......+.....+...+.........+.+..+....+.........+..+...+..........+..+...+......+.+...+......+.....++++++
-----
(1/1) Installieren: nvidia-driver-G06-kmp-default-580.159.03_k6.12.0_160000.28-lp160.48.1.x86_64 .....................................................................................[fertig]
Ausgabe des Skripts %posttrans(nvidia-driver-G06-kmp-default-580.159.03_k6.12.0_160000.28-lp160.48.1.x86_64):
/usr/bin/ldd: line 159: /lib/ld-linux.so.2: cannot execute binary file: Exec format error
/usr/bin/ldd: line 159: /lib/ld-linux.so.2: cannot execute binary file: Exec format error
/usr/bin/ldd: line 159: /lib/ld-linux.so.2: cannot execute binary file: Exec format error
/usr/bin/ldd: line 159: /lib/ld-linux.so.2: cannot execute binary file: Exec format error
Nach-Transaktions-Skripte werden ausgeführt ..........................................................................................................................................[fertig]

After a reboot, the system was still working!

Then I tried the zypper up again to see if that resolved the error, but unfortunately not. It shows again just

Failed to start NVIDIA Persistence Daemon.
Failed to start NVIDIA Persistence Daemon.
Failed to start NVIDIA Persistence Daemon.
Failed to start NVIDIA Persistence Daemon.
Failed to start User Manager for UID 1000.
Failed to start NVIDIA Persistence Daemon.
Failed to start User Manager for UID 463.

@sebix You need to make sure all the nvidia packages are the same versions, did you check this?

These are the NVIDIA-related packages:

i  | kernel-firmware-nvidia                     | Paket   | 20250516-160000.2.2                         | noarch | repo-oss (16.0)
i+ | libnvidia-egl-gbm1                         | Paket   | 1.1.2-160000.3.2                            | x86_64 | repo-oss (16.0)
i+ | libnvidia-egl-wayland1                     | Paket   | 1.1.22-160000.1.1                           | x86_64 | repo-oss (16.0)
i+ | libnvidia-egl-x111                         | Paket   | 1.0.5-160000.1.1                            | x86_64 | repo-oss (16.0)
i+ | libnvidia-gpucomp-G06                      | Paket   | 580.159.03-lp160.48.1                       | x86_64 | repo-non-free (16.0)
i+ | libnvidia-gpucomp-G06-32bit                | Paket   | 580.159.03-lp160.48.1                       | x86_64 | repo-non-free (16.0)
i+ | nvidia-common-G06                          | Paket   | 580.159.03-lp160.48.1                       | x86_64 | repo-non-free (16.0)
i+ | nvidia-compute-G06                         | Paket   | 580.159.03-lp160.48.1                       | x86_64 | repo-non-free (16.0)
i+ | nvidia-compute-G06-32bit                   | Paket   | 580.159.03-lp160.48.1                       | x86_64 | repo-non-free (16.0)
i+ | nvidia-compute-utils-G06                   | Paket   | 580.159.03-lp160.48.1                       | x86_64 | repo-non-free (16.0)
i+ | nvidia-driver-G06-kmp-default              | Paket   | 580.159.03_k6.12.0_160000.28-lp160.48.1     | x86_64 | repo-non-free (16.0)
i+ | nvidia-driver-G06-kmp-meta                 | Paket   | 580.159.03-lp160.48.1                       | x86_64 | repo-non-free (16.0)
i+ | nvidia-gl-G06                              | Paket   | 580.159.03-lp160.48.1                       | x86_64 | repo-non-free (16.0)
i+ | nvidia-libXNVCtrl                          | Paket   | 580.95.05-lp160.45.1                        | x86_64 | repo-non-free (16.0)
i+ | nvidia-modprobe                            | Paket   | 580.159.03-lp160.26.1                       | x86_64 | repo-non-free (16.0)
i+ | nvidia-persistenced                        | Paket   | 580.159.03-lp160.2.1                        | x86_64 | repo-non-free (16.0)
i+ | nvidia-userspace-meta-G06                  | Paket   | 580.159.03-lp160.48.1                       | x86_64 | repo-non-free (16.0)
i+ | nvidia-video-G06                           | Paket   | 580.159.03-lp160.48.1                       | x86_64 | repo-non-free (16.0)
i+ | nvidia-xconfig                             | Paket   | 595.71.05-lp160.2.1                         | x86_64 | repo-non-free (16.0)

and the driver packages are all on 580.159.03-lp160.48.1 as far as I recognize

@sebix nvidia-xconfig isn’t, should be 580.159.03-lp160.2.1