Hi. This system was upgraded from Leap 15.1 to 15.2 about 4-5 weeks ago. The upgrade itself seemed to go smoothly and I had rebooted the system a number of times since then following updates.
Yesterday, things went sideways. A lot of updates were pushed by the OpenSuSE update repos this week, some of which set the needs-reboot flag. I think I saw a systemd update, a udev update, maybe a new kernel and several others that each would have set the needs-reboot flag. I couldn’t afford to reboot at the time so I deferred rebooting until Saturday.
Saturday I performed the reboot and was left with a machine that hung midway through the kernel bring-up. The keyboard is unresponsive and I’m almost immediately presented with a splash screen with 3 green question marks. Eventually I’m returned to an emergency shell prompt but, again, the keyboard is unresponsive so I can’t log in. I do recall seeing a message indicating that systemd-journal failed to start. (this becomes important below). Even ALT-SYSRQ-B is unresponsive. My only option seems to be to power-off.
Here’s the thing: Since this machine was originally a Leap 15.1 machine, the grub menu still contains a 4.12.14 kernel entry that dates back to 15.1. **This old kernel boots successfully. **However, since journald never started during the failed boot, there’s no log information saved anywhere from the failed boot to guess what’s going wrong.
On a hunch that maybe the 5.3.x kernel itself was broken, I installed the 5.8.x kernel from Kernel:stable repository. This is where I made a mistake: the 5.8 kernel yields the same mid-boot hang with a dead keyboard. But my zypp.conf was the opensuse default version which means it only kept 3 kernels which means the act of installing 5.8 caused zypper to delete the 4.12 kernel that DID boot. ****.
So I gave up and re-imaged my hard drive using a bitwise copy that I’d made just prior to performing the 15.1–>15.2 upgrade. Booted back into 15.1 and performed the upgrade to 15.2 again. This, again, left me with a kernel that hung midway through the boot if I selected the 5.3 kernel from Grub. I suppose this should be expected since the latest-available packages were installed during the upgrade to 15.2. So ostensibly, at this point, the new 15.2 system’s packages matched the old broken 15.2 system’s packages and the boot behavior also matched.
Once again, though, the old 4.12.14 kernel left over from 15.1 boots fine. This time I changed zypp.conf to preserve that kernel!
So I’m left scratching my head. 5.3 booted fine until updates were installed this week. The fact that multiple kernels (5.3 from the official repo and 5.8 from the Kernel:stable repo) hang, it’s probably not a kernel problem. Since 4.12.14 boots okay and appears to be completely usable, it’s probably not a hardware problem.
That suggests that there’s a problem inside the initrd images. But what? I tried manually reverting systemd, udev and dracut to 31.4.1 which rebuilt the initrd images but the failed boot behavior on 5.3/5.8 persists. I was unable to install earlier versions…when I tried to install systemd- or udev-30.x.x, yast complained that systemd-mini and udev-mini had unresolved dependencies (even though those packages aren’t installed). Since I didn’t understand the nature of that error (why would I get dependency errors for packages that aren’t even installed?) I didn’t force-install the older versions out of fear of rendering the system in even worse state.
Throughout this, 4.12.14 remains rock solid. This is curious because its initrd image should have been rebuilt alongside the others when I reverted systemd, etc. So if there’s problem with the contents of the initrds or with the way dracut is constructing initrd, I would have expected this to break the 4.12.14 kernel. But no, it seems fine.
I’m open to suggestions about how to proceed. The inability to log into the emergency shell during the failed boot means that I have very little information about what went wrong.