Greetings Community,
a couple of days ago I upgraded my server from openSUSE leap 15.5 to openSUSE Leap 15.6 and I broke the system partially. I can only boot on the 5.14 Kernel not the 6.4 Kernel shipped by Leap 15.6.
I choose the offline Upgrade method via the .iso file.
A bit to the setup
for my boot-drives I use two 128GB Sata SSDs in btrfs raid-1 it’s not a md-raid but native btrfs raid 1.
/boot/efi is ext4 and only in drive sdg
/boot is an unecrypted btrfs raid 1 on sdg and sdh
/ is an encryped (luks2) btrfs raid 1 on sdg and sdh
besides some snapshots there are no btrfs subvolumes.
The upgrade
For the offline upgrade I first decrypted the / root partition on the btrfs raid 1 Then the upgrade process suggests you a list of partitions with Operating Systems to upgrade.
My openSUSE Leap 15.5 install did not appear in this list. Only when I checked the checkbox below the list, to show all partitions, than my system appeared. not just once but twice. Once no /dev at all and one on /dev/mapper/cr_root.
As chose the latter option on the crypttab-mapper device. This was probably the wrong choice.
Than they upgrade process reported some package conflicts which surprisingly downgraded some packages like libgphoto2-6-2.5.30 downgraded to libgphoto2-6-2.27. I don’t even know why this package is installed on an headless-server install but whatever. I allowed the installer to “upgrade” any conflicting packages as I considered non to be essential.
After the system upgrade I got some dracut warnings like systemd-coredump not being installed but I get them too when I simply run zypper up.
Else the upgrade was rather smooth.
First boot
After reboot I then noticed the issue with the with the Kernel 6.4 not being able to boot. grub couldn’t find file system root.
I reset the system and booted on the older 5.14 Kernel.
The first thing I looked at was of course the grub.cfg file and the structure if the /boot partition.
.rw-r--r-- 255k root 23 May 11:38 config-5.14.21-150500.55.65-default
.rw-r--r-- 273k root 18 May 21:29 config-6.4.0-150600.21-default
.rw-r--r-- 0 root 17 Jun 09:05 do_purge_kernels
drwxr-xr-x - root 1 Jan 1970 efi
drwxr-xr-x - root 17 Jun 09:05 grub2
lrwxrwxrwx - root 17 Jun 09:05 initrd -> initrd-6.4.0-150600.21-default
.rw------- 55M root 15 Jun 13:47 initrd-5.14.21-150500.55.65-default
.rw-r--r-- 1.5M root 17 May 2023 symtypes-5.14.21-150500.53-default.gz
.rw-r--r-- 1.5M root 23 May 11:55 symtypes-5.14.21-150500.55.65-default.gz
.rw-r--r-- 1.6M root 18 May 21:53 symtypes-6.4.0-150600.21-default.gz
.rw-r--r-- 471k root 23 May 11:54 symvers-5.14.21-150500.55.65-default.gz
.rw-r--r-- 363k root 18 May 21:52 symvers-6.4.0-150600.21-default.gz
.rw-r--r-- 484 root 23 May 11:54 sysctl.conf-5.14.21-150500.55.65-default
.rw-r--r-- 484 root 18 May 21:52 sysctl.conf-6.4.0-150600.21-default
.rw-r--r-- 5.1M root 23 May 11:52 System.map-5.14.21-150500.55.65-default
.rw-r--r-- 7.7M root 18 May 21:48 System.map-6.4.0-150600.21-default
.rw-r--r-- 13M root 23 May 11:57 vmlinux-5.14.21-150500.55.65-default.gz
.rw-r--r-- 16M root 18 May 21:56 vmlinux-6.4.0-150600.21-default.gz
lrwxrwxrwx - root 17 Jun 09:05 vmlinuz -> vmlinuz-6.4.0-150600.21-default
.rw-r--r-- 12M root 23 May 12:22 vmlinuz-5.14.21-150500.55.65-default
.rw-r--r-- 14M root 18 May 22:28 vmlinuz-6.4.0-150600.21-default
First thing I noticed was the link
lrwxrwxrwx - root 17 Jun 09:05 initrd -> initrd-6.4.0-150600.21-default
was broken. There isn’t an initrd-6.4.0-150600.21-default file in /boot
The
vmlinuz -> vmlinuz-6.4.0-150600.21-default
link is fine a vmlinuz-6.4.0-150600.21-default file exists.
In the grub.cfg I noticed that the entries between the the 5.14 Kernel and the 6.4 Kernel are different in two (I presume) crucial lines.
For the 6.4 Kernel I saw following entry
### BEGIN /etc/grub.d/10_linux ###
menuentry 'openSUSE Leap 15.6' --class opensuse --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-simple-e89b00e9-b03c-4334-ac39-33f34533747a' {
load_video
set gfxpayload=keep
insmod gzio
insmod part_gpt
insmod part_gpt
insmod btrfs
set root='hd6,gpt2'
if [ x$feature_platform_search_hint = xy ]; then
search --no-floppy --fs-uuid --set=root --hint-bios=hd6,gpt2 --hint-efi=hd6,gpt2 --hint-baremetal=ahci6,gpt2 --hint-bios=hd7,gpt1 --hint-efi=hd7,gpt1 --hint-baremetal=ahci7,gpt1 cdc54ea5-5ed4-4cf9-9480-28c9c9d9249f
else
search --no-floppy --fs-uuid --set=root cdc54ea5-5ed4-4cf9-9480-28c9c9d9249f
fi
echo 'Loading Linux 6.4.0-150600.21-default ...'
linux /vmlinuz-6.4.0-150600.21-default root=/dev/mapper/cr_root
/dev/mapper/cr_root_2 ${extra_cmdline} splash=silent preempt=full mitigations=auto quiet security=apparmor nosimplefb=1
}
For the 5.14 Kernel
menuentry 'openSUSE Leap 15.6, with Linux 5.14.21-150500.55.65-default' --class opensuse --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-5.14.21-150500.55.65-default-advanced-e89b00e9-b03c-4334-ac39-33f34533747a' {
load_video
set gfxpayload=keep
insmod gzio
insmod part_gpt
insmod part_gpt
insmod btrfs
set root='hd6,gpt2'
if [ x$feature_platform_search_hint = xy ]; then
search --no-floppy --fs-uuid --set=root --hint-bios=hd6,gpt2 --hint-efi=hd6,gpt2 --hint-baremetal=ahci6,gpt2 --hint-bios=hd7,gpt1 --hint-efi=hd7,gpt1 --hint-baremetal=ahci7,gpt1 cdc54ea5-5ed4-4cf9-9480-28c9c9d9249f
else
search --no-floppy --fs-uuid --set=root cdc54ea5-5ed4-4cf9-9480-28c9c9d9249f
fi
echo 'Loading Linux 5.14.21-150500.55.65-default ...'
linux /vmlinuz-5.14.21-150500.55.65-default root=UUID=e89b00e9-b03c-4334-ac39-33f34533747a ${extra_cmdline} splash=silent preempt=full mitigations=auto quiet security=apparmor nosimplefb=1
echo 'Loading initial ramdisk ...'
initrd /initrd-5.14.21-150500.55.65-default
}
The difference is in the 6.4 Kernel
echo 'Loading Linux 6.4.0-150600.21-default ...'
linux /vmlinuz-6.4.0-150600.21-default root=/dev/mapper/cr_root
/dev/mapper/cr_root_2 ...
grub tries to set the root to /dev/mapper/cr_root. Followed by a newline. Than /dev/mapper/cr_root_2 followed by the parameters.
On the 5.14 Kernel it looks like that.
echo 'Loading Linux 5.14.21-150500.55.65-default ...'
linux /vmlinuz-5.14.21-150500.55.65-default root=UUID=e89b00e9-b03c-4334-ac39-33f34533747a # parameters removed not relavant
echo 'Loading initial ramdisk ...'
initrd /initrd-5.14.21-150500.55.65-default
Here the root is set to UUID=e89b00e9-b03c-4334-ac39-33f34533747a which is the UUID for /dev/dm-0.
In addition there is the initrd line.
What did I try?
I tried reinstalling the Kernel package with sudo zypper in -f kernel-default-6.4.0-150600.21.2
and regenerating the grub config afterwards with sudo grub2-mkconfig -o /boot/grub2/grub.cfg
sudo grub2-mkconfig -o /boot/grub2/grub.cfg outputs following:
Generating grub configuration file ...
Found theme: /boot/grub2/themes/openSUSE/theme.txt
Found linux image: /boot/vmlinuz-6.4.0-150600.21-default
Found linux image: /boot/vmlinuz-5.14.21-150500.55.65-default
Found initrd image: /boot/initrd-5.14.21-150500.55.65-default
Warning: os-prober will be executed to detect other bootable partitions.
Its output will be used to detect bootable binaries on them and create new boot entries.
338165.311478 | DM multipath kernel driver not loaded
Found openSUSE Leap 15.6 on /dev/mapper/cr_root_2
Adding boot menu entry for UEFI Firmware Settings ...
done
When I understand the output correctly, grub2-mkconfig finds openSUSE Leap 15.6 as a second bootable system (dual-booting) on /dev/mapper/cr_root_2
Help wanted
Any suggestions how to first get the initrd-6.4.0-150600.21-default one Idea I had was to spin up a VM and copy the file to the server via scp.
But this won’t really fix the issue. I still have the invalid grub.cfg with the wrong root partition. I do not like to disobey the warning to not manually edit the /boot/grub2/grub.cfg to fix it. Because as soon as another kernel is in the repos and I download it, it would potentially regenerate the broken grub.cfg.
I could also just reinstall the system, at the moment it olny runs a minecraft server. But than I have to download all the packages over my slow internet.
Thanks for any help in advance.
Regards TheHolypumpkin.
PS: Answers can be provided in English and German.