Disk not found after Tumbleweed Update

tpat · November 15, 2024, 11:44pm

Hello! I’m having some issues with a tumbleweed upgrade. I’d be very grateful if someone more experienced could help me debug this.

I’m upgrading from the 11/04 image to the 11/14 one. It adds a new kernel version of 6.11.7. Do note I am using systemd-boot, btrfs, and luks encryption.

When I reboot from the update, I’m not prompted for the encryption password. I’m dropped into a dracut emergency shell. It says it failed to locate my root partition, which won’t be visible until the drive is decrypted. I’m still seeing this if I boot it with the previous 6.11.5 kernel.

This is the list of partitions I see before the update. nvme0n1p2 is my luks encrypted drive with tumbleweed

/dev/mapper/system-swap: UUID="e4ff092b-cc74-4164-8271-c866607403a7" TYPE="swap"
/dev/nvme0n1p5: BLOCK_SIZE="512" UUID="B8A857E2A8579DA6" TYPE="ntfs" PARTUUID="20955822-e873-4f6d-a0d7-f772ea848288"
/dev/nvme0n1p3: PARTLABEL="Microsoft reserved partition" PARTUUID="a330c4ec-4106-4e3d-8a60-4fdce5b49d8d"
/dev/nvme0n1p1: UUID="B6B9-1229" BLOCK_SIZE="512" TYPE="vfat" PARTLABEL="EFI" PARTUUID="2fa0b225-0ffc-41d0-b1d8-b5607c0d979b"
/dev/nvme0n1p4: BLOCK_SIZE="512" UUID="464C9CE34C9CCED5" TYPE="ntfs" PARTLABEL="Basic data partition" PARTUUID="66339627-648f-44dd-b4ea-4cbcd09dce7e"
/dev/nvme0n1p2: UUID="780554f2-3be1-419b-9473-bdb4e56d3446" TYPE="crypto_LUKS" PARTUUID="dd7f5396-5555-43aa-86d4-7fa7813de5b8"
/dev/mapper/system-root: UUID="44eccd62-698f-4d82-bf08-ec74a12655ae" UUID_SUB="72db1930-c41e-479e-95f2-4106ff28d2e9" BLOCK_SIZE="4096" TYPE="btrfs"
/dev/mapper/cr_nvme-eui.044a500191c00fbc-part2: UUID="o51KNT-q2PT-d8KH-po3P-FXtf-Minn-aqa2Ej" TYPE="LVM2_member"
/dev/mapper/system-home: UUID="f2480c00-27fe-43ef-b440-f60510589790" BLOCK_SIZE="4096" TYPE="ext4"

Luckily, I’ve used snapper to rollback while I figure out this issue. It’s much less stressful than when I previously used Arch with ext4

arvidjaar · November 16, 2024, 8:00am

Boot with kernel options printk.devkmsg=on log_buf_len=16M and get the file rdsosreport.txt (you can e.g. copy it into ESP which should be available in initrd) and upload to https://paste.opensuse.org/. Also upload the complete output of journalctl -b as root after successful boot with the same kernel options for comparison.

tpat · November 17, 2024, 8:03pm

Thanks for the suggestion.

Here is the rdsoreport
and the journalctl output

This portion seems relevant:

[  135.610847] x1 dracut-initqueue[495]: Warning: dracut-initqueue: timeout, still waiting for following initqueue hooks:
[  135.612981] x1 dracut-initqueue[495]: Warning: /lib/dracut/hooks/initqueue/finished/90-crypt.sh: "[ -e /dev/disk/by-id/dm-uuid-CRYPT-LUKS?-*780554f23be1419b9473bdb4e56d3446*-* ] || exit 1"
[  135.615065] x1 dracut-initqueue[495]: Warning: /lib/dracut/hooks/initqueue/finished/devexists-\x2fdev\x2fdisk\x2fby-uuid\x2f44eccd62-698f-4d82-bf08-ec74a12655ae.sh: "if ! grep -q After=remote-fs-pre.target /run/systemd/generator/systemd-cryptsetup@*.service 2>/dev/null; then
[  135.615065] x1 dracut-initqueue[495]:     [ -e "/dev/disk/by-uuid/44eccd62-698f-4d82-bf08-ec74a12655ae" ]
[  135.615065] x1 dracut-initqueue[495]: fi"
[  135.617134] x1 dracut-initqueue[495]: Warning: /lib/dracut/hooks/initqueue/finished/devexists-\x2fdev\x2fsystem\x2froot.sh: "[ -e "/dev/system/root" ]"
[  135.619026] x1 dracut-initqueue[495]: Warning: /lib/dracut/hooks/initqueue/finished/devexists-\x2fdev\x2fsystem\x2fswap.sh: "[ -e "/dev/system/swap" ]"

arvidjaar · November 18, 2024, 4:03am

This output is from the failed boot, not from the normal boot. I asked for the output from the normal boot for comparison. Output you provided is already contained in rdsosreport.txt.

arvidjaar · November 18, 2024, 4:18am

Also, provide output of the

lsinitrd /boot/initrd-x.y.z

for both “good” and “bad” kernel versions.

arvidjaar · November 18, 2024, 4:30am

Also, when system stops in dracut provide full output of

udevadm info --export-db

yoman · November 18, 2024, 5:36am

Hi

With the information you’ve provided from your logs, let’s try a few commands to verify if your system hasn’t modified the encryption options:

Checking the bootloader configuration:

sudo cat /boot/efi/loader/entries/opensuse-tumbleweed.conf

Make sure the following lines are present and correct:

options root=UUID=44eccd62-698f-4d82-bf08-ec74a12655ae rootflags=subvol=@/.snapshots/463/snapshot rd.luks.uuid=luks-780554f2-3be1-419b-9473-bdb4e56d3446

Try recreating the initramfs:

sudo mkinitrd

Checking the integrity of the LUKS volume:

sudo cryptsetup luksDump /dev/nvme0n1p2

Attempting to manually unlock the LUKS volume:

sudo cryptsetup luksOpen /dev/nvme0n1p2 my_encrypted_volume

Updating the system:

sudo zypper update

hui · November 18, 2024, 8:51am

Please don‘t provide outdated and/or wrong informations.

mkinitrd got deprecated in 2021 and finaly removed 2023 in Tumbleweed. dracut is the replacement.

No. Tumbleweed needs to be upgraded via sudo zypper dup

tpat · November 18, 2024, 7:38pm

Shoot, sorry for the mixup. Here is the journal b output from a successful boot.
https://paste.opensuse.org/pastes/4dc78e794046

For lsinitrd, here is the good kernel one.
https://paste.opensuse.org/pastes/f12dad89848f
In the dracut emergency shell, I can’t run the lsinitrd command.

I was able to run the udevadm command however.
https://paste.opensuse.org/pastes/469842b2674d

arvidjaar · November 19, 2024, 4:11am

You can run this command after booting for any initrd that you have.

tpat · November 19, 2024, 10:00pm

Sorry, I don’t understand. My boot fails and I only have access to the dracut shell, where the lsinitrd program isn’t accessible.

arvidjaar · November 20, 2024, 7:48am

You said you can boot using earlier snapshot and you can access any file from any snapshot in this case.

tpat · November 20, 2024, 7:07pm

Ah okay, I can boot the previous working snapshot, but I’m not sure how to access the initrd of the broken snapshot.

arvidjaar · November 20, 2024, 7:14pm

You said you are using systemd-boot, so all initrd’s are under /boot/efi. The bootloader entries are under /boot/efi/loader/entries, file names include snapshot number as the last component and initrd= line gives initrd path below /boot/efi.

tpat · November 20, 2024, 8:27pm

I see what you mean now. Thanks for you patience btw

https://paste.opensuse.org/pastes/aeabb5627442

tpat · November 29, 2024, 8:35am

So I chose the nuclear option and performed a full disk format and reinstalled with the latest tumbleweed iso. Surprisingly, this did not resolve my error.

I tried a few different install configurations and what seemed to fix it was installing grub instead of systemd-boot.

I guess the combination of systemd-boot with Luks disk encryption was broken from an update. Hopefully someone finds this useful if they’re running into the same issue.

arvidjaar · November 29, 2024, 9:27am

Sorry for delay.

Comparing your two initrds, the initrd for the kernel 6.11.8 includes these extra modules:

systemd-sysusers
dbus-broker
dbus
bluetooth

Try to explicitly disable them.

echo 'omit_dracutmodules+=" systemd-sysusers dbus-broker dbus bluetooth "' > /etc/dracut.conf.d/test.conf
sdbootutil mkinitrd

tpat · November 29, 2024, 4:32pm

Hey, no problem. Thanks for all your help, but i decided to do a full wipe. Im pretty sure now the systemd-boot with luks encryption got broken in an update.