Stuck in grub shell, can't find files not in a subvolume

English Install/Boot/Login
1

When I power on my computer I end up in the grub shell. This happened suddenly, I have not had this problem previously in the year or so I have been running this system.

I have tried to follow various guides to boot from grub but I run into the problem of not finding the files I expect. Particularly, I expect to find vmlinuz with grub> ls (hd3,gpt2)/@/boot/ but instead I only find the grub2 directory. It contains the directories x86_64-efi and i386-pc. These directories are subvolumes in my btrfs system.

Here is the full list I get when running grub> ls with their filesystems

(proc)                    procfs
(hd0)
(hd0, gpt4)               ntfs      win10 disk, C:/
(hd0, gpt3)               unknown
(hd0, gpt2)               fat       win10 boot
(hd0, gpt1)               ntfs      win10 recovery partition
(hd1)
(hd1, msdos1)             ntfs      D:/
(hd2)
(hd2, gpt2)               ntfs      E:/
(hd2, gpt1)               unknown
(hd3)
(hd3, gpt3)               Linux swap
(hd3, gpt2)               btrfs     openSUSE disk
(hd3, gpt1)               fat
(memdisk)

For completeness, I should mention that the win10 installation works as expected if I change the bootorder in my BIOS.
I have also tried to access the Linux disk by booting from a live USB like the openSUSE rescue system. I have not had any problems doing this and I have been able to confirm that the expected content of /boot is present, as listed here

lrwxrwxrwx 1 root root       48 Mar 19 21:48 .vmlinuz-6.7.9-1-default.hmac -> ../usr/lib/modules/6.7.9-1-default/.vmlinuz.hmac
lrwxrwxrwx 1 root root       48 Mar 22 21:43 .vmlinuz-6.8.1-1-default.hmac -> ../usr/lib/modules/6.8.1-1-default/.vmlinuz.hmac
lrwxrwxrwx 1 root root       45 Mar 19 21:48 System.map-6.7.9-1-default -> ../usr/lib/modules/6.7.9-1-default/System.map
lrwxrwxrwx 1 root root       45 Mar 22 21:43 System.map-6.8.1-1-default -> ../usr/lib/modules/6.8.1-1-default/System.map
lrwxrwxrwx 1 root root       41 Mar 19 21:48 config-6.7.9-1-default -> ../usr/lib/modules/6.7.9-1-default/config
lrwxrwxrwx 1 root root       41 Mar 22 21:43 config-6.8.1-1-default -> ../usr/lib/modules/6.8.1-1-default/config
drwxr-xr-x 4 root root     4096 Jan  1  1970 efi
drwxr-xr-x 1 root root       98 Mar 24 13:25 grub2
lrwxrwxrwx 1 root root       22 Mar 23 15:59 initrd -> initrd-6.8.1-1-default
-rw------- 1 root root 50284706 Mar 23 15:59 initrd-6.7.9-1-default
-rw------- 1 root root 50412534 Mar 23 15:59 initrd-6.8.1-1-default
lrwxrwxrwx 1 root root       46 Mar 19 21:48 sysctl.conf-6.7.9-1-default -> ../usr/lib/modules/6.7.9-1-default/sysctl.conf
lrwxrwxrwx 1 root root       46 Mar 22 21:43 sysctl.conf-6.8.1-1-default -> ../usr/lib/modules/6.8.1-1-default/sysctl.conf
lrwxrwxrwx 1 root root       23 Mar 23 15:59 vmlinuz -> vmlinuz-6.8.1-1-default
lrwxrwxrwx 1 root root       42 Mar 19 21:48 vmlinuz-6.7.9-1-default -> ../usr/lib/modules/6.7.9-1-default/vmlinuz
lrwxrwxrwx 1 root root       42 Mar 22 21:43 vmlinuz-6.8.1-1-default -> ../usr/lib/modules/6.8.1-1-default/vmlinuz

Apparently, when I am in the grub shell, I can not see files that are not included in a btrfs subvolume.

Here is the output of btrfs subvolume list /

ID 256 gen 32 top level 5 path @
ID 257 gen 310939 top level 256 path @/var
ID 258 gen 310787 top level 256 path @/usr/local
ID 259 gen 310896 top level 256 path @/tmp
ID 260 gen 308322 top level 256 path @/srv
ID 261 gen 309523 top level 256 path @/root
ID 262 gen 308322 top level 256 path @/opt
ID 263 gen 310937 top level 256 path @/home
ID 264 gen 310938 top level 256 path @/boot/grub2/x86_64-efi
ID 265 gen 308322 top level 256 path @/boot/grub2/i386-pc
ID 266 gen 309550 top level 256 path @/.snapshots
ID 815 gen 138196 top level 266 path @/.snapshots/372/snapshot
ID 818 gen 138196 top level 266 path @/.snapshots/375/snapshot
ID 1104 gen 178878 top level 266 path @/.snapshots/492/snapshot
ID 1115 gen 178878 top level 266 path @/.snapshots/502/snapshot
ID 1434 gen 178878 top level 266 path @/.snapshots/626/snapshot
ID 1770 gen 290155 top level 266 path @/.snapshots/684/snapshot
ID 1771 gen 290159 top level 266 path @/.snapshots/685/snapshot
ID 1772 gen 293877 top level 266 path @/.snapshots/686/snapshot
ID 1773 gen 294379 top level 266 path @/.snapshots/687/snapshot
ID 1774 gen 294387 top level 266 path @/.snapshots/688/snapshot
ID 1775 gen 294391 top level 266 path @/.snapshots/689/snapshot
ID 1848 gen 308452 top level 266 path @/.snapshots/762/snapshot
ID 1850 gen 308308 top level 266 path @/.snapshots/763/snapshot
ID 1866 gen 308451 top level 266 path @/.snapshots/779/snapshot
ID 1867 gen 310937 top level 266 path @/.snapshots/780/snapshot
ID 1868 gen 308456 top level 266 path @/.snapshots/781/snapshot
ID 1869 gen 308459 top level 266 path @/.snapshots/782/snapshot
ID 1870 gen 308460 top level 266 path @/.snapshots/783/snapshot
ID 1871 gen 308461 top level 266 path @/.snapshots/784/snapshot
ID 1872 gen 308473 top level 266 path @/.snapshots/785/snapshot
ID 1873 gen 308476 top level 266 path @/.snapshots/786/snapshot
ID 1874 gen 308478 top level 266 path @/.snapshots/787/snapshot
ID 1875 gen 308494 top level 266 path @/.snapshots/788/snapshot

Back in the grub shell, if I run grub> btrfs-list-subvols (hd3,gpt2) I get

ID 256 path @

and if I run grub> btrfs-get-default-subvol (hd3,gpt2) I get

/@/.snapshots/780/snapshot

which seems strange to me.

What do I need to do to be able to boot from grub again?

2

Show

lsblk -f -o +partuuid
efibootmgr
ls -l /boot/efi/EFI/opensuse

(where /boot/efi refers to whatever partition is mounted in your openSUSE instance).

3

For clarity, my procedure here is that I boot from a live USB with Tumbleweed on it and pick the “rescue system” option. When I am logged in I run the mount-rootfs-and-do-chroot command with my rootfs as the argument (/dev/nvme0n1p2).

Here are the results of

 # lsblk -f -o +partuuid
NAME        FSTYPE FSVER LABEL UUID FSAVAIL FSUSE% MOUNTPOINTS PARTUUID
loop0
loop1
loop2
loop3
loop4
loop5
sda
├─sda1
├─sda2
├─sda3
└─sda4
sdb
└─sdb1
sdc
├─sdc1
└─sdc2
sdd
├─sdd1
│ └─ventoy
└─sdd2
zram0
nvme0n1
├─nvme0n1p1
├─nvme0n1p2                            1.5T    18% /
└─nvme0n1p3

sda is my windows disk, sdb and sdc are other ntfs disks, sdd is the USB I am booting from, nvme0n1 is my linux disk;

 # efibootmgr
BootCurrent: 0008
Timeout: 1 seconds
BootOrder: 0003,0006,0000,0001,0007,0008,0009
Boot0000* Windows Boot Manager	HD(2,GPT,5e85e831-e57f-4a96-be97-366b3ce2b017,0xfa000,0x32000)/File(\EFI\MICROSOFT\BOOT\BOOTMGFW.EFI)57494e444f5753000100000088000000780000004200430044004f0042004a004500430054003d007b00390064006500610038003600320063002d0035006300640064002d0034006500370030002d0061006300630031002d006600330032006200330034003400640034003700390035007d0000002e000100000010000000040000007fff0400
Boot0001* Hard Drive	BBS(HD,,0x0)0000474f00004e4fad0000000100000073004b0049004e004700530054004f004e00200053004e005600320053003200300030003000470000000501090002000000007fff040002010c00d041030a0000000001010600041b01010600000003171000010000000026b76863d007457fff040001043600ef47642dc93ba041ac194d51d01b4ce6350030003000320036004200370036003800360033004400300030003700340000007fff04000000424f00004e4fb5000000010000006f00530061006d00730075006e006700200053005300440020003800350030002000450056004f0020003200350030004700420000000501090002000000007fff040002010c00d041030a0000000001010600001703120a000000ffff00007fff040001043e00ef47642dc93ba041ac194d51d01b4ce6320053003600520058004e004800300030003700330036003600330020004100200020002000200000007fff04000000424f00004e4fa9000000010000006f005700440043002000570044003100300045004100560053002d0030003000440037004200300000000501090002000000007fff040002010c00d041030a0000000001010600001703120a000200ffff00007fff040001043e00ef47642dc93ba041ac194d51d01b4ce62000200020002000570020002d00440043005700550041003000340036003100320036003700330000007fff04000000424f00004e4fab000000010000006f0048004700530054002000480044004e0037003200340030003300300041004c00450036003400300000000501090002000000007fff040002010c00d041030a0000000001010600001703120a000300ffff00007fff040001043e00ef47642dc93ba041ac194d51d01b4ce62000200020002000200020004b00500032003200340033003900500031004b003700330059004d0000007fff04000000424f
Boot0003* opensuse-secureboot	HD(1,GPT,980554b7-02e3-4b59-ad78-bcb1177b2742,0x800,0x100000)/File(\EFI\OPENSUSE\SHIM.EFI)
Boot0006* opensuse	HD(1,GPT,980554b7-02e3-4b59-ad78-bcb1177b2742,0x800,0x100000)/File(\EFI\OPENSUSE\GRUBX64.EFI)0000424f
Boot0007* UEFI: KingstonDataTraveler 3.0PMAP, Partition 1	PciRoot(0x0)/Pci(0x14,0x0)/USB(22,0)/HD(1,MBR,0x9e5d4308,0x800,0x7345800)0000424f
Boot0008* UEFI: KingstonDataTraveler 3.0PMAP, Partition 2	PciRoot(0x0)/Pci(0x14,0x0)/USB(22,0)/HD(2,MBR,0x9e5d4308,0x7346000,0x10000)0000424f
Boot0009* USB	BBS(HD,,0x0)0000474f00004e4fbf0000000100000073004b0069006e006700730074006f006e004400610074006100540072006100760065006c0065007200200033002e00300050004d004100500000000501090002000000007fff040002010c00d041030a000000000101060000140305060016007fff040001044600ef47642dc93ba041ac194d51d01b4ce634004300450044004600420037003400410033003500420045003700410030004600390031003200300037003800390000007fff04000000424f

and my /boot/efi directory is empty.

This is obviously not right so I check what I have mounted

 # mount
/dev/nvme0n1p2 on / type btrfs (rw,relatime,ssd,discard=async,space_cache,subvolid=1867,subvol=/@/.snapshots/780/snapshot)
devtmpfs on /dev type devtmpfs (rw,relatime,size=8019764k,nr_inodes=2004941,mode=755,inode64)
devpts on /dev/pts type devpts (rw,relatime,gid=5,mode=620,ptmxmode=000)
proc on /proc type proc (rw,relatime)
sysfs on /sys type sysfs (rw,relatime)
efivarfs on /sys/firmware/efi/efivars type efivarfs (rw,nosuid,nodev,noexec,relatime)

Since I suspect things in subvolumes not being accessible is related to the issues I run mount -a and try everything again. Now this is what I have mounted;

 # mount
/dev/nvme0n1p2 on / type btrfs (rw,relatime,ssd,discard=async,space_cache,subvolid=1867,subvol=/@/.snapshots/780/snapshot)
devtmpfs on /dev type devtmpfs (rw,relatime,size=8019764k,nr_inodes=2004941,mode=755,inode64)
devpts on /dev/pts type devpts (rw,relatime,gid=5,mode=620,ptmxmode=000)
proc on /proc type proc (rw,relatime)
sysfs on /sys type sysfs (rw,relatime)
efivarfs on /sys/firmware/efi/efivars type efivarfs (rw,nosuid,nodev,noexec,relatime)
/dev/nvme0n1p2 on /var type btrfs (rw,relatime,ssd,discard=async,space_cache,subvolid=257,subvol=/@/var)
/dev/nvme0n1p2 on /usr/local type btrfs (rw,relatime,ssd,discard=async,space_cache,subvolid=258,subvol=/@/usr/local)
/dev/nvme0n1p2 on /tmp type btrfs (rw,relatime,ssd,discard=async,space_cache,subvolid=259,subvol=/@/tmp)
/dev/nvme0n1p2 on /srv type btrfs (rw,relatime,ssd,discard=async,space_cache,subvolid=260,subvol=/@/srv)
/dev/nvme0n1p2 on /root type btrfs (rw,relatime,ssd,discard=async,space_cache,subvolid=261,subvol=/@/root)
/dev/nvme0n1p2 on /opt type btrfs (rw,relatime,ssd,discard=async,space_cache,subvolid=262,subvol=/@/opt)
/dev/nvme0n1p2 on /home type btrfs (rw,relatime,ssd,discard=async,space_cache,subvolid=263,subvol=/@/home)
/dev/nvme0n1p2 on /boot/grub2/x86_64-efi type btrfs (rw,relatime,ssd,discard=async,space_cache,subvolid=264,subvol=/@/boot/grub2/x86_64-efi)
/dev/nvme0n1p2 on /boot/grub2/i386-pc type btrfs (rw,relatime,ssd,discard=async,space_cache,subvolid=265,subvol=/@/boot/grub2/i386-pc)
/dev/nvme0n1p2 on /.snapshots type btrfs (rw,relatime,ssd,discard=async,space_cache,subvolid=266,subvol=/@/.snapshots)
/dev/nvme0n1p1 on /boot/efi type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,utf8,errors=remount-ro)

and

 # lsblk -f -o +partuuid
NAME        FSTYPE FSVER LABEL UUID FSAVAIL FSUSE% MOUNTPOINTS            PARTUUID
loop0
loop1
loop2
loop3
loop4
loop5
sda
├─sda1
├─sda2
├─sda3
└─sda4
sdb
└─sdb1
sdc
├─sdc1
└─sdc2
sdd
├─sdd1
│ └─ventoy
└─sdd2
zram0
nvme0n1
├─nvme0n1p1                          505.1M     1% /boot/efi
├─nvme0n1p2                            1.5T    18% /.snapshots
│                                                  /boot/grub2/i386-pc
│                                                  /boot/grub2/x86_64-efi
│                                                  /home
│                                                  /opt
│                                                  /root
│                                                  /srv
│                                                  /tmp
│                                                  /usr/local
│                                                  /var
│                                                  /
└─nvme0n1p3

the return from efibootmgr is unchanged but now /boot/efi/EFI/opensuse contains

 # ls -l /boot/efi/EFI/opensuse/
total 4112
-rwxr-xr-x 1 root root  846096 Mar 31 12:43 MokManager.efi
-rwxr-xr-x 1 root root      58 Mar 31 12:43 boot.csv
-rwxr-xr-x 1 root root     151 Mar 31 12:43 grub.cfg
-rwxr-xr-x 1 root root 2078576 Mar 31 12:43 grub.efi
-rwxr-xr-x 1 root root  335872 Mar 31 12:43 grubx64.efi
-rwxr-xr-x 1 root root  934024 Mar 31 12:43 shim.efi
4

lsblk reads udev database in /run and the script mount-rootfs-and-do-chroot does not bind mount /run. Theoretically lsblk should fall back to reading devices directly, not sure, why it does not happen … hmm, man page says “if the udev db is not available”, but as far as I can tell in this case lsblk gets empty database and so never attempts direct probing. You do not need to run it in chroot. Or you can manually bind mount /run before running this script (hopefully script will not unmount it).

You may consider bug report for live image (script). Unless there are good reasons not to, it probably should bind mount /run as well.

Unfortunately we cannot verify which partition it is, but otherwise it looks right.

Show the content of this file (assuming it is the correct partition).

5

Here is that file.

 # cat /boot/efi/EFI/opensuse/grub.cfg
search --fs-uuid --set=root 624bb25f-78d8-4053-aa5c-54144f08e546
set prefix=(${root})/@/.snapshots/673/snapshot/boot/grub2
source "${prefix}/grub.cfg"

I find this strange. Snapshot 673 does not exist so relying on it seems very bad. I also can’t find a device with uuid 624bb25f-78d8-4053-aa5c-54144f08e546, I checked my physical devices with fdisk -l and my subvolumes with btrfs subvolumes list -u / but none matched. I assume it was snapshot 673 that had that uuid.

I am unfamiliar with what this file does, but it seems to want to read grub.cfg but it is pointing at one in a deleted snapshot instead of the “real” one at /boot/grub2/.

Do I edit this file or is there a smarter way of dealing with this?

6

It looks like this file was created with support for snapshots disabled. Post

grep -v '^$' -v '^#' /etc/default/grub

where /etc/default/grub refers to the file on your root filesystem.

7

Here is the output from that command (I slightly modified your command to remove empty lines),

 # grep -ve '^$' -e '^#' /etc/default/grub
GRUB_DISTRIBUTOR=
GRUB_DEFAULT=saved
GRUB_HIDDEN_TIMEOUT=0
GRUB_HIDDEN_TIMEOUT_QUIET=true
GRUB_TIMEOUT=8
GRUB_CMDLINE_LINUX_DEFAULT="splash=silent preempt=full mitigations=auto quiet security=apparmor nosimplefb=1"
GRUB_CMDLINE_LINUX=""
GRUB_TERMINAL="gfxterm"
GRUB_GFXMODE="auto"
GRUB_BACKGROUND=
GRUB_THEME=/boot/grub2/themes/openSUSE/theme.txt
SUSE_BTRFS_SNAPSHOT_BOOTING="true"
GRUB_USE_LINUXEFI="true"
GRUB_DISABLE_OS_PROBER="false"
GRUB_ENABLE_CRYPTODISK="n"
GRUB_CMDLINE_XEN_DEFAULT="vga=gfx-1024x768x16"
8

Update!
I asked a friend who runs Tumbleweed and copied their /boot/efi/EFI/opensuse/grub.cfg (with the UUID found in /etc/fstab) so it is now

 # cat /boot/efi/EFI/opensuse/grub.cfg
set btrfs_relative_path="yes"
search --fs-uuid --set=root 624bb25f-78d8-4053-aa5c-54144f08e546
set prefix=(${root})/boot/grub2
source "${prefix}/grub.cfg"

and after doing this change (and nothing else) my system boots as normal.

Interestingly, this is the UUID that I couldn’t find before. Apparently, it was correct all along.

9

After several reboots and a system update, this seems stable so I will mark this as solved.

10

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.