Wont boot and missing snapshots option from boot menu

Been a couple of years since I dealt with a system that fails to boot. However this morning I did a ‘sudo zypper dup’. There was two conflicts related to python libs which I chose option 1 on both (uninstall - I don’t recall them) and the update finished ok. The system now boots to the emergency mode console. Went to boot to a snapshot as usual but the menu option is no longer there.

> cat /etc/os-release

NAME="openSUSE Tumbleweed"
# VERSION="20240922"
ID="opensuse-tumbleweed"
ID_LIKE="opensuse suse"
VERSION_ID="20240922"
PRETTY_NAME="openSUSE Tumbleweed"
ANSI_COLOR="0;32"
# CPE 2.3 format, boo#1217921
CPE_NAME="cpe:2.3:o:opensuse:tumbleweed:20240922:*:*:*:*:*:*:*"
#CPE 2.2 format
#CPE_NAME="cpe:/o:opensuse:tumbleweed:20240922"
BUG_REPORT_URL="https://bugzilla.opensuse.org"
SUPPORT_URL="https://bugs.opensuse.org"
HOME_URL="https://www.opensuse.org"
DOCUMENTATION_URL="https://en.opensuse.org/Portal:Tumbleweed"
LOGO="distributor-logo-Tumbleweed"

Based on Missing snapshots option from boot menu and other posts, I’ve looked around and can’t see anything obviously wrong with files, except for one thing.

> snapper create

Failure (org.freedesktop.DBus.Error.FileNotFound).

Things seem to be mounted ok and all the snapshots are there.

>  cat /etc/fstab

UUID=d8ab3a11-371d-4244-8327-063049454a95  /                        btrfs  defaults                      0  0
UUID=d8ab3a11-371d-4244-8327-063049454a95  /.snapshots              btrfs  subvol=/@/.snapshots          0  0
UUID=9c55cc0a-c911-4aca-a748-33fecbdc820e  swap                     swap   defaults                      0  0
UUID=d8ab3a11-371d-4244-8327-063049454a95  /var                     btrfs  subvol=/@/var                 0  0
UUID=d8ab3a11-371d-4244-8327-063049454a95  /usr/local               btrfs  subvol=/@/usr/local           0  0
UUID=d8ab3a11-371d-4244-8327-063049454a95  /tmp                     btrfs  subvol=/@/tmp                 0  0
UUID=d8ab3a11-371d-4244-8327-063049454a95  /srv                     btrfs  subvol=/@/srv                 0  0
UUID=d8ab3a11-371d-4244-8327-063049454a95  /root                    btrfs  subvol=/@/root                0  0
UUID=d8ab3a11-371d-4244-8327-063049454a95  /opt                     btrfs  subvol=/@/opt                 0  0
UUID=726ede17-0e41-4f90-8e45-ebf5d7bf1f69  /home                    xfs    defaults                      0  0
UUID=d8ab3a11-371d-4244-8327-063049454a95  /boot/grub2/x86_64-efi   btrfs  subvol=/@/boot/grub2/x86_64-efi  0  0
UUID=d8ab3a11-371d-4244-8327-063049454a95  /boot/grub2/i386-pc      btrfs  subvol=/@/boot/grub2/i386-pc  0  0
UUID=7765-E385                             /boot/efi                vfat   defaults                      0  2
UUID=208074BE80749C4A                      /mnt/win                 ntfs   fmask=133,dmask=022           0  0

btrfs subvolume list /

ID 256 gen 2527804 top level 5 path @
ID 257 gen 2805221 top level 256 path @/var
ID 258 gen 2805178 top level 256 path @/usr/local
ID 259 gen 2805213 top level 256 path @/tmp
ID 260 gen 2799376 top level 256 path @/srv
ID 261 gen 2805225 top level 256 path @/root
ID 262 gen 2805130 top level 256 path @/opt
ID 263 gen 2805122 top level 256 path @/boot/grub2/x86_64-efi
ID 264 gen 2588211 top level 256 path @/boot/grub2/i386-pc
ID 265 gen 2805175 top level 256 path @/.snapshots
ID 93104 gen 2805218 top level 265 path @/.snapshots/136/snapshot
ID 93265 gen 2749288 top level 257 path @/var/lib/machines
ID 93295 gen 2796058 top level 265 path @/.snapshots/261/snapshot
ID 93296 gen 2796058 top level 265 path @/.snapshots/262/snapshot
ID 93297 gen 2796058 top level 265 path @/.snapshots/263/snapshot
ID 93298 gen 2796058 top level 265 path @/.snapshots/264/snapshot
ID 93299 gen 2796058 top level 265 path @/.snapshots/265/snapshot
ID 93300 gen 2796058 top level 265 path @/.snapshots/266/snapshot
ID 93302 gen 2796058 top level 265 path @/.snapshots/267/snapshot
ID 93303 gen 2796058 top level 265 path @/.snapshots/268/snapshot
ID 93304 gen 2796058 top level 265 path @/.snapshots/269/snapshot
ID 93305 gen 2796058 top level 265 path @/.snapshots/270/snapshot
ID 93310 gen 2799727 top level 265 path @/.snapshots/271/snapshot
ID 93311 gen 2799738 top level 265 path @/.snapshots/272/snapshot
ID 93312 gen 2804079 top level 265 path @/.snapshots/273/snapshot
ID 93313 gen 2805124 top level 265 path @/.snapshots/274/snapshot
if [ -z "$extra_cmdline" ]; then
  submenu  "Start bootloader from a read-only snapshot" {
    if [ -f "/.snapshots/274/grub-snapshot.cfg" ]; then
      source "/.snapshots/274/grub-snapshot.cfg"
    fi
    if [ -f "/.snapshots/273/grub-snapshot.cfg" ]; then
      source "/.snapshots/273/grub-snapshot.cfg"
    fi
    if [ -f "/.snapshots/272/grub-snapshot.cfg" ]; then
      source "/.snapshots/272/grub-snapshot.cfg"
    fi
    if [ -f "/.snapshots/271/grub-snapshot.cfg" ]; then
      source "/.snapshots/271/grub-snapshot.cfg"
    fi
    if [ -f "/.snapshots/270/grub-snapshot.cfg" ]; then
      source "/.snapshots/270/grub-snapshot.cfg"
    fi
    if [ -f "/.snapshots/269/grub-snapshot.cfg" ]; then
      source "/.snapshots/269/grub-snapshot.cfg"
    fi
    if [ -f "/.snapshots/268/grub-snapshot.cfg" ]; then
      source "/.snapshots/268/grub-snapshot.cfg"
    fi
    if [ -f "/.snapshots/267/grub-snapshot.cfg" ]; then
      source "/.snapshots/267/grub-snapshot.cfg"
    fi
    if [ -f "/.snapshots/266/grub-snapshot.cfg" ]; then
      source "/.snapshots/266/grub-snapshot.cfg"
    fi
    if [ -f "/.snapshots/265/grub-snapshot.cfg" ]; then
      source "/.snapshots/265/grub-snapshot.cfg"
    fi
    if [ -f "/.snapshots/264/grub-snapshot.cfg" ]; then
      source "/.snapshots/264/grub-snapshot.cfg"
    fi
    if [ -f "/.snapshots/263/grub-snapshot.cfg" ]; then
      source "/.snapshots/263/grub-snapshot.cfg"
    fi
    if [ -f "/.snapshots/262/grub-snapshot.cfg" ]; then
      source "/.snapshots/262/grub-snapshot.cfg"
    fi
    if [ -f "/.snapshots/261/grub-snapshot.cfg" ]; then
      source "/.snapshots/261/grub-snapshot.cfg"
    fi
    if [ x$snapshot_found != xtrue ]; then
      submenu "Not Found" { true; }
    fi
  }
fi
> cat etc/default/grub

GRUB_DISTRIBUTOR=
GRUB_DEFAULT=saved
GRUB_HIDDEN_TIMEOUT=0
GRUB_HIDDEN_TIMEOUT_QUIET=true
GRUB_TIMEOUT=8
GRUB_CMDLINE_LINUX_DEFAULT="splash=silent resume=/dev/disk/by-id/nvme-KBG40ZPZ1T02_NVMe_KIOXIA_1024GB_Y9D100U8NTRL-part10 quiet nosgx mitigations=auto"
GRUB_CMDLINE_LINUX=""
GRUB_SAVEDEFAULT="true"
GRUB_TERMINAL="gfxterm"
GRUB_GFXMODE="auto"
GRUB_BACKGROUND=
GRUB_THEME=/boot/grub2/themes/openSUSE/theme.txt
SUSE_BTRFS_SNAPSHOT_BOOTING="true"
GRUB_USE_LINUXEFI="true"
GRUB_DISABLE_OS_PROBER="false"
GRUB_ENABLE_CRYPTODISK="n"
GRUB_CMDLINE_XEN_DEFAULT="vga=gfx-1024x768x16"

What exactly do you mean? There is grub emergency mode, there is dracut emergency mode, there is systemd emergency mode and there is simple plain text mode login which for many users cries emergency as well.

There was nothing wrong in the topic you mentioned either except using a third-party grub theme.

Hey thanks.

What exactly do you mean? There is grub emergency mode, …

Sorry. Here’s the screenshot

However, for booting, it turned out to the ntfs entry in fstab. When I commented it out I was able to boot:
#UUID=208074BE80749C4A /mnt/win ntfs fmask=133,dmask=022 0 0

Seems the windows parition somehow got itself into a partial update state (no idea). When I booted Windows and let it complete, I was able to boot back into TW and uncomment the above line. So all good regarding booting.

Still can’t see the Grub snapshot option though

Start with showing /boot/grub2/grub.cfg, upload to https://paste.opensuse.org/

Done: openSUSE Paste

Arrg! wrong laptop. Let me try again. :face_with_open_eyes_and_hand_over_mouth:

Start with showing /boot/grub2/grub.cfg, upload to https://paste.opensuse.org/

Done properly this time: openSUSE Paste

And

findmnt -u /.snapshots
TARGET      SOURCE                        FSTYPE OPTIONS
/.snapshots /dev/nvme0n1p8[/@/.snapshots] btrfs  rw,relatime,ssd,discard=async,space_cache,subvolid=265,subvol=/@/.snapshots

In case it’s relevant

$ sudo btrfs inspect-internal rootid /
93104

In grub menu go to command line (press c) and do

echo $root
ls ($root)/.snapshots

Does $root match your root partition? Do you see the expected content? In the first place, grub-snapshot.cfg?
image

Even if @/.snapshots is not mounted, it is still wrong. Show

ls ($root)
ls ($root)/

And also after boot

cat /boot/efi/EFI/opensuse/grub.cfg
lsblk -f +o partuuid

Unfortunately SUSE grub2 signed image does not include full ls command, I am not sure how to display filesystems UUID from within grub.

cat /boot/efi/EFI/opensuse/grub.cfg

set btrfs_relative_path="yes"
search --no-floppy --fs-uuid --set=root d8ab3a11-371d-4244-8327-063049454a95
set prefix=(${root})/boot/grub2
source "${prefix}/grub.cfg"

lsblk -f +o partuuid

NAME         FSTYPE   FSVER LABEL       UUID                                 FSAVAIL FSUSE% MOUNTPOINTS                               PARTUUID
loop0        squashfs 4.0                                                          0   100% /snap/authy/23                            
loop1        squashfs 4.0                                                          0   100% /snap/authy/22                            
loop2        squashfs 4.0                                                          0   100% /snap/bare/5                              
loop3        squashfs 4.0                                                          0   100% /snap/core18/2823                         
loop4        squashfs 4.0                                                          0   100% /snap/core18/2829                         
loop5        squashfs 4.0                                                          0   100% /snap/core20/2318                         
loop6        squashfs 4.0                                                          0   100% /snap/core20/2379                         
loop7        squashfs 4.0                                                          0   100% /snap/core22/1586                         
loop8                                                                              0   100% /snap/core22/1612                         
loop9                                                                              0   100% /snap/gnome-3-28-1804/194                 
loop10                                                                             0   100% /snap/gnome-3-28-1804/198                 
loop11                                                                             0   100% /snap/gnome-3-34-1804/90                  
loop12                                                                             0   100% /snap/gnome-3-34-1804/93                  
loop13                                                                             0   100% /snap/gtk-common-themes/1534              
loop14                                                                             0   100% /snap/gtk-common-themes/1535              
loop15                                                                             0   100% /snap/kde-frameworks-5-qt-5-15-3-core20/8 
loop16                                                                             0   100% /snap/snapd/21465                         
loop17                                                                             0   100% /snap/slack/155                           
loop18                                                                             0   100% /snap/slack/158                           
loop19                                                                             0   100% /snap/tradingview/59                      
loop20       squashfs 4.0                                                          0   100% /snap/tradingview/58                      
loop21                                                                             0   100% /snap/teams-for-linux/667                 
loop22                                                                             0   100% /snap/teams-for-linux/671                 
loop23                                                                             0   100% /snap/snapd/21759                         
sda                                                                                                                                   
└─sda1       vfat     FAT32 OPENSUSE-TU 381D-D3D8                               7.3G     3% /mnt/usb                                  003eb6ee-01
nvme0n1                                                                                                                               
├─nvme0n1p1  vfat     FAT32 ESP         9A43-2622                                                                                     1687c016-6515-4b88-870e-0b8295886cb0
├─nvme0n1p2                                                                                                                           b1de8f73-f100-4296-b1f4-bbc06631cd42
├─nvme0n1p3  ntfs           OS          208074BE80749C4A                       78.8G    50% /mnt/win                                  a620a95b-2926-4ddc-bf5d-03e9c9723d92
├─nvme0n1p4  ntfs           WINRETOOLS  A89A33C39A338CB6                                                                              fc19714e-1f34-467e-a9e9-11466d901734
├─nvme0n1p5  ntfs           Image       460C33E10C33CAA7                                                                              a7d8fa14-be96-4a1b-9ad4-12a1af51b977
├─nvme0n1p6  ntfs           DELLSUPPORT 72549D0D549CD4E7                                                                              9ce6610c-37a4-4c18-99e1-da982a868ab8
├─nvme0n1p7  vfat     FAT16             7765-E385                             493.6M     1% /boot/efi                                 00f215cb-7a03-46f0-a649-fcc3c316f95d
├─nvme0n1p8  btrfs                      d8ab3a11-371d-4244-8327-063049454a95   60.8G    37% /var                                      23cf8466-1fa8-4475-aa99-11f2dccf8347
│                                                                                           /usr/local                                
│                                                                                           /tmp                                      
│                                                                                           /opt                                      
│                                                                                           /root                                     
│                                                                                           /srv                                      
│                                                                                           /boot/grub2/x86_64-efi                    
│                                                                                           /boot/grub2/i386-pc                       
│                                                                                           /.snapshots                               
│                                                                                           /                                         
├─nvme0n1p9  xfs                        726ede17-0e41-4f90-8e45-ebf5d7bf1f69  353.3G    43% /home                                     437c59cd-0801-4ea6-860e-86859f59403f
└─nvme0n1p10 swap     1                 9c55cc0a-c911-4aca-a748-33fecbdc820e                [SWAP]                                    26c8e2f8-4ac0-457f-879b-08839d517628

That starts to sound like some obscure btrfs issue. You may consider booting from live image and running

btrfs check /dev/nvme0n1p8

Booted into live image using openSUSE-Tumbleweed-DVD-x86_64-Snapshot20240923-Media.iso

Still nothing suspicious. Could you try manually mount @/.snapshots subvolume in case there is some error that is hidden behind menu. Rename /boot/grub2/grub.cfg. Reboot. You will stop in grub2 CLI. Now do

echo $root
ls /.snapshots
btrfs-mount-subvol ($root) /.snapshots @/.snapshots
ls /.snapshots

You can boot your system by using in grub CLI

configfile /boot/grub2/renamed-grub.cfg

it will show the same grub menu as usual.

ls /.snapshots
btrfs-mount-subvol ($root) /.snapshots @/.snapshots

I renamed grub.cfg and booted but the above two lines both report

‘/.snapshots’ not found
so I can’t mount.

(/.snaphots does exist when I boot normally of course)

Show

btrfs subvolume get-default /

$ sudo btrfs subvolume get-default /
ID 93104 gen 2818243 top level 265 path @/.snapshots/136/snapshot