Can't boot anymore (likely due to messed up btrfs subvolumes?)

English Install/Boot/Login
1

Hello everyone!

so this morning I couldn’t boot into my openSUSE Tumbleweed system at all, so I booted into tumbleweed rescue cd and with a mix of guides and AI tried to restore a btrfs snapshot the hard way as I had no access to snapper.

I think I messed up somewhere and made things even worse because now my system boots into emergency mode and in the emergency mode mount -a throws file or directory not found for either of the

/var, /usr/local, /srv, /root /opt, /home, /boot/grub2/x86_64-efi, /boot/grub2/i386-pc

which kinda makes sense, I guess since when I’m in the rescue cd terminal environment and run

sudo mount /dev/nvme0n1p3 /mnt
sudo mount -o subvol=@/.snapshots /dev/nvme0n1p3 /mnt/.snapshots

I don’t actually see any of the aforementioned subvolumes there only snapshots, really (and also my desperate attempts at creating the subvolumes manually)

linux@localhost:~> sudo btrfs subvolume list /mnt
ID 256 gen 425041 top level 5 path @
ID 265 gen 425044 top level 256 path @/.snapshots
ID 765 gen 377467 top level 265 path @/.snapshots/494/snapshot
ID 790 gen 377467 top level 265 path @/.snapshots/519/snapshot
ID 791 gen 377467 top level 265 path @/.snapshots/520/snapshot
ID 806 gen 377467 top level 265 path @/.snapshots/new-backup
ID 816 gen 425074 top level 265 path @/.snapshots/544/snapshot
ID 823 gen 393147 top level 265 path @/.snapshots/551/snapshot
ID 824 gen 425011 top level 265 path @/.snapshots/552/snapshot
ID 837 gen 400402 top level 265 path @/.snapshots/565/snapshot
ID 838 gen 424992 top level 265 path @/.snapshots/566/snapshot
ID 852 gen 425057 top level 265 path @/.snapshots/580/snapshot
ID 853 gen 424971 top level 265 path @/.snapshots/581/snapshot
ID 859 gen 425057 top level 816 path @/.snapshots/544/snapshot/root.broken
ID 860 gen 425037 top level 859 path @/.snapshots/544/snapshot/root.broken/snapshot
ID 861 gen 425045 top level 859 path @/.snapshots/544/snapshot/root.broken/root
ID 864 gen 425053 top level 816 path @/.snapshots/544/snapshot/@/root
ID 865 gen 425053 top level 816 path @/.snapshots/544/snapshot/@/srv
ID 866 gen 425053 top level 816 path @/.snapshots/544/snapshot/@/opt
ID 867 gen 425053 top level 816 path @/.snapshots/544/snapshot/@/tmp
ID 868 gen 425054 top level 816 path @/.snapshots/544/snapshot/@/var
ID 869 gen 425055 top level 816 path @/.snapshots/544/snapshot/@/.snapshots
ID 870 gen 425056 top level 816 path @/.snapshots/544/snapshot/@/usr/local
ID 871 gen 425057 top level 816 path @/.snapshots/544/snapshot/@/snapshot

so, can I somehow make this right? I’ve been trying to fix this now for about 10 hours. The snapshots should be “healthy” as yesterday everything worked fine. Could be update or something. Anyway, some things on the system have deeper and personal meaning to me and I cannot afford to lose them by reinstalling everything from scratch.

2

Which is when you should have stopped and post your question showing the actual errors.

What you listed are not subvolumes, you listed mount points. You did not show what is being mounted on these mount points.

When was this system installed originally? Show

btrfs subvolume get-default /mnt
cat /mnt/etc/fstab
lsblk -f

also provide /mnt/boot/grub2/grub.cfg (upload to https://paste.opensuse.org).

3

When was this system installed originally? Show

btrfs subvolume get-default /mnt
cat /mnt/etc/fstab
lsblk -f

of course, here are the outputs of these commands

linux@localhost:~> sudo mount /dev/nvme0n1p3 /mnt
linux@localhost:~> btrfs subvolume get-default /mnt
Absolute path to 'btrfs' is '/usr/sbin/btrfs', so running it may require superuser privileges (eg. root).
linux@localhost:~> sudo btrfs subvolume get-default /mnt
ID 852 gen 425057 top level 265 path @/.snapshots/580/snapshot
linux@localhost:~> cat /mnt/etc/fstab
UUID=b2d21e20-d49d-4aa8-87f1-45fcb1257f05  /                       btrfs  defaults                      0  0
UUID=b2d21e20-d49d-4aa8-87f1-45fcb1257f05  /var                    btrfs  subvol=/@/var                 0  0
UUID=b2d21e20-d49d-4aa8-87f1-45fcb1257f05  /usr/local              btrfs  subvol=/@/usr/local           0  0
UUID=b2d21e20-d49d-4aa8-87f1-45fcb1257f05  /srv                    btrfs  subvol=/@/srv                 0  0
UUID=b2d21e20-d49d-4aa8-87f1-45fcb1257f05  /root                   btrfs  subvol=/@/root                0  0
UUID=b2d21e20-d49d-4aa8-87f1-45fcb1257f05  /opt                    btrfs  subvol=/@/opt                 0  0
UUID=b2d21e20-d49d-4aa8-87f1-45fcb1257f05  /home                   btrfs  subvol=/@/home                0  0
UUID=b2d21e20-d49d-4aa8-87f1-45fcb1257f05  /boot/grub2/x86_64-efi  btrfs  subvol=/@/boot/grub2/x86_64-efi  0  0
UUID=b2d21e20-d49d-4aa8-87f1-45fcb1257f05  /boot/grub2/i386-pc     btrfs  subvol=/@/boot/grub2/i386-pc  0  0
UUID=60BD-A99B                             /boot/efi               vfat   utf8                          0  2
UUID=defa8162-5d7a-4a34-b75e-437416320a75  swap                    swap   defaults                      0  0
UUID=b2d21e20-d49d-4aa8-87f1-45fcb1257f05  /.snapshots             btrfs  subvol=/@/.snapshots          0  0
linux@localhost:~> lsblk -f
NAME FSTYPE FSVER LABEL                         UUID                                 FSAVAIL FSUSE% MOUNTPOINTS
loop0
     squash 4.0                                                                            0   100% /run/overlay/squashfs_container
loop1
     ext4   1.0                                 9d7f4027-9990-4fcd-9b96-312322d2525b  728.6M    72% /run/overlay/rootfsbase
sda  iso966 Jolie openSUSE_Tumbleweed_Rescue_CD 2024-11-27-08-56-55-00                              
|-sda1
|    iso966 Jolie openSUSE_Tumbleweed_Rescue_CD 2024-11-27-08-56-55-00                     0   100% /run/overlay/live
|-sda2
|    vfat   FAT16 BOOT                          2151-8C8D                                           
`-sda3
     ext4   1.0   cow                           a182ebba-a67e-4dc7-ad16-eda366e64bc1   12.7G     1% /run/overlay/overlayfs
nvme0n1
|                                                                                                   
|-nvme0n1p1
|    vfat   FAT32 SYSTEM                        60BD-A99B                                           
|-nvme0n1p2
|                                                                                                   
|-nvme0n1p3
|    btrfs                                      b2d21e20-d49d-4aa8-87f1-45fcb1257f05    1.2T    32% /mnt
|-nvme0n1p4
|    ntfs         Recovery                      AA90BEFD90BECF57                                    
`-nvme0n1p5
     swap   1                                   defa8162-5d7a-4a34-b75e-437416320a75

and here is the grub file

https://paste.opensuse.org/pastes/16d8181ef8c6

I’m very worried about the amount of available space reported by lsblk -f I remember having only about 40-50gb of free disk space left before this has happened.

4

I forgot to reply on the question “When was this system installed originally”. I’ve installed it somewhere around last December, I believe, so I’ve been daily driving it for around a year.

5

It may be useful if you described what you did following “guides and AI”. Normally the default subvolume is the oldest snapshot. Having most recent snapshot as the default suggests that you tried to perform rollback.

Anyway, your subvolumes appear to be gone. No idea how it is possible. You could try to mount top level of the filesystem:

mount -r -o subvol=/ /dev/nvme0n1p3 /mnt

and look under it. May be subvolumes became plain directories and they are still there. You will need to drill down the directory tree.

Otherwise this something for the btrfs mailing list. May be some developer will have an idea how to recover data if it is still possible.

You most certainly should avoid changing this filesystem until you recovered data. “Changing” includes mounting read-write.

6

Essentially I’ve wanted to restore an earlier btrfs snapshot from when the system was working, (which was the day before yesterday). I’ve only ever done it with snapper before when system broke long time ago which was extremely easy and life saving at the same time, but this time around, I couldn’t get to it, so I’ve had to boot into recovery disk and use btrfs directly and followed guides and AI, since it was new to me.

I used ChatGPT:

First I was told that before restoring snapshot I should clear subvolumes

sudo btrfs subvolume delete /mnt/root/var
sudo btrfs subvolume delete /mnt/root/usr/local
sudo btrfs subvolume delete /mnt/root/srv
sudo btrfs subvolume delete /mnt/root/root
sudo btrfs subvolume delete /mnt/root/opt
sudo btrfs subvolume delete /mnt/root/home
sudo btrfs subvolume delete /mnt/root/boot/grub2/x86_64-efi
sudo btrfs subvolume delete /mnt/root/boot/grub2/i386-pc
sudo btrfs subvolume delete /mnt/root/.snapshots

Then I was told to remove /root but I was skeptical of that so I just ran mv /mnt/root /mnt/root.broken just in case I’ll need it. That suggestion seemed extreme.

and then finally I was told to restore a snapshot

sudo btrfs subvolume snapshot /mnt/.snapshots/580/snapshot /mnt/root

I unmouted everything with sudo umount -R /mnt and ran reboot. Ended up in emergency screen.

I’ve then tried to recreate the subvolumes manually and I suspect that’s what all these @/.snapshots/544/snapshot/@/var@/.snapshots/544/snapshot/@/var and so on… most likely are. I believe I tried to restore earlier ones too, hoping it will make a difference which is probably why their ids don’t match the one from the command I showed above.

Is there any hope or am I royally screwed? The earlier snapshots should be fine considering the system worked just fine the day before yesterday, so hopefully they’re good for something. Can they be used to re-create the subvolumes with original files?

7

I have no idea what /mnt/root was at this point, but as subvolumes are gone, they apparently were mounted below this directory when you were deleting them.

Looks like it. You yourself deleted missing subvolumes. That what you get for blindly following any junk presented by

No. They are snapshots of the root subvolume only, they do not include content of the other subvolumes.

Where content that you want to recover was located? Under /home? Or in some other directory?

8

Reinstalling stuff is tedious, but I can live with that. The important files were in my user’s home directory, loss of those would be much worse than reinstalling :frowning:

9

Assuming your user’s home directory was under /home I am afraid it is lost now.

10

After reinstallation, simply restore the /home from you last backup :+1:
.
We backup our /home subdir once a week. We don’t worry about backing up the system itself - if it crashes in a bad way, it’s easier to simply reinstall the system, then restore /home from the recent backup.

11

Yeah, but unless Tumbleweed somehow does it, which I don’t think it does (understandably, since that’s up to the user) many of the important things aren’t backed up.

Tbf I’ve been using Tumbleweed heavily since around 2021 on various devices, and I always did it the other way. In the few instances where my system broke, I fixed it with snapper right away and the system was otherwise very stable, so I was kinda counting that on the few occassions when something breaks, I will just rollback with snapper.

Ofc this is as indicated above, my fault, and could’ve been prevented.

12

Deleted obvious post.

13

Sorry, I don’t want to add insult to injury. But I think it should be pointed out for other users, that

“rollback” is not a replacement for backup!

In case of, say a physical problem there’s no rollback - all is lost.