Opensuse stopped booting (even from live usb), and there are btrfs errors

After the latest update, my laptop won’t boot (it just complains about mouse not being recognized or something, not sure it’s related).
It stops a second after selecting it from the grub menu, and the only thing on screen are these messages about the mouse not being recognized.

It also does the same when starting the with latest image from a usb drive.

Other distros work without a problem (typing this right now from ArcoLinux live session, and Pop!_OS works as well).

The other problem I discovered when trying to salvage my files - the /home dir appears to be empty (not even a user dir inside). other dirs seem to be ok… running btrfs check on the partition finds errors:

  Opening filesystem to check...
Checking filesystem on /dev/nvme0n1p3
UUID: b47fd5d1-a277-4a8a-8a8a-46b9771ecaaf
[1/7] checking root items
[2/7] checking extents
[3/7] checking free space tree
[4/7] checking fs roots
[5/7] checking only csums items (without verifying data)
[6/7] checking root refs
[7/7] checking quota groups
Counts for qgroup id: 0/257 are different
our:		referenced 13693177856 referenced compressed 13693177856
disk:		referenced 13693030400 referenced compressed 13693030400
diff:		referenced 147456 referenced compressed 147456
our:		exclusive 13693177856 exclusive compressed 13693177856
disk:		exclusive 13693030400 exclusive compressed 13693030400
diff:		exclusive 147456 exclusive compressed 147456
Counts for qgroup id: 0/262 are different
our:		referenced 90383990784 referenced compressed 90383990784
disk:		referenced 90383953920 referenced compressed 90383953920
diff:		referenced 36864 referenced compressed 36864
our:		exclusive 90383990784 exclusive compressed 90383990784
disk:		exclusive 90383953920 exclusive compressed 90383953920
diff:		exclusive 36864 exclusive compressed 36864
Counts for qgroup id: 0/340 are different
our:		referenced 14452690944 referenced compressed 14452690944
disk:		referenced 14452690944 referenced compressed 14452690944
our:		exclusive 1689124864 exclusive compressed 1689124864
disk:		exclusive 1736704 exclusive compressed 1736704
diff:		exclusive 1687388160 exclusive compressed 1687388160
Counts for qgroup id: 1/0 are different
our:		referenced 22157225984 referenced compressed 22157225984
disk:		referenced 22157225984 referenced compressed 22157225984
our:		exclusive 9310900224 exclusive compressed 9310900224
disk:		exclusive 1551155200 exclusive compressed 1551155200
diff:		exclusive 7759745024 exclusive compressed 7759745024
found 126464532480 bytes used, error(s) found
total csum bytes: 109196180
total tree bytes: 1261862912
total fs tree bytes: 1041530880
total extent tree bytes: 80035840
btree space waste bytes: 283922674
file data blocks allocated: 905943941120
 referenced 140640972800

Is there a way to restore my files? (and also preferably restore suse to working order… but that’s a secondary goal right now).

A general remark. Please always include the line with the prompt and the command before the output you copy/paste. It is only one more line and then people will know what exactly you did to get the output. Now it is only guessing :frowning:

You’re right.
It was

sudo btrfs check /dev/nvme0n1p3

I suggest you use advanced options in grub and boot either:

  • a previous kernel, see if that option works
  • a previous snapshot, if that works then use sudo snapper rollback to switch to it

Likely /home wasn’t mounted. If you follow the procedure above you should see it again.

I am a bit confused here.

Either there is a separate file system for /home or it is part of /.

When it is part of /, then why is it empty?

If it is a separate file system and not mounted at boot, then a file system check of it may be a good thing to start with. But a separate /home is most often not Btrfs. Nevertheless a check of btrfs file system on nvme0n1p3 was done.

What is supposed to be on /dev/nvme0n1p3?

Because directory /home on the root subvolume is by default a mount point and as any mount point it is empty - the content is in another subvolume. Which is normally @/home by default but it can really be anything, we do not have any information at all.

That is what I suggest with my post.

This is a symptom OP noticed, there was no indication in the original post warranting a file system check. Problem is, OP is unaware how mounting works (in particular a standard openSUSE scheme) on a live session.

Sorry for not giving enough information.
The the only partition relevant to opensuse is /dev/nvme0n1p3 - the rest are just boot, swap and os backup (from laptop vendor).

Here’s the fstab:

UUID=b47fd5d1-a277-4a8a-8a8a-46b9771ecaaf  /                       btrfs  defaults                      0  0
UUID=b47fd5d1-a277-4a8a-8a8a-46b9771ecaaf  /var                    btrfs  subvol=/@/var                 0  0
UUID=b47fd5d1-a277-4a8a-8a8a-46b9771ecaaf  /usr/local              btrfs  subvol=/@/usr/local           0  0
UUID=b47fd5d1-a277-4a8a-8a8a-46b9771ecaaf  /srv                    btrfs  subvol=/@/srv                 0  0
UUID=b47fd5d1-a277-4a8a-8a8a-46b9771ecaaf  /root                   btrfs  subvol=/@/root                0  0
UUID=b47fd5d1-a277-4a8a-8a8a-46b9771ecaaf  /opt                    btrfs  subvol=/@/opt                 0  0
UUID=b47fd5d1-a277-4a8a-8a8a-46b9771ecaaf  /home                   btrfs  subvol=/@/home                0  0
UUID=b47fd5d1-a277-4a8a-8a8a-46b9771ecaaf  /boot/grub2/x86_64-efi  btrfs  subvol=/@/boot/grub2/x86_64-efi  0  0
UUID=b47fd5d1-a277-4a8a-8a8a-46b9771ecaaf  /boot/grub2/i386-pc     btrfs  subvol=/@/boot/grub2/i386-pc  0  0
UUID=B8AE-88C7                             /boot/efi               vfat   utf8                          0  2
UUID=b47fd5d1-a277-4a8a-8a8a-46b9771ecaaf  /.snapshots             btrfs  subvol=/@/.snapshots          0  0
UUID=48bb021a-04d9-41e7-886f-477f095ca967  swap                    swap   defaults                      0  0

I wasn’t aware that btrfs has subvolumes that need to be mounted separately… After mounting withsudo mount -o /@/home /dev/nvme0n1p3 /mount/old/home I was able to get access to my home files!

Nevertheless the btrfs check tool complained about problems…
Anyway, I guess it’s not the main issue preventing me from booting opensuse on this system, as even a newly downloaded image on a usb fails to boot.
It doesn’t even seem to start a system journal log, as the latest I have in /var/log/journal seems to be from the last good boot…

I don’t know where I can get more info on what’s wrong, but at least I can backup my data from my home dir :slight_smile:

I’m glad you figured the /home situation.

Complaints about qgroups i.e. the quota feature. I find it a bit unstable, and if you don’t need this feature, just disable them with sudo btrfs quota disable <mountpoint>. You can re-enable them later.

I’m missing your report on my advice on booting from #4, with that you’ll have another data point to share with us, whether it works or not.

Well… The advanced boot options only incluse one kernel version (two entries for the same version - and both won’t boot).
The recovery mode also won’t boot.

Ok. I don’t recall the standard setting, but it’s a good idea to keep more versions around:

$ grep multiversion.kernels /etc/zypp/zypp.conf
multiversion.kernels = latest,latest-1,latest-2,running

This mode boots the system without extra boot options we generally configure in YaST bootloader.

The other option you can try is called Start bootloader from a read-only snapshot, this gives you a few older states to revert to.

Ok, the most recent snapshot that worked was 2 kernels ago… Which is weird considering I was using this laptop up to yesterday with a more recent kernel…
Also, after rolling back, should I still attempt zypper dup?

You can try sudo btrfs scrub start /home and it might be able to fix the errors. Keep in mind that if you are running on a single disk with a single copy of the data odds are not in your favor. See:

So… I have rolled back to a working snapshot, but I’m pretty sure that upgrading would make my system unbootable again (since the live usb of the latest iso isn’t booting on this computer).
I don’t think it’s practical to keep this machine in an out of date state for long…
What data should I collect to report a bug regarding this?
journalctl didn’t log any of the failed boot attempts…

It is possible, though maybe not likely, that the .iso you have was corrupted during the download. I have three machines with a variety of consumer hardware that have had no issues with 15.5 . On the other hand it is possible that your hardware has some strange corner case with the new version.

Try booting the new version and at the GRUB screen hit e before starting the install. Then find the line that loads the kernel and remove quiet, splash=foo (it doesn’t really say “foo” but I don’t remember the exact word) and then at the end of the line add nomodeset=1. This will let you read everything that happens during the boot process and if there is an error or a warning you can take a pic of it with your phone. Also, you might be able to pause the boot process with the pause/break key. On QWERTY keyboards this is normally located right above the Page Up key.

Isnt that equivalent to choosing the safe boot option? I already posted such a picture on this thread… Couldn’t see anything out of the ordinary there…

Shot in the dark - recently there were several cases of users suddenly using kernel-default-base instead of kernel-default causing boot issues. Explicitly installing kernel-default fixed the problems. You may want to keep attention during update; also check snapshot that did not boot what is installed there.

Otherwise you need to open bug report on, explain situation and attach information you have (screenshot). btrfs warnings are red herring - according to the screenshot kernel hangs before trying to launch initrd, so no attempt to access btrfs at this point. Kernel developers are more familiar with possibilities to obtain output at this early stage.

I’ve just tried booting the latest installation iso (28 of july) after verifying its sha256sum. Edited the grub option for installation to not have splash, so I can see the startup lines as suggested above… Still freezes at exactly the same point as the faulty state I got to after the upgrade (see the picture I uploaded in this thread)… So unless the installation iso also has that kernel-default-base situation, I’d say it isn’t the problem.

Well, if you can boot from previous snapshot and cannot boot the current kernel it is a regression and must certainly be reported.