Stuck at boot because of broken BTRFS

Dear all,

I am running openSUSE Tumbleweed on my Laptop and yesterday my storage ran out, I think because some issue with snapper snapshots and btrfs. I tried to clean up snapshots I didn’t need anymore by creating a new default snapshot using snapper rollback and delete other snapshots. Then I tried to reboot and now my system is stuck with the message “”. This never finishes, even if I give it a lot of time.

Here the complete things that is shown (has some errors because of OCR):

[  OK  ] Created slice Slice /systen/modprobe *
[  OK  ] Started Show Plymouth Boot Screen
[  OK  ] Started Forward Password Requests to Plynouth Directory Watch
[  OK  ] Reached target Path Units
[  OK  ] Stopped Virtual Console Setup
         Stopping. Virtual Console Setup
         Starting Virtual Console Setup
         Mounting Kernel Configuration File Systen
[  OK  ] Mounted Kernel Configuration File Systen
[  OK  ] Finished Virtual Console Setup.
[  OK  ] Found device Samsung_SSD_860_EVO_M.2_250GB 2
[  OK  ] Found ‘device Sansung_SSD_860_EVO_M.2_250GB 1
        Starting Cryptography Setup for cr:_M.2_250GB_5413NXOK914024P-part2
[  OK  ] Finished Cryptography Setup for cr_ata-Sansung_SSD_860_EVO_M.2_250GB_S413NXOK914024P-part2Z
[  OK  ] Reached target Local Encrypted Volunes
         Starting Validate LUKS2 devices.
[  OK  ] Finished Validate LUKS2 devices.
[  OK  ] Found device /deu/systen/suap
         Starting Resune fron hibernation.
[  OK  ] Finished Resune fron hibernation
[  OK  ] Reached target Preparation for Local File Systens
[  OK  ] Reached target Local File Systems
         Starting Create Systen Files and Directories
[  OK  ] Finished Create System Files and Directories
[  OK  ] Reached target Systen Initialization
[  OK  ] Reached target Basic Systen
[  OK  ] Finished dracut initqueue hook.
[  OK  ] Reached target Preparation for Renote File Systens
[  OK  ] Reached target Remote Encrypted Volumes.
[  OK  ] Reached target Remote File Systems.
[  *** ] A start job is ruming for /dev/napper/systen-root (10min 42s / no hnit)

I already tried to find out what the exact issue is by using GRUB edit and rd.break=pre-mount to get into a shell and then running btrfs check --readonly /dev/mapper/system-root and got a lot of error messages, which I wrote to a file which is 305 MB (!). These are the kind of messages it contains:

pening filesystem to check...
Checking filesystem on /dev/mapper/system-root
UUID: cbc61247-0782-417c-8a96-8b7085454757
found 217717202944 bytes used, error(s) found
total csum bytes: 186207412
total tree bytes: 4951703552
total fs tree bytes: 4443947008
total extent tree bytes: 234536960
btree space waste bytes: 1123367526
file data blocks allocated: 529878421504
 referenced 296918888448
[1/8] checking log skipped (none written)
[2/8] checking root items
bad tree block 496795959296, bytenr mismatch, want=496795959296, have=0
ERROR: failed to repair root items: Input/output error
[3/8] checking extents
bad tree block 496790781952, bytenr mismatch, want=496790781952, have=0
bad tree block 496793665536, bytenr mismatch, want=496793665536, have=0
bad tree block 496795959296, bytenr mismatch, want=496795959296, have=0
bad tree block 496797220864, bytenr mismatch, want=496797220864, have=0
bad tree block 496797351936, bytenr mismatch, want=496797351936, have=0
bad tree block 496797712384, bytenr mismatch, want=496797712384, have=0
bad tree block 496798367744, bytenr mismatch, want=496798367744, have=0
bad tree block 496798973952, bytenr mismatch, want=496798973952, have=0
bad tree block 496799137792, bytenr mismatch, want=496799137792, have=0
bad tree block 496799350784, bytenr mismatch, want=496799350784, have=0
bad tree block 496799547392, bytenr mismatch, want=496799547392, have=0
bad tree block 496799809536, bytenr mismatch, want=496799809536, have=0
bad tree block 496799907840, bytenr mismatch, want=496799907840, have=0

(hundreds of these)

ref mismatch on [1217990656 4096] extent item 0, found 1
data extent[1217990656, 4096] referencer count mismatch (root 264 owner 26883545 offset 0) wanted 0 have 1
backpointer mismatch on [1217990656 4096]
ref mismatch on [1217994752 4096] extent item 0, found 1
data extent[1217994752, 4096] referencer count mismatch (root 258 owner 49738997 offset 0) wanted 0 have 1
backpointer mismatch on [1217994752 4096]
ref mismatch on [1217998848 4096] extent item 0, found 1
data extent[1217998848, 4096] referencer count mismatch (root 264 owner 22410836 offset 0) wanted 0 have 1
backpointer mismatch on [1217998848 4096]
ref mismatch on [1218002944 28672] extent item 0, found 1
data extent[1218002944, 28672] referencer count mismatch (root 264 owner 20996783 offset 0) wanted 0 have 1
backpointer mismatch on [1218002944 28672]
ref mismatch on [1218031616 8192] extent item 0, found 2
data extent[1218031616, 8192] referencer count mismatch (root 3986 owner 8389677 offset 0) wanted 0 have 1
data extent[1218031616, 8192] referencer count mismatch (parent 92861169664) wanted 0 have 1
backpointer mismatch on [1218031616 8192]
ref mismatch on [1218039808 4096] extent item 0, found 2
data extent[1218039808, 4096] referencer count mismatch (root 3986 owner 8389676 offset 0) wanted 0 have 1
data extent[1218039808, 4096] referencer count mismatch (parent 92861169664) wanted 0 have 1
backpointer mismatch on [1218039808 4096]

(same here)

	unresolved ref dir 289 index 17668 namelen 11 name zypp.X8ipct filetype 2 errors 4, no inode ref
root 258 inode 380214 errors 2001, no inode item, link count wrong
	unresolved ref dir 289 index 17728 namelen 11 name zypp.ANieFZ filetype 2 errors 4, no inode ref
root 258 inode 381274 errors 2001, no inode item, link count wrong
	unresolved ref dir 289 index 17989 namelen 11 name zypp.jnPl0L filetype 2 errors 4, no inode ref
root 258 inode 381395 errors 2001, no inode item, link count wrong
	unresolved ref dir 289 index 18001 namelen 11 name zypp.LNz7Io filetype 2 errors 4, no inode ref
root 258 inode 381793 errors 2001, no inode item, link count wrong
	unresolved ref dir 289 index 18013 namelen 11 name zypp.pt0BBc filetype 2 errors 4, no inode ref
root 258 inode 384670 errors 2001, no inode item, link count wrong
	unresolved ref dir 289 index 18159 namelen 11 name zypp.B58uMC filetype 2 errors 4, no inode ref
root 258 inode 384778 errors 2001, no inode item, link count wrong
	unresolved ref dir 289 index 18171 namelen 11 name zypp.1VP6Zx filetype 2 errors 4, no inode ref

(and here, and it ends with)

ERROR: errors found in fs roots

I already tried running btrfs check --readonly /dev/mapper/system-root but had no success doing so.

I also tried most of what is mentioned here: https://en.opensuse.org/SDB:BTRFS until btrfs restore /dev/sda1 /mnt/usbdrive. For that I started from an installation medium into rescue, because I wasn’t able to mount my exfat disk in the other shell. This worked, but now there is no /dev/mapper/system-root anymore and I am out of my depth. How to I unlock my disk so I can then try the other btrfs commands from the guide?

Thank you already and if you have any other tips, they are extremely welcome!

jnnk

I was now able to use cryptsetup luksOpen /dev/sda2 system-root to decrypt it, but mount shows /dev/mapper/system-root is of type “LVM2_member”. And I couldn’t yet figure out how to work with this.

it’s been a while since I last used lvm, but if I remember correctly there should be another “device” spawning in /dev/mapper/ that’s named after the actual logical volume. What you call system-root is the physical volume instead, which is part of the volume group but is not the logical volume your file system resides on.
you can run lvs in the rescue system I think to see your logical volumes and there should be something with a matching name under /dev/mapper for you to run your btrfs commands against.

might want to wait until more knowledgeable people chime in though.

Thank you! Running lvs results in:

  LV         VG        Attr          LSize       Pool Origin Data% Meta% Move Log Cpy%Sync Convert
  root       system    -wi-------    230.39g
  swap       system    -wi-a-----      2.00g

/dev/mapper only contains control, system-root and system-swap

I just checked one of my ubuntu systems that is also set up with lvm, the logical volume should be under /dev/<volumegroup>/<logical volume>,
so in your case probably /dev/system/root.
see if you can work with that. the lvdisplay command can show you more information, including the actual paths.

If I try to mount /dev/system/root using mount /dev/system/root /mnt/root I still get this error:

/mnt/root: unknown filesystem type 'LVM2_member'

Running vgchange -ay results in

device-mapper create ioctl on system-root LVM->...> failed. Device or resource busy
1 logical volume(s) in volume group "system" now active

I reinstalled it from scratch now. Thank you @xm3t4l !

1 Like

that always works :smiley: