Recover from broken btrfs on /

English Install/Boot/Login
1

Due to an interrupted btrfs balance on my root partition I can’t boot my Tumbleweed installation. I have a luks container with an LVM inside: 1 volume group system, then 3 logical volumes root, swap and home. root is a BTRFS and home is a XFS.

During the boot the mounting of / timeouts (exceeds the 1m30s), it can only be mounted to. So I get the rescue console: LUKS container is already open and I see all the volumes with lvs. Tough I can’t mount / when doing it manually (it takes forever).
For btrfs balance I need a rw mounted filesystem (to resume or cancel the balance), for btrfs check and btrfs recover I need it unmounted. Ok, that won’t work so easily.

On another machine with Leap 42.2 I created two copies of the full disk with dd. I can open the LUKS container with cryptsetup luksOpen just fine. But then I am stuck: I can’t find out how to read the LVM volumes. pv/vg/lvscan and pvs/vgs/lvs all showed nothing:

> sudo cryptsetup luksOpen /dev/loop0p2 mnt
Enter passphrase for /dev/loop0p2: 
> ls -la /dev/mapper/
total 0
drwxr-xr-x  2 root root      80 Jun  4 12:12 .
drwxr-xr-x 19 root root    4700 Jun  4 12:12 ..
crw-------  1 root root 10, 236 Jun  4 11:00 control
lrwxrwxrwx  1 root root       7 Jun  4 12:12 mnt -> ../dm-0
huabo@faulty:~> sudo pvscan
  No matching physical volumes found
> sudo pvs
>

How can I access the volumes?

On my TW rescue console I can see that there should be 2 files more in /dev/mapper (two additional links to /dev/dm-1 and /dev/dm-2), so probably I opened the LUKS wrong?

I’d prefer to recover the broken btrfs over making a new installation.
Any help appreciated.

2

Output of blkid with LUKS configured would be useful.

3

On my TW system:


# blkid
/dev/sda1: UUID="11b1eef7-..." TYPE="ext4" PTTYPE="dos" PARTUUID="e45e5566-01"
/dev/sda2: UUID="e5986de8-..." TYPE="crypto_LUKS" PARTUUID="e45e5566-02"
/dev/mapper/cr_ata-INTEL_SSD...-part2: UUID="uM6Hr0-..." TYPE="LVM2_member"
/dev/mapper/system-swap: UUID="c4f6e8d3-..." TYPE="swap"
/dev/mapper/system-root: UUID="5de700bb-..." UUID_SUB="c69120c0-..." TYPE="btrfs"

/dev/sda1 is the /boot partition

On Leap 42.2 (stripped):


/dev/loop0: PTUUID="e45e5566" PTTYPE="dos"
/dev/loop0p1: UUID="11b1eef7-540d-41b9-9d44-58c54df18915" TYPE="ext4" PTTYPE="dos" PARTUUID="e45e5566-01"
/dev/loop0p2: UUID="e5986de8-67c1-412d-a884-180a2eb1ac32" TYPE="crypto_LUKS" PARTUUID="e45e5566-02"
/dev/mapper/mnt: UUID="uM6HrO-qsJk-aXgE-ftXi-8zEW-dNK2-uUVQZQ" TYPE="LVM2_member"
4

The /etc/fstab from the broken system:

/dev/system/swap       swap   swap   defaults 0 0
UUID=5de700bb-...     /       btrfs   defaults  0 0
UUID=5de700bb-...     [all the subvolumes]...
UUID=5de700bb-...     /.snapshots btrfs   subvol=@/.snapshots  0 0
UUID=11b1eef7-...     /boot    ext4  acl,user_xattr 1 2
/dev/system/home      /home    xfs   defaults 1 2

The /etc/crypttab:

cr_ata-INTEL_SSD...-part2 /dev/disk-by-id/ata-INTEL_SSD...-part2 none none
5

Showing partial or edited output just makes it harder for everyone to help you. I just briefly tested moving LVM-on-LUKS from TW to Leap and it worked (although the difference was not loop device, but I reconnected disk to another VM). Actually as soon as I started LUKS container system automatically setup VG and LV. So there is something in your environment that prevents it; but as you deleted information about other devices, it is hard to make any guess.

6

These are other harddisks on the machine I use to analyze the problem and thus are only “noise” here. But anyway, here they are:


/dev/sda1: LABEL="data" UUID="e28582a8-e3e5-459d-b02d-a287d3bc6148" TYPE="ext4" PARTUUID="00010dec-01"
/dev/sdb1: UUID="09a80e77-d812-4c2e-8750-c7d171cafe4d" UUID_SUB="b6f8e178-d3a5-4075-9ff0-abae79b24b71" TYPE="btrfs" PTTYPE="dos" PARTUUID="000247de-01"
/dev/sdb2: UUID="5f1b432c-a06b-4b2a-ac86-81ed8ae3b09d" TYPE="swap" PARTUUID="000247de-02"
/dev/sdc1: UUID="fe826746-e4d0-423e-a850-f5b7a71de338" TYPE="ext4" PARTUUID="000184d7-01"
/dev/sde1: UUID="6720e5fc-a3d3-4d1c-a86c-6b7c179e64e5" TYPE="ext4" PARTUUID="918f54a6-01"
/dev/sde5: UUID="289efdcf-3574-4bd3-823f-66aced590662" TYPE="swap" PARTUUID="918f54a6-05"

After some time and some more scans (vgscan and vgchange -a) I got all the volume groups and logical volumes and could mount the volumes. I’m not sure what caused that but it seems to work now. I’m now making copies of these partitions themselves (so I have the data easily accessible) and afterwards I try to repair the btrfs filesystem on the SSD itself.

7

btrfs will resume interrupted balance and in some cases it may take quite a lot of time. Try mounting with “skip_balance” option, it should hopefully allow you to proceed further.

8

Will this be done during the mount-process? Because the mount did take very long and thus it got interrupted, so it was mountable read-only. As balance needs it rw, that couldn’t continue.
In the meantime I used btrfsck --repair, and btrfs check to fix the filesystem (and waited for the automatically started balance) and it boots now!

Can mark this thread as solved somehow?