Boot fails with BTRFS RAID1 array as /home - open ctree failed

rsking84 · January 4, 2015, 11:19pm

Hello everyone,
I recently did a clean install of OpenSUSE 13.2 into a system with three drives - 1 SSD containing the / partition (btrfs format with default subvolumes), and 2 identical HDD’s containing /home in an encrypted BTRFS RAID1 array. Initially, I was using only one of the HDD’s as /home (still encrypted BTRFS). Once I added the second drive and started the RAID1 mirroring, my system will no longer boot without manual intervention. I get dropped into “emergency mode”, but then all I have to do is exit (Ctrl+D) to continue booting and everything works normally.

The last entry in the system log before Emergency Mode is invoked is:

BTRFS: open_ctree failed

It appears that a similar (maybe the same) bug was reported on ArchWiki here. But the solution refers to mkinitcpio rather than dracut that OpenSUSE uses, so I’m unsure how to apply it to my system.

Can anyone help me figure out the equivalent fix for dracut? Thanks!

Some relevant system information:
excerpt from /etc/fstab:


...
# encrypted RAID1 array containing /home
UUID=e91f611f-524a-43f5-bde5-8ebb9672f146 /home btrfs defaults 0 0

/etc/crypttab:


encrypted-home-sdb UUID=9c8fb7d0-74e2-4e38-b7c7-6211bbb6d2b1 none luks, retry=1
encrypted-home-sdc UUID=b82e5894-cea2-4fe7-a0a5-80f918e9db61 none luks, retry=1

excerpt from fdisk -l:


Device     Start        End    Sectors   Size Type
/dev/sdb1   2048 1953523711 1953521664 931.5G Microsoft basic data

Disk /dev/sdc: 931.5 GiB, 1000204886016 bytes, 1953525168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: CAC38602-D5A5-4F28-8344-65B559124514

Device     Start        End    Sectors   Size Type
/dev/sdc1   2048 1953525134 1953523087 931.5G Linux filesystem

Disk /dev/mapper/encrypted-home-sdc: 931.5 GiB, 1000201723392 bytes, 1953518991 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk /dev/mapper/encrypted-home-sdb: 931.5 GiB, 1000200994816 bytes, 1953517568 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes

Device UUID’s:


/dev/sda1: UUID="04e20317-f554-43cf-a95d-b0385ebd22bb" UUID_SUB="cb9dc431-1855-466b-ae9a-5c0ddcd7f95a" TYPE="btrfs" PTTYPE="dos" PARTLABEL="primary" PARTUUID="860c637a-83c8-4399-8dc4-5b5ab66c0f06" 
/dev/sdb1: UUID="9c8fb7d0-74e2-4e38-b7c7-6211bbb6d2b1" TYPE="crypto_LUKS" PARTLABEL="primary" PARTUUID="532a7bd2-6774-422e-90c0-31f19796098d" 
/dev/sdc1: UUID="1ca9d3ba-c409-4127-91f5-e3d9c21242bd" TYPE="crypto_LUKS" PARTLABEL="Linux filesystem" PARTUUID="5a6830cc-9064-476c-aa79-6189dd0964bd" 
/dev/mapper/encrypted-home-sdc: UUID="e91f611f-524a-43f5-bde5-8ebb9672f146" UUID_SUB="6007c4c7-5119-4b0f-ab21-134832c4774c" TYPE="btrfs" 
/dev/mapper/encrypted-home-sdb: UUID="e91f611f-524a-43f5-bde5-8ebb9672f146" UUID_SUB="020bae26-b64f-413e-8263-a6e832ccb224" TYPE="btrfs"

gogalthorp · January 5, 2015, 12:57am

Not all of BTRFS bells and whistles work. YOu should report this problem on Bugzilla

arvidjaar · January 5, 2015, 8:57am

Please, try the following:

echo 'c /dev/btrfs-control 0660 root root - 10 234' > /etc/tmpfiles.d/btrfs-control.conf

This will pre-create btrfs control device and should trigger loading of btrfs module.
And yes, you should open bug report in any case; this is prerequisite for issuing update for released openSUSE version.

rsking84 · January 8, 2015, 2:38am

arvidjaar:

Please, try the following:
echo 'c /dev/btrfs-control 0660 root root - 10 234' > /etc/tmpfiles.d/btrfs-control.conf
This will pre-create btrfs control device and should trigger loading of btrfs module.
And yes, you should open bug report in any case; this is prerequisite for issuing update for released openSUSE version.

Thank you for the suggestion; I issued this command as root and then rebooted, but it did not change the behavior.

I have opened a bug report here:
https://bugzilla.opensuse.org/show_bug.cgi?id=912170

rsking84 · January 10, 2015, 2:28pm

A workaround is now available in the bug report.

san100 · April 3, 2015, 11:05am

Hi I seem to have the same issue.

I have seen the bug report (Bug 912170) with workaround however as a newbie I do not know how to edit the file 64-btrfs.rules.

Here is a copy of the workaround from bugzilla…
@Ryan - as a workaround, copy /usr/lib/udev/rules.d/64-btrfs.rules into /etc/udev/rules.d/64-btrfs.rules and replace the following two lines

ENV{DM_NAME}=="", IMPORT{builtin}=“btrfs ready $devnode”
ENV{DM_NAME}=="?*", IMPORT{builtin}=“btrfs ready /dev/mapper/$env{DM_NAME}”

with single one

IMPORT{builtin}=“btrfs ready $devnode”

If you will have any issues, please mention them here.

two questions please:

do I edit the original file in /usr/udev/rules.d or the copied file in /etc/udev/rules.d ?
how do I edit the file in the btrfs shell since that is the only interface i can access on that HDD ?

Please help as this hdd is from a work machine with a ton of Data on it.

arvidjaar · April 3, 2015, 6:49pm

You can edit original file but it will be overwritten on update. Which is the reason to copy it and edit copy.

how do I edit the file in the btrfs shell

What is btrfs shell?

rsking84 · April 3, 2015, 8:08pm

I only edited the copied file (the one in /etc/udev/rules.d)
I assume by “btrfs shell” you just mean a Linux command prompt? If so, you need to use a command line text editor like nano or vim. I find nano the easiest to use. Assuming it’s installed, just execute

nano /etc/udev/rules.d/64-btrfs.rules

You will probably need to use su or sudo to get the necessary permissions.

san100 · April 3, 2015, 9:38pm

thanks for the reply’s. I am not a techie and have very limited knowledge however let me try explain in simple terms:

Firstly some info, OS is opensuse 13.2 64 bit. Hdd is a 128G ssd drive, Root is formated BTRFS and Home is EXT4

Grub loads as per normal however after selecting the kernel to start, Grub cannot mount the root partition. so defaults to a simple command line interface. The following is displayed:

1.815222] BTRFS: Failed to read block groups: -5
1.844513] BTRFS: open_ctree failed

Generating “/run/initramfs/rdsosreport.txt”

Entering emergency mode. Exit the shell to continue.
Type “journalctl” to view system logs.
You might want to save “/run/initramfs/rdsosreport.txt” to a USB stick or /boot
after mounting them and attach it to a bug report.

:/#

From here if I "ls’ I can see all the root files and the contents but do not know how to edit the "64-btrfs.rules file as per the workaround detailed in my previous post from bugzilla.

If I Boot with a different hard drive and open dolphin, it lists the two partitions from the ssd drive as sbd1 and sdb2, but can only mount the home (EXT4) partition not the root partition.

rsking84 · April 3, 2015, 9:55pm

Looking at your last post I’m not positive that this is the same bug as described in my report. In my case I got the “Emergency Mode” prompt. It says something to he effect of “type root password for maintenance or press Ctrl+D to exit” and then puts you into a shell. Assuming this is where you are:

See the previous post. At that prompt, you first need to copy the file:


sudo cp /usr/lib/udev/rules.d/64-btrfs.rules /etc/udev/rules.d/64-btrfs.rules

This creates a copy of the file for you to edit. Then type


sudo nano /etc/udev/rules.d/64-btrfs.rules

This should open a text editor within the shell. Make your edits then press Ctrl+X. At the bottom of the editor, you will see a prompt asking you to save the file and if it is OK to overwrite. Type Y and then enter. You have now edited the file.

Reboot and with any luck your system will boot.

Note that you should also be able to boot by just pressing Ctrl+D once you get to that emergency mode prompt (as documented in the bug report). It’s not a fix but it will get you into your system. You just have to manually intervene every time your reboot.

san100 · April 3, 2015, 10:01pm

Sorry I should have added that I have tried to edit with vi, nano, no editor starts. I just get

sh: nano: command not found

or

sh: vi: command not found

su also returns command not found.

rsking84 · April 3, 2015, 10:43pm

OK, based on this and your previous post that has the error message, my best guess is that the mount is failing before your system can create the InitRD, so therefore you don’t have access to some of the usual commands. I’m not 100% sure though.

It seems clear that this is a different bug than the one that is the topic of this thread. Note that the open_ctree failed message is common to many btrfs issues, so just because you have that message in your log doesn’t mean this is the right bug. I would focus on deciphering the message

Failed to read block groups: -5

I’m afraid I’m out of ideas beyond that. Good luck.

arvidjaar · April 4, 2015, 7:03am

If it is single disk you have different problem. Better to ask in separate thread.

1.815222] BTRFS: Failed to read block groups: -5
1.844513] BTRFS: open_ctree failed

Generating “/run/initramfs/rdsosreport.txt”

You are in initrd before root is mounted, so only very limited number of commands are available. Try “btrfs device scan -d” and then Ctrl-D to exit command line prompt - it may continue to boot.

san100 · April 7, 2015, 9:14am

Thanks for the suggestions, turns out the easiest way to resolve the problem is simply reformat root to EXT4 and reinstall Opensuse. (remember do not format home partition)