UEFI + BTRFS Raid 1 + Tumbleweed

Dear community,

I am searching since days how I can correctly set up a Open Suse Tumblweed with 2 SSDs with a BTRFS raid 1 on an UEFI System in my Virtual Box.
I have read plenty of threads here in the forum but it didn’t help me.

Three other sources I stumbled over in this order were:
https://www.thomas-krenn.com/de/wiki/Ubuntu_Software_RAID_mit_redundanten_UEFI_Boot_Einträgen
https://seravo.fi/2016/perfect-btrfs-setup-for-a-server
https://www.complang.tuwien.ac.at/anton/btrfs-raid1.html

The interessting thing is, that the third source is already referencing the two former sources.

What I did:
[ol]
[li]Use Open Suse Tumbleweed DVD iso file to boot from [/li][li]Create GPT partition table on /dev/sda with the installer GUI[/li][LIST=1]
[li]/dev/sda1 with 500 MB, formatted with vfat, mounted at /boot/efi as suggested from the installer [/li][li]/dev/sda2 with 40 GB, formatted with btrfs, mounted at / as suggested from the installer (including all the suggested subvolumes) [/li][/ol]

[li]Let the installer do its job. [/li][li]Add second hard drive (/dev/sdb) (with the same size) when the machine is switched off [/li][li]Copy GPT partition table from /dev/sda to /dev/sdb [/li][li]Use [/li]```
btrfs device add /dev/sdb2 /

 and 

btrfs fi balance start -mconvert=raid1,soft -dconvert=raid1,soft /

 to create the btrfs raid1. 
[/LIST]


Until here it is clear to me and I think this is the correct way to go. But now I have the problem, that if I remove Disk1 (dev/sda) from the virtual machine, I can't boot anymore, because the bootloader (GRUB 2) is located on this disk.
So I tried the proposals from the given sources from above but unfortunately none of them worked for me. 
I tried this https://www.thomas-krenn.com/de/wiki/Ubuntu_Software_RAID_mit_redundanten_UEFI_Boot_Eintr%C3%A4gen:

sudo umount /boot/efi
sudo mkfs.vfat /dev/sdb1
sudo parted /dev/sdb set 1 boot on
sudo mount /dev/sdb1 /boot/efi
sudo grub-install --bootloader-id RedundancyBootloader /dev/sdb
sudo umount /boot/efi
sudo mount /boot/efi



Now I did a shutdown of the virtual machine and removed Disk1. Then I tried to start it again.

No luck with this one. From the EFI menu I can get to the bootloader of Disk2 and select Open Suse Tumbleweed but it gets stuck afterwards. I just see 3 green little squares.

I did a rollback of the virtual machine.
Then I combined https://seravo.fi/2016/perfect-btrfs-setup-for-a-server with https://www.complang.tuwien.ac.at/anton/btrfs-raid1.html because I wanted to have EFI and the first mentioned source here is apparently for BIOS.
I copied the bootloader partition with 

dd if=/dev/sda1 of=/dev/sdb1



I also changed in my /etc/fstab from 

UUID=3d0ce18b-dc2c-4943-b765-b8a79f842b88 / btrfs defaults 0 0

 to 

UUID=3d0ce18b-dc2c-4943-b765-b8a79f842b88 / btrfs degraded,strictatime 0 0

 (I have adapted the UUID) and I commented out in my /etc/fstab to mount the vfat boot partition because the UUID is not the same on /dev/sda1 and /dev/sdb1. Otherwise the system would not boot either.
I added the options 

rootflags=degraded

 to the two lines in the file /etc/grub.d/10_linux and did a 

grub2-mkconfig -o /boot/grub2/grub.cfg

.

Then I did a shutdown and removed Disk1 again.

No luck with this one either. From the EFI menu I can get to the  bootloader and select Open Suse Tumbleweed but it gets stuck afterwards.  I just see 3 green little squares.

Is there a better tutorial how this should done? Can someone enlighten me or can someone give me some hints?

Every help is appreciated!

Thank you very much!

The main things to know is that there were major changes in how openSUSE sets up and stores on disks in 4Q 1999. Everything published prior to Aug 2019 might not work the same way from then on.

I can’t locate a number of posts I made about various storage changes, I think at least one might have been related to RAID, but in any case the LEAP documentation should be the ultimate resource with the SUSE Storage guide for supplemental info. Also keep in mind if you use BTRFS as your root file system, you will also automatically set up BTRFS volumes (similar to, but not the same as LvM).

LEAP documentation
https://doc.opensuse.org/documentation/leap/reference/html/book.opensuse.reference/cha-expert-partitioner.html#sec-yast-system-raid
SUSE Storage
https://documentation.suse.com/sles/15-SP1/pdf/book-storage_color_en.pdf

Aside from the above,
I seem to remember that when I did some quick exploration reviewing the changes, I didn’t notice anything that struck me as particularly unintuitive, the YaST Partitioner automatically identified available disk space and I was able to configure RAID with a few quick selections and checking some boxes. If my recollection is correct, then I’d say that you’re working too hard at manually identifying, selecting and configuring… Maybe typical of how things worked some time ago but not likely required anymore.

So, that’s at least my initial and perhaps most important advice… Try using the Partitioner, especially during a new install so you can set up your RAID then and not later after install and first boot. If you choose the “Based on… Guided Setup” (or something like that) you’ll start with the default non-RAID recommended layout but give you the opportunity to select and configure numerous changes you might want. If your two virtual disk files are both available during your install (even in virtualization), the Installer should discover both disks and make them available for whatever layout you want.

HTH,
Post again if you find what I describe isn’t what you find, then I’ll take a look at it to see what is there

TSU

One small addendum,
I’ve found a UEFI setup in Virtualbox interesting but at least for me wasn’t stable in VBox 6.0 (I haven’t tried again in 6.1, but that may not be easily available yet with an openSUSE HostOS but would be OK on a MSWindows HostOS). Unless you have a special reason to configure UEFI (Secure Boot? Just practice?), there shouldn’t be any special reason why UEFI is required instead of MBR (It’s likely your virtual disks wouldn’t so large or complex to require UEFI/GPT).

Also, all the gyrations you did to configure UEFI should be unnecessary.
In Virtualbox, all you need to do is configure the VM to be presented a UEFI environment, then when the installer run, it will recognize the requirement to configure UEFI.
Big tip…
In all virtualization, it can be critical to set up the VM’s environment correctly before first boot. Every virtualization will have a different set of features, but in every case you should click through practically every setting to verify everything is set up correctly… Else your install could set things up wrong from the get-go and your effort ruined.

IMO,
TSU

I found my BTRFS RAID references…
In my openSUSE Wiki… :slight_smile:

https://en.opensuse.org/User:Tsu2/systemd-1#BTRFS_RAID

Enjoy. Hopefully.
TSU

Thank you very much for your fast reply.

At first I also wanted to use the Partitioner from the installation medium during the installation but I think it is using mdadm raid controller. But as far as I understood, btrfs can completly replace mdadm and LVM together. (Allthough I know, BTRFS has still some problems for raid5/raid6.) So this is the reason why I did it manually.

In the meantime, another idea occured to me. I wondered how I could debug the boot process and I found out, that I can actually start from the second bootloader I installed and continue to the kernel. The btrfs module is also loaded but it gets stuck at

A start job is running for /dev/disk/by-uuid/...

.

I have found another useful bug report here: https://bugzilla.redhat.com/show_bug.cgi?id=1212925I think I already knew but it looks like RHEL has abandoned BTRFS.
And the important link in this conversion: https://lists.freedesktop.org/archives/systemd-devel/2014-May/019308.htmlIt appears to me it is a BTRFS problem and I don’t know if it is already fixed, if will ever be fixed or how to deal with it the correct way.

Best regards
Georg

Thank you for the links! I see there are a lot more options and possibilities to discover!

Also could be of interest to you…

The first link in my openSUSE WIKI -BTRFS RAID
points to the Lizards announcement about the Partitioner supporting BTRFS RAID.
In the comments at the end of the article I described what I did to create a RAID 10 array in VMware, converting a default openSUSE 15 install from a default single disk to the multi-disk RAID array.

Because the Partitioner does not support RAID 10, I used command line to do the conversion which was not difficult.
Of course, if a person chose not to use the Partitioner, the steps I described could be used to convert to any type of RAID.

Very straightforward and easy to do, whether you use YaST or command line (of course YaST is visually less complex and should be less prone to errors).

TSU

Is in the details somewhere, I think mdadm is deprecated altogether by now (Am I wrong? I don’t think so but may need to check on that).
So, when you use YaST to set up your RAID, it will just use whatever is appropriate and supported.

As for RHEL,
Yes BTRFS was officially dropped but IMO is likely due to that one BTRFS bug that affected “large spindle RAID” which might be found in RHEL Enterprise customers but unlikely by any openSUSE (Anyone running on a RAID exceeding 16 spindles?). In any case, AFAIK the bug has been addressed but maybe RHEL isn’t comfortable supporting or recommending something that could still have a similar bug somewhere on very large spindle RAID.

In any case, no problem for the typical openSUSE User, BTRFS has a number of features that would be immensely more difficult to setup, configure and maintain (snapshots on ZFS), auto-healing (as well as it can), file recovery (Yes, little known but you don’t have to roll back an entire partition if you only want a single file that might have been changed or deleted), more.

TSU

I will definitely have a look at your links tomorrow. Can I also find something in there how to boot if one of two hard drives fail?

Thanks a lot!
Georg

Apparently,
The BTRFS Wiki says there is still an issue with “large number of spindles” RAID 5 and 6.
https://btrfs.wiki.kernel.org/index.php/RAID56

AFAIK that issue does not affect RAID 10 (or 01), so if you really do need to deploy a large array RAID, consider 10.
Which generally is the automatic preference of everyone I’ve spoken to, anyway.

TSU

This is how it looks like in the link you sent me:
https://www.directupload.net/file/d/5688/2srzgjli_png.htm

This it how it looks like in the Tumbleweed installer:
https://www.directupload.net/file/d/5688/e7hgcwhz_png.htm

I don’t have “Add btrfs…” button in the Yast partionioner of the installation medium. I can only see devices which are already added with a btrfs filesystem. I can only add a mdadm raid which I don’t want to use.

Whereas when I have finished the insallation and then use the yast partitioner I have the “Add btrfs…” button:
https://www.directupload.net/file/d/5688/68h76pij_png.htm

This is still a bit confusing to me but nevertheless it is not really the problem for me as I can first install the system and afterwards create the raid either with the yast partitioner or on command line. I can also add a bootloader to the second device. But If I remove one disk a can’t start the system anymore, even not with the degraded mount option. My guess is still, that the issue described in here (https://lists.freedesktop.org/archives/systemd-devel/2014-May/019308.html) has still not been fixed which would be odd because it is a conversation from the year 2014.

I did a test install and verified pretty much what you posted…
I see from your screenshots that you accepted the Installer option to deploy the second disk with a BTRFS partition, I tried both that and also without to see if the RAID setup would create the mirror or use an existing BTRFS partition and convert to mirror. Both failed.

So, it looks like the Partitioner does not set up RAID during an install.
I didn’t test, but should be able to do so when the system is fully installed, the RAID creation tool like other YaST tools is supposed to create the RAID array under the hood the appropriate way and especially for BTRFS not implement mdadm. So, at least for RAID 1 which is supposed to be supported, the Partitioner on an already installed system should “just work” without having to know the special BTRFS create commands.

TSU

Thanks for your confirmation!
Nevertheless, I am now able to establish the raid after the installation either on command line or with the Yast partitioner.
Additionally I know now, how to recover from a bad disk in my raid1 setup.

  1. If the raid1 was composed of just two disks (Disk1 and Disk2), then we need to add a new disk (Disk3) first because raid1 always needs at minimum two disks.
  2. Boot to a live system and open a terminal (I used the Open Suse Tumbleweed DVD and selected “Installation”. Then I pressed Strg-Alt-F2 to open the terminal.)
  3. Use e.g. cfdisk /dev/sdb to create a new GPT partition table on Disk3. Add a partition with 500MB of type “EFI System” for the new backup bootloader. Add a partition with the rest of the free space on Disk3 with type “Linux raid” (I don’t know if the type is really important)
  4. Prepare the filesystem for the backup bootloader: mkfs.vfat /dev/sdb1
  5. Mount the BTRFS filesystem of the disk that is not defect in degraded mode: mount -o degraded /dev/sda2 /mnt (degraded is necesarry because the mount would not work as there is one disk missing in the raid1 filesystem)
  6. Add the new partition /dev/sdb2 of the new disk to the BTRFS filesystem: btrfs device add /dev/sdb2 /mnt
  7. Remove the missing / broken disk from the btrfs raid: btrfs device delete missing /mnt (This will take some time as all the data that has been on the broken disk will be recreated on the new disk.
  8. It is possible to watch the process in another terminal (Strg-Alt-F5) by using e.g. watch -n 1 btrfs filesystem usage /mnt. At the beginning you will find an entry “missing”. If the previous command finishes, the missing entry will vanish and the used data will grow for the new device.
  9. Additionally, if the device that failed was the one with the boot partition that was in use, we also need to adapt the /etc/fstab entry. Use lsblk -f to find out the UUID for the new boot partition and replace the old UUID via an editor: vi /mnt/etc/fstab
  10. Do a reboot now and wacht the system working again.
  11. In my case I had some chunks that were not raid1 any more, so I ran: btrfs balance start -mconvert=raid1,soft -dconvert=raid1,soft /
  12. Then I installed the Grub2 bootloader to the newly created boot partition of Disk3.

I think thats it for now :slight_smile:
Thank you very much for your help!

Very nice.
Covers the “more difficult” scenario mounting a degraded array if you don’t already have a replacement disk available.

Others following this,
Should be noted that the various commands how to create and manage an array using command line are all covered in the BTRFS WikI linik in the collection of BTRFS RAID links I provide through my own Wiki link in the above post…
Although YaST using the Partitioner module should be able to help set up a simple and common RAID (including RAID 1),
the CLI is an alternative and what you would be required to use later to maintain and manage your array.

I can’t find it today, but it seems to me a few months ago there was another YaST module besides the Partitioner that was relevant to BTRFS… Was some kind of Volume manager that was originally designed to support LVM but was adapted to view (I can’t remember if it could also create) volumes for BTRFS as well (supports both LVM and BTRFS volumes transparently and seamlessly). But I don’t see it now.

TSU