How do I fresh install BTRFS encrypted RAID10?

I have four identical drives I want in RAID10, and full disk encryption. According to section 2.8 in https://code.google.com/p/cryptsetup/wiki/FrequentlyAskedQuestions#2._Setup “Unless you have special needs, place encryption between RAID and filesystem, i.e. encryption on top of RAID.”

Starting with blank drives and an openSUSE 13.1 x64 DVD, what are the steps to achieve this configuration? Also, I would prefer key (read from USB thumbdrive) and passphrase as opposed to just passphrase or just key to boot the system. How do I do that?

not really sure how to do encryption, but want to ask about how exactly are you setting things up.

are you using MD raid and put btrfs volume on it?
are you using btrfs built in raid capability to create multidisk volume in raid10 configuration?

do you know that btrfs has the raid capability built-in, as in that you do not need to use MD raid?

I am sure many people would need/want to know answers to this questions, as well as what kind of setup you have/plan.

https://btrfs.wiki.kernel.org/index.php/FAQ#Does_btrfs_support_encryption.3F

AFAIK BTRFS’s advanced features such as RAID is not supported ib Yast so not supported on install. So you will need to do the RAID setup from the command line. Read up on the BTRFS doc’s. My guess is you create the partitions formatted BTRFS and then link them but that is just a guess. Adding encryption is another matter

actually it is easy to create btrfs supported raid with single command
you do not even need to create any partitions on any drives, although I would recommend to create a btrfs partition on one of the drives and than expand it using “btrfs device add /dev/sdc /mnt” as it makes it easier to mount the device later. but it is optional

 mkfs.btrfs -m raid10 -d raid10 /dev/sdb /dev/sdc /dev/sdd /dev/sde

I think “-L VOLUMELABEL” in above command also allow you to add label to the volume

once the volume is created you can mount it using any device in the group.

any one of this will mount the whole volume.
mount /dev/sdb /mnt
mount /dev/sdc /mnt
mount /dev/sdd /mnt
mount /dev/sde /mnt

here is a wiki page Using Btrfs with Multiple Devices - btrfs Wiki

gogalthorp is right though that the multidevice raid for btrfs is not supported in Yast and on install.
in fact Yast would not even recognize the drives as in use so be very careful when using this setup.
what you can do though is add btrfs volume to a single device on install, and install the system to it.
than when all is done and running, add devices to the volume and do

btrfs balance start -dconvert=raid10 -mconvert=raid10 /mnt

it will convert the single device pool into multidevice on the fly, and yes you can do this on running system. with device mounted.

YaST (at least in 13.1) does not allow encrypting of root filesystem. It is possible to manually create partitions for MD, create MD array and mark it for encryption but you cannot then use this MD array for “/” mount point.

It is possible manually (I did it once for testing) by installing in separate partition, creating manually btrfs on top of encrypted MD and moving OS there. But of course it makes recovery even more difficult and would need some experience.

why would you need an encrypted “/” volume ?
I mean I understand encripted /home volume and data volumes but “/”

Possible tmp files and caches (but much of this is in tmp file systems now) and also swap should be encrypted for try high security. But you can have a seperate non-encrypted boot partition.

Of course encrypting stuff only really protects against someone with actual possession of a non running system since a running system is decrypted by definition.

On 2014-06-20 18:56, vl1969 wrote:
>
> why would you need an encrypted “/” volume ?
> I mean I understand encripted /home volume and data volumes but “/”

Because many things important for security or privacy, like for instance
if a laptop is stolen, are stored outside of home. There are tmp files
in /tmp, which may be unencrypted temporary copies of stuff in home;
there are config files in /etc holding passwords, like the WiFi
passwords, there are logs in /var/log that may contain many sensitive
details, depending on how you use it. Mine would contain all mail
addresses of the mails I send, dates, mesageids. Some services, if
logging debug mode, may include passwords.

There may be databases under /var somewhere. Pending mail spool is there
(if using system mail). Faxes (hylafax) are there. Print jobs (cups) can
keep copies of old jobs there, specially if configured for not
autodelete finished jobs.

You may have server type services, like apache or ftp, under /srv.

You may have internal, job related, programs on /opt or /usr/local/,
that could give information to a competitor.

Many things… Those are what I could remember now.

Or simply you may wish to deny the robber the use of the computer as is.

On the other hand, you may wish they do boot it and connect it to
internet, because there is software that will automatically take the
first available chance to send all data it can to a server somewhere,
including webcam shots, so that the thief can be located and properly
punished.


Cheers / Saludos,

Carlos E. R.
(from 13.1 x86_64 “Bottle” at Telcontar)

I don’t see how that can be, because the encryption (presumably LUKS) is to be between the BTRFS and the RAID – so I don’t see how BTRFS can manage the RAID, when the encryption itself is separate from BTRFS.

I appreciate this post, and here’s hoping it will put an end to the questions as to why I want all of / encrypted.

The machine has a role as a server and I’ve rigged the office alarm to trip the mains power. The UPS still powers the server, but it’s USB-connected to it so apcupsd would shut it down. In test with our current server, it takes around 20 seconds (it takes a few seconds for the server to email my wireless provider’s SMS relay, and around 15 seconds for actual shutdown).

But how would I create the LUKS encryption on top of RAID, and then have BTRFS working on top of the encryption? It seems to me then BTRFS can’t be what actually manages the RAID.
Worst case, I don’t mind using the hardware RAID on the motherboard, but I’d rather rely on Linux to do it so it has a standard format to ease recovery in another machine, should that be necessary.

Yes indeed! I figure out why you was asking. I have no technical help or advise to offer unfortunately :P.

Regards

Can you elaborate on this? I look at some other Linux forums (Ubuntu, Archlinux) and similar setups have disks|MD|LUKS|filesystem
But it’s not clear for me whether BTRFS might not be a poor choice for filesystem (but what else can I use? I prefer something that’s good for a performance and reliability–I’m already doing RAID10 for that reason, so I’d like the best choice of FS as well). What other FS should I look at? I considered ZFS but it seems administration is quite complex (which is an issue as I’m not an IT admin; we’re a team of four programmers in the office and I’m moonlighting as an IT guy).
Looking at the Yast issue, I see http://lizards.opensuse.org/2009/03/18/encrypted-root-file-system-on-lvm/ but this is for 11.1, and also it talks about using LUKS on an encrypted partition–how would I change that to run on the LUKS-over-MD setup?

[Edit:] At SDB:Encrypted root file system - openSUSE Wiki, under “Information for current Opensuse Versions (Opensuse 11.2 and newer)” it says that “Encrypting the root file system, as well as the home, tmp and other partitions is now fully supported in the Opensuse graphical installer.”
But then it gives a link to the general documentation and therein I only can find instructions on creating an encrypted partition, and no mention of encrypting root (looking at http://activedoc.opensuse.org/book/opensuse-security-guide/chapter-10-encrypting-partitions-and-files ). And it’s not clear from that how to do that creation of encrypted partition together with RAID under it.

did you read the WiKi for btrfs? do your self a favor and read it.

as a recap : yes BTRFS contains all features of RAID and LVM with in.
well not sure about ALL features but it is targeted as a full replacement for LVM and software RAID(MD).
or so they say.

if you read the functionality:
in a nutshell : RAID give you the ability to create a pool of multiple devices to provide a platform for file system and protect your data/filesystem from a device failure using Parity or mirroring of blocks across several devices.

limitation of regular Raid (hardware and software) is that all devices with in a pool MUST be of the same size and types. all devices used in software raid must have a properly setup partition(s) before raid is setup.

LVM - allows you to pool multiple devices, including raid pools, and present them as a single raw volume to be partitioned as need for data storage.
(yes you can setup an LVM on raw disks but this is the most common configuration)

| / | /var | /usr | /home |

LVM Volume
RAID1

| Disk 1 | Disk 2 | Disk 3 |
courtesy of “http://serverfault.com

BTFS combines both of this functions and allows you to do both as needed. and do all of it on a live system and live mounted volumes for most part.

example: you have 4 disks @ 1TB each. aka sda , adb , adc , sdd

you can setup all 4 disks as individual 1TB btrfs disk/volumes => sda1 , adb1 , adc1 , sdd1

OR

as a raid pool or several pools of supported raid type (RAID 0/1/10 support for 5/6 is still not ready yet but promised soon)

Raid 0 will give you ~3.5TB (no redundancy remember but by default metadata is still in Raid1 configuration.)
you can use pool of all 4 or 2 pools of 2 device each

Raid 1 will give you 2TB full redundancy metadata and data. sigle device failure tolerance
you can use pool of all 4 or 2 pools of 2 device each

Raid 10 also 2TB but with 2 device tolerance depended on what 2 devices had failed.
one pool of 4 or more devices. can not use less than 4 devices.

this provides the raid functionality as in protecting data blocks from device failure, btrfs can work with partitions on a single devices or just raw devices.

now in the same time BTRFS provides the LVM functionality to you as you can expose this pools as either a single volume or multiple volues similar as you would do with LVM.
btrfs also have somewhat new concept in working with sub-volumes added.
sub-volume appears as a normal folder with in fs structure but you have the ability to mount the sub-volume directly as if it was a file system on its own.

I really don’t know how would you use all of this with encryption.
it seams if you want to use encryption you would need to compromise between getting all the benefits of btrfs and
sacrificing some of them to get the encryption.

google it for btrfs+LUKS

http://nyeggen.com/blog/2014/04/05/full-disk-encryption-with-btrfs-and-multiple-drives-in-ubuntu/

http://www.oracle.com/technetwork/articles/servers-storage-admin/advanced-btrfs-1734952.html

https://btrfs.wiki.kernel.org/index.php/FAQ#Does_btrfs_support_encryption.3F

vl1969, as I posted above, none of this matters as BTRFS will not be able to manage the RAID through the encryption layer. The RAID has to be done independently.

The biggest problem I see now is the “Yast doesn’t support encrypted /” which has been mentioned. I don’t think any other distro lacks support for full disk encryption. I looked for manual work-arounds but none of them assume RAID as well.

On the bottom of https://features.opensuse.org/305633 it says it will work if it’s over LVM. Does that mean I can just riff off the Ubuntu instructions at http://ubuntuforums.org/showthread.php?t=1935300 ?
How much of a performance hit would I get from the additional indirection due to an LVM layer as well? Should I use another FS, like ext4, instead of BTRFS atop such a setup?

You can encrypt root but if you do you must have a separate boot partition that is not encrypted. BTW same with LVM you have to have a separate non LVM boot. This is a limitation of grub since it can not read either and must live some where it can also access R/W.

If a serious installation you could use real hardware RAID (ie no drivers transparent operation) and encrypted LVM with a separate ext2 boot

I’d avoid FAKE RAID (BIOS assisted)/ Downs side of real RAID is that cards start at about $250 Pay close attention since you can also buy FAKE RAID in the same price range.

On 2014-06-20 22:06, Prune wrote:
>
> arvidjaar;2649880 Wrote:

> Can you elaborate on this? I look at some other Linux forums (Ubuntu,
> Archlinux) and similar setups have disks|MD|LUKS|filesystem
> But it’s not clear for me whether BTRFS might not be a poor choice for
> filesystem (but what else can I use?

If you require fast emergency shutdown, like you described on another
post, connected to the building alarm, I don’t think you should use
btrfs - just in case it decides that instant to sync or prune or do
housekeeping task at precisely that instant.

> I prefer something that’s good for
> a performance and reliability–I’m already doing RAID10 for that
> reason, so I’d like the best choice of FS as well). What other FS should
> I look at? I considered ZFS but it seems administration is quite complex
> (which is an issue as I’m not an IT admin; we’re a team of four
> programmers in the office and I’m moonlighting as an IT guy).

I like XFS. It has dedicated backup tools that work very fast, for instance.


Cheers / Saludos,

Carlos E. R.
(from 13.1 x86_64 “Bottle” at Telcontar)

On 2014-06-20 22:56, Prune wrote:

> The biggest problem I see now is the “Yast doesn’t support encrypted /”
> which has been mentioned.

It does, with a trick. What YaST does is encrypt the disk as a single
partition (/boot is outside). On this device, it creates an LVM layer,
and this one has one space for root, another for swap, and another for
home, using anyfilesystem you like (well, btrfs is special, dunno about it).

Adding raid to that is possible, but yast would not set it up automatically.

Me, I’m attempting to find a way to install to separate traditional
partitions, including root, encrypted. I started doing tests, but then I
got stuck and I had other things to do, so I did not progress. I know it
is possible, but I have not found out the exact method to do it.

This would allow to use software raid as an addition, but everything
without YaST aid.

And maybe the resulting system could not be upgraded when the time
comes. Dunno.


Cheers / Saludos,

Carlos E. R.
(from 13.1 x86_64 “Bottle” at Telcontar)