Multivolume encrypyted Btrfs RAID

***Gareth Jones*** · #1 26th June 2012, 11:02 PM

Hi all,

In my ongoing struggles with F17, after badblocks and SMART confirmed the good health of my drives, I decided to dump the mdraid set-up that keeps going flaky on my machine and use Btrfs RAID instead.

Obviously, Anaconda doesn't support Btrfs properly yet, so I instead installed single-volume ext4 versions of /, /var, /tmp and /home (as well as boot partitions and swap of course). All but the boot and root partitions are encrypted. To lessen the boot-time overhead of having six separate encrypted partitions in the final set-up, I used LVM with one encrypted physical volume on each drive.

After formatting the spare RAID counter-parts as Btrfs and copying the files to them, I booted using those partitions and added the original ext4 versions as RAID partners for the corresponding volumes.

Everything worked fine (apart from a bit of wrestling with SELinux), until I converted the single-volume Btrfs file-systems into RAID. Now, when I boot I get dumped at an emergency prompt. Root, boot and swap volumes are mounted correctly (even though root is RAID Btrfs), but /var and /tmp (subvolumes of the same Btrfs) and /home are not, and won't mount until I manually run "btrfs dev scan". The only different between root and the other Btrfses is that the latter are on the encrypted LVM.

So, the question is, how do I tell Fedora/systemd/dracut that it needs to rerun "btrfs dev scan" after loading the LVM volumes, before mounting /var, /tmp and /home?

In case this question isn't long enough, below is a diagram of my volumes and file-systems. Thanks for any help!

Code:

sda (GPT)
|– sda1 –    1 MiB – BIOS Boot "GRUB"
|– sda2 – 1023 MiB – Ext4      "Boot"
|– sda3 –   10 GiB – Btrfs     "Root1"
`– sda4 –   ~1 TiB – LUKS      "Crypt1"
sdb (GPT)
|– sdb1 –   10 GiB – Btrfs     "Root2"
`– sdb2 –   ~1 TiB – LUKS      "Crypt2"

crypt1 (LVM Volume Group)
|– Physical Volumes
|  `– Crypt1
`– Logical Volumes
   |– swap   4 GiB – Swap      "Swap1"
   |– var    5 GiB – Btrfs
   `– home  ~1 TiB – Btrfs
crypt2 (LVM Volume Group)
|– Physical Volumes
|  `– Crypt2
`– Logical Volumes
   |– swap   5 GiB – Swap      "Swap2"
   |– var    5 GiB – Btrfs
   `– home  ~1 TiB – Btrfs

"Root" (Btrfs RAID0 Pool)
|– Volumes
|  |– Root1
|  `– Root2
`– Subvolumes
   `– @root
"Var" (Btrfs RAID0 Pool)
|– Volumes
|  |– crypt1-var
|  `– crypt2-var
`– Subvolumes
   |– @tmp
   `– @var
"Home" (Btrfs RAID1 Pool)
|– Volumes
|  |– crypt1-home
|  `– crypt2-home
`– Subvolumes
   `– @home

Mounts
Root/@root on /
Boot       on /boot
Home/@home on /home
Var/@tmp   on /tmp
Var/@var   on /var
Swap1      as swap
Swap2      as swap

Quantum · #2 27th June 2012, 04:50 PM

I'm sorry that you chose BTRFS for this, and that your install is so complex. First thing I will say is that BTRFS is not ready. I've used it off and on for the past two years, and each time it has royally screwed me with data loss. Just so you know.

Second, it's not clear what you mean by making BTRFS "RAID". Do you mean Linux software RAID (never, never with BTRFS), or BTRFS' multivolume feature? BTRFS' multivolume feature is much better than RAID, if only the filesystem were reliable.

Third, you do not, not want to use BTRFS with LVM. Don't do it. Use BTRFS' built-in encryption.

Fourth, it's not clear what you're meaning to do. Apparently you have several volumes that you want to combine into a tank, but what for? Usually, given that BTRFS has not been folded into grub, you'd want /boot and / as some regular filesystem like ext4 and /home as BTRFS. Your big data would normally be in /home, or in some special directory under /media (like 'backups').

And finally don't use BTRFS. Use ZFS with the DKMS kernel module. Just do a regular install with /home in / as ext4, and set aside your big volumes. After that set up your big volumes as ZFS, encrypt or whatever, then modify fstab to make it /home, and set up initrd.img to include the ZFS driver, then reboot.

***Gareth Jones*** · #3 27th June 2012, 11:57 PM

Quote:

Originally Posted by Quantum

First thing I will say is that BTRFS is not ready.

I've not tested it particularly strenuously yet, but I've never had a problem with Btrfs. Actually it's a problem with md-raid RAID1 that's persuaded me to switch from RAID/Ext4 to Btrfs. I don't mind living on the bleeding edge (hence Fedora), and I have up-to-date back-ups, just in case.

Quote:

Second, it's not clear what you mean by making BTRFS "RAID". Do you mean Linux software RAID (never, never with BTRFS), or BTRFS' multivolume feature?

Btrfs multi-volume. I know it isn't technically RAID in the classical sense, but the documentation and commands insist on using "raid0" and "raid1" etc.

Quote:

Third, you do not, not want to use BTRFS with LVM. Don't do it. Use BTRFS' built-in encryption.

Btrfs doesn't have built-in encryption yet. The reason for using LVM is so that I have two encrypted physical volumes instead of six encrypted partitions. Each encrypted volume adds 1 second to the boot time. With hind-sight, it might have been better to just live with the extra four seconds, but if I can avoid another reinstall I will.

Quote:

Fourth, it's not clear what you're meaning to do.

I'm using Ext4 for /boot, as Btrfs doesn't make much sense for a small boot partition (although allegedly GRUB2 does support Btrfs). / is unencrypted for performance reasons, as there is little need for it to be encrypted. /home, /tmp, /var, and swap are encrypted for obvious reasons. The reason for using RAID is that I have two 1 TiB disks and I want to make the most of them.

Basically all I need is a way to convince the boot system to run "btrfs device scan" before mounting /home, /tmp and /var, although failing that I'll just go with explicitly adding the extra volumes to the fstab mount options, but that feels wrong somehow...

Quantum · #4 28th June 2012, 02:01 PM

I am new to Fedora, and there doesn't seem to be an rcS.d directory, but in Debian we'd make a mount script and symlink for /etc/rcS.d/S10btrfs to set up the volume right before mountall.

No idea where Fedora does mountall...

I remember that there is some catch, some reason why btrfs volumes are not set up automatically on boot, but I've done my best to forget everything about btrfs. I think it's another of their glitches. So no crypto, even now. ZFS has it.

If using SATA+LVM, you want to be sure to turn off the writeback cache on the drives, as it doesn't do cache flushing, and ensure NCQ is on. (hdparm -I /dev/sda) Maybe add to rc.local hdparm -W0 /dev/sda.

I know it looks OK now, but you will be sorry. And when you complain to the listserv they are guaranteed to remind you that IT IS EXPERIMENTAL (even after 4 years of so-called development).

I tried to tell you...

***Gareth Jones*** · #5 28th June 2012, 04:35 PM

Well I've resigned myself to using explicit "defaults,device=...,device=..." mount options in /etc/fstab. It's ugly, but it works. (Although I suspect that the only reason it isn't a race condition is that swap is also in the encrypted LVM, indirectly ensuring that the LVM is loaded before Systemd attempts to mount the non-root file-systems...)

Quote:

Originally Posted by Quantum

I am new to Fedora, and there doesn't seem to be an rcS.d directory, but in Debian we'd make a mount script and symlink for /etc/rcS.d/S10btrfs to set up the volume right before mountall.

Fedora uses Systemd rather than the traditional SysVinit. I'm still getting the hang of where things go I'm afraid, but /etc/systemd (which overrides and adds to defaults in /usr/lib/systemd) and /etc/udev (which overrides and adds to defaults in /usr/lib/udev) replace most of the /etc/rc.d configuration

Quote:

No idea where Fedora does mountall...

I don't think it does. Systemd reads /etc/fstab itself and mounts the individual file-systems when they are first needed.

Quote:

I remember that there is some catch, some reason why btrfs volumes are not set up automatically on boot, but I've done my best to forget everything about btrfs.

Single Btrfs devices are fine, but to mount a multi-device Btrfs, "btrfs device scan" must be run first to build the kernel's list of which devices are pooled together. The alternative is to explicitly list all devices in a pool with "mount /dev/xxx -o device=/dev/xxx,device=/dev/yyy,... /mntpnt", which can also be done in /etc/fstab but is rather ugly.

EDIT: My problem is that the device scan is run correctly before / is mounted, but it isn't run after the LVM is started, so I've had to resort to the "device=..." mount option.

Quote:

So no crypto, even now. ZFS has it.

It's on the to-do list, but unless you want per-file encryption, dm-crypt is more secure and still the recommended way to do it. In the meantime, eCryptFS allows per-directory encryption on any file-system.

Quote:

I know it looks OK now, but you will be sorry. And when you complain to the listserv they are guaranteed to remind you that IT IS EXPERIMENTAL (even after 4 years of so-called development).

I tried to tell you...

That's okay, I'm experimental too.

Quantum · #6 28th June 2012, 05:57 PM

Quote:

Originally Posted by Gareth Jones

Well I've resigned myself to using explicit "defaults,device=...,device=..." mount options in /etc/fstab. It's ugly, but it works.

Yes, this is absolutely necessary.

Quote:

Originally Posted by Gareth Jones

Fedora uses Systemd rather than the traditional SysVinit. I'm still getting the hang of where things go I'm afraid, but /etc/systemd (which overrides and adds to defaults in /usr/lib/systemd) and /etc/udev (which overrides and adds to defaults in /usr/lib/udev) replace most of the /etc/rc.d configuration

Thank you, I sure needed this. I'm taking the RHCSA exam tomorrow. Been running Debian exclusively for 14 years, but weak on RH-specific stuffs.

Just passed my CEH Monday!