Multivolume encrypyted Btrfs RAID
FedoraForum.org - Fedora Support Forums and Community
Results 1 to 6 of 6
  1. #1
    Join Date
    Jul 2011
    Location
    Birmingham, UK
    Age
    39
    Posts
    2,761
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    Multivolume encrypyted Btrfs RAID

    Hi all,

    In my ongoing struggles with F17, after badblocks and SMART confirmed the good health of my drives, I decided to dump the mdraid set-up that keeps going flaky on my machine and use Btrfs RAID instead.

    Obviously, Anaconda doesn't support Btrfs properly yet, so I instead installed single-volume ext4 versions of /, /var, /tmp and /home (as well as boot partitions and swap of course). All but the boot and root partitions are encrypted. To lessen the boot-time overhead of having six separate encrypted partitions in the final set-up, I used LVM with one encrypted physical volume on each drive.

    After formatting the spare RAID counter-parts as Btrfs and copying the files to them, I booted using those partitions and added the original ext4 versions as RAID partners for the corresponding volumes.

    Everything worked fine (apart from a bit of wrestling with SELinux), until I converted the single-volume Btrfs file-systems into RAID. Now, when I boot I get dumped at an emergency prompt. Root, boot and swap volumes are mounted correctly (even though root is RAID Btrfs), but /var and /tmp (subvolumes of the same Btrfs) and /home are not, and won't mount until I manually run "btrfs dev scan". The only different between root and the other Btrfses is that the latter are on the encrypted LVM.

    So, the question is, how do I tell Fedora/systemd/dracut that it needs to rerun "btrfs dev scan" after loading the LVM volumes, before mounting /var, /tmp and /home?

    In case this question isn't long enough, below is a diagram of my volumes and file-systems. Thanks for any help!

    Code:
    sda (GPT)
    | sda1     1 MiB  BIOS Boot "GRUB"
    | sda2  1023 MiB  Ext4      "Boot"
    | sda3    10 GiB  Btrfs     "Root1"
    ` sda4    ~1 TiB  LUKS      "Crypt1"
    sdb (GPT)
    | sdb1    10 GiB  Btrfs     "Root2"
    ` sdb2    ~1 TiB  LUKS      "Crypt2"
    
    crypt1 (LVM Volume Group)
    | Physical Volumes
    |  ` Crypt1
    ` Logical Volumes
       | swap   4 GiB  Swap      "Swap1"
       | var    5 GiB  Btrfs
       ` home  ~1 TiB  Btrfs
    crypt2 (LVM Volume Group)
    | Physical Volumes
    |  ` Crypt2
    ` Logical Volumes
       | swap   5 GiB  Swap      "Swap2"
       | var    5 GiB  Btrfs
       ` home  ~1 TiB  Btrfs
    
    "Root" (Btrfs RAID0 Pool)
    | Volumes
    |  | Root1
    |  ` Root2
    ` Subvolumes
       ` @root
    "Var" (Btrfs RAID0 Pool)
    | Volumes
    |  | crypt1-var
    |  ` crypt2-var
    ` Subvolumes
       | @tmp
       ` @var
    "Home" (Btrfs RAID1 Pool)
    | Volumes
    |  | crypt1-home
    |  ` crypt2-home
    ` Subvolumes
       ` @home
    
    Mounts
    Root/@root on /
    Boot       on /boot
    Home/@home on /home
    Var/@tmp   on /tmp
    Var/@var   on /var
    Swap1      as swap
    Swap2      as swap

  2. #2
    Join Date
    Jun 2012
    Location
    Seattle, Ecotopia
    Posts
    33
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    Re: Multivolume encrypyted Btrfs RAID

    I'm sorry that you chose BTRFS for this, and that your install is so complex. First thing I will say is that BTRFS is not ready. I've used it off and on for the past two years, and each time it has royally screwed me with data loss. Just so you know.

    Second, it's not clear what you mean by making BTRFS "RAID". Do you mean Linux software RAID (never, never with BTRFS), or BTRFS' multivolume feature? BTRFS' multivolume feature is much better than RAID, if only the filesystem were reliable.

    Third, you do not, not want to use BTRFS with LVM. Don't do it. Use BTRFS' built-in encryption.

    Fourth, it's not clear what you're meaning to do. Apparently you have several volumes that you want to combine into a tank, but what for? Usually, given that BTRFS has not been folded into grub, you'd want /boot and / as some regular filesystem like ext4 and /home as BTRFS. Your big data would normally be in /home, or in some special directory under /media (like 'backups').

    And finally don't use BTRFS. Use ZFS with the DKMS kernel module. Just do a regular install with /home in / as ext4, and set aside your big volumes. After that set up your big volumes as ZFS, encrypt or whatever, then modify fstab to make it /home, and set up initrd.img to include the ZFS driver, then reboot.

  3. #3
    Join Date
    Jul 2011
    Location
    Birmingham, UK
    Age
    39
    Posts
    2,761
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    Re: Multivolume encrypyted Btrfs RAID

    Quote Originally Posted by Quantum
    First thing I will say is that BTRFS is not ready.
    I've not tested it particularly strenuously yet, but I've never had a problem with Btrfs. Actually it's a problem with md-raid RAID1 that's persuaded me to switch from RAID/Ext4 to Btrfs. I don't mind living on the bleeding edge (hence Fedora), and I have up-to-date back-ups, just in case.

    Second, it's not clear what you mean by making BTRFS "RAID". Do you mean Linux software RAID (never, never with BTRFS), or BTRFS' multivolume feature?
    Btrfs multi-volume. I know it isn't technically RAID in the classical sense, but the documentation and commands insist on using "raid0" and "raid1" etc.

    Third, you do not, not want to use BTRFS with LVM. Don't do it. Use BTRFS' built-in encryption.
    Btrfs doesn't have built-in encryption yet. The reason for using LVM is so that I have two encrypted physical volumes instead of six encrypted partitions. Each encrypted volume adds 1 second to the boot time. With hind-sight, it might have been better to just live with the extra four seconds, but if I can avoid another reinstall I will.

    Fourth, it's not clear what you're meaning to do.
    I'm using Ext4 for /boot, as Btrfs doesn't make much sense for a small boot partition (although allegedly GRUB2 does support Btrfs). / is unencrypted for performance reasons, as there is little need for it to be encrypted. /home, /tmp, /var, and swap are encrypted for obvious reasons. The reason for using RAID is that I have two 1 TiB disks and I want to make the most of them.

    Basically all I need is a way to convince the boot system to run "btrfs device scan" before mounting /home, /tmp and /var, although failing that I'll just go with explicitly adding the extra volumes to the fstab mount options, but that feels wrong somehow...

  4. #4
    Join Date
    Jun 2012
    Location
    Seattle, Ecotopia
    Posts
    33
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    Re: Multivolume encrypyted Btrfs RAID

    I am new to Fedora, and there doesn't seem to be an rcS.d directory, but in Debian we'd make a mount script and symlink for /etc/rcS.d/S10btrfs to set up the volume right before mountall.

    No idea where Fedora does mountall...

    I remember that there is some catch, some reason why btrfs volumes are not set up automatically on boot, but I've done my best to forget everything about btrfs. I think it's another of their glitches. So no crypto, even now. ZFS has it.

    If using SATA+LVM, you want to be sure to turn off the writeback cache on the drives, as it doesn't do cache flushing, and ensure NCQ is on. (hdparm -I /dev/sda) Maybe add to rc.local hdparm -W0 /dev/sda.

    I know it looks OK now, but you will be sorry. And when you complain to the listserv they are guaranteed to remind you that IT IS EXPERIMENTAL (even after 4 years of so-called development).

    I tried to tell you...
    Last edited by Quantum; 28th June 2012 at 02:15 PM.

  5. #5
    Join Date
    Jul 2011
    Location
    Birmingham, UK
    Age
    39
    Posts
    2,761
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    Re: Multivolume encrypyted Btrfs RAID

    Well I've resigned myself to using explicit "defaults,device=...,device=..." mount options in /etc/fstab. It's ugly, but it works. (Although I suspect that the only reason it isn't a race condition is that swap is also in the encrypted LVM, indirectly ensuring that the LVM is loaded before Systemd attempts to mount the non-root file-systems...)

    Quote Originally Posted by Quantum
    I am new to Fedora, and there doesn't seem to be an rcS.d directory, but in Debian we'd make a mount script and symlink for /etc/rcS.d/S10btrfs to set up the volume right before mountall.
    Fedora uses Systemd rather than the traditional SysVinit. I'm still getting the hang of where things go I'm afraid, but /etc/systemd (which overrides and adds to defaults in /usr/lib/systemd) and /etc/udev (which overrides and adds to defaults in /usr/lib/udev) replace most of the /etc/rc.d configuration

    No idea where Fedora does mountall...
    I don't think it does. Systemd reads /etc/fstab itself and mounts the individual file-systems when they are first needed.

    I remember that there is some catch, some reason why btrfs volumes are not set up automatically on boot, but I've done my best to forget everything about btrfs.
    Single Btrfs devices are fine, but to mount a multi-device Btrfs, "btrfs device scan" must be run first to build the kernel's list of which devices are pooled together. The alternative is to explicitly list all devices in a pool with "mount /dev/xxx -o device=/dev/xxx,device=/dev/yyy,... /mntpnt", which can also be done in /etc/fstab but is rather ugly.

    EDIT: My problem is that the device scan is run correctly before / is mounted, but it isn't run after the LVM is started, so I've had to resort to the "device=..." mount option.

    So no crypto, even now. ZFS has it.
    It's on the to-do list, but unless you want per-file encryption, dm-crypt is more secure and still the recommended way to do it. In the meantime, eCryptFS allows per-directory encryption on any file-system.

    I know it looks OK now, but you will be sorry. And when you complain to the listserv they are guaranteed to remind you that IT IS EXPERIMENTAL (even after 4 years of so-called development).

    I tried to tell you...
    That's okay, I'm experimental too.
    Last edited by Gareth Jones; 28th June 2012 at 04:42 PM.

  6. #6
    Join Date
    Jun 2012
    Location
    Seattle, Ecotopia
    Posts
    33
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    Re: Multivolume encrypyted Btrfs RAID

    Quote Originally Posted by Gareth Jones
    Well I've resigned myself to using explicit "defaults,device=...,device=..." mount options in /etc/fstab. It's ugly, but it works.
    Yes, this is absolutely necessary.



    Quote Originally Posted by Gareth Jones
    Fedora uses Systemd rather than the traditional SysVinit. I'm still getting the hang of where things go I'm afraid, but /etc/systemd (which overrides and adds to defaults in /usr/lib/systemd) and /etc/udev (which overrides and adds to defaults in /usr/lib/udev) replace most of the /etc/rc.d configuration
    Thank you, I sure needed this. I'm taking the RHCSA exam tomorrow. Been running Debian exclusively for 14 years, but weak on RH-specific stuffs.

    Just passed my CEH Monday!

Similar Threads

  1. Separate (encrypted/btrfs-raid) /var and F17
    By Gareth Jones in forum Installation, Upgrades and Live Media
    Replies: 0
    Last Post: 4th June 2012, 02:25 AM
  2. Partitioning Software RAID, LVM and BTRFS
    By smannem in forum Servers & Networking
    Replies: 3
    Last Post: 4th May 2012, 07:55 AM
  3. [SOLVED]
    Btrfs RAID 1 Problem
    By SiriusStarr in forum Using Fedora
    Replies: 0
    Last Post: 4th October 2011, 07:19 AM
  4. [SOLVED]
    BTRFS and F16
    By tox in forum F16 Development
    Replies: 10
    Last Post: 10th August 2011, 12:48 AM
  5. Replies: 0
    Last Post: 4th July 2011, 01:21 AM

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •