Fedora Linux Support Community & Resources Center
  #1  
Old 8th February 2013, 03:51 PM
drinkingcoffee Offline
Registered User
 
Join Date: Jun 2009
Posts: 46
linuxchrome
upgrade form f16 to f18 with LUKS/LVM/RAID

Hi --

My system is currently still running f16 and is ready to update to 18.
I'm planning on doing a fresh install this time, since I understand things have changed quite a bit between 16 -> 17, and I was never able to get preupgrade working because of my setup.

my current setup is:
- two RAID1 devices (two drives total), with md0 for /boot and md1 for everything else
- lvm on md1, with separate volumes for /home swap and /
- encryption with LUKS on md1 and the three logical volumes (so, 4 entries total in my /etc/crypttab )


here's my question(s):

- I don't know exactly what I was thinking when I installed with that scheme, but it seems a bit crazy. Specifically, it should have been sufficient (and better?) to only turn on the encryption for the md1 device, right? In this current scheme, am I running through multiple layers of encryption? If so, that sounds like it would be a decent performance hit, no? Or does it not matter so much?

- I would also like to add a 3rd drive (not on the raid) to do backups. What is the best way to handle encryption for this backup drive? I'd ideally like to only enter one password when booting up, though it's not horrible if I have to enter it twice, I suppose. It should be in its own Volume group? maybe I don't even need to bother with lvm for this drive?


here's what I am thinking:

- for two of the drives, the same RAID1 setup, with md0 unencrypted as /boot and md1 encrypted
- lvm using md1 to create logical volumes for /home / and swap (no extra encryption layers here), and using the 3rd drive to create a logical volume (encrypted) in its own volume group (?) for /backup

Now, to get there, I am thinking:

- create encrypted volume group, and volume (encrypted) for /backup on the 3rd drive. Create the backup of everything from my /home to the 3rd drive.
- install f18 on the two drives with the raid/lvm scheme above
- restore the backup of /home to the new f18 system.



Is there a better way? I understand LVM can support some raid functionality on its own now, but it's pretty new...
Am I making any stupid assumptions/mistakes?

thanks
Reply With Quote
  #2  
Old 9th February 2013, 10:56 AM
chrismurphy Offline
Registered User
 
Join Date: May 2010
Posts: 969
macoschrome
Re: upgrade form f16 to f18 with LUKS/LVM/RAID

Fedora 18's GUI installer doesn't have a way to do LVM on RAID. I think it will recognize, I think it will see VG/LV's on md raid if you configure them both in advance from command line. I'm going to guess you're at least semi-comfortable with this, because if you had a drive meltdown, you'd need to use CLI to do recovery.

To get LVM-like and raid1 features together in the GUI, you'd need to use btrfs, actually. So you'd need to be comfortable with that. But the same thing applies, you'd need familiarity with btrfs commands for disaster recovery should a drive die.

Quote:
encryption with LUKS on md1 and the three logical volumes
Totally unnecessary, yes your are taking a performance hit. If your CPU has AES-NI, the hit is much less, maybe 3-5% cpu for one layer of encryption. It's maybe 4x that if your CPU doesn't support AES-NI. Double it for your case because you have two layers of encryption.

To have this same arrangement: md raid > encryption > LVM > file system, you will need to do the first three steps with the command line first. You ready for that?

---------- Post added at 02:11 AM ---------- Previous post was at 01:51 AM ----------

Quote:
Originally Posted by drinkingcoffee View Post
I would also like to add a 3rd drive (not on the raid) to do backups. What is the best way to handle encryption for this backup drive? I'd ideally like to only enter one password when booting up, though it's not horrible if I have to enter it twice, I suppose. It should be in its own Volume group? maybe I don't even need to bother with lvm for this drive?
LVM would let you add this 3rd drive to the Home LV. The gotcha is if the 3rd drive dies, you end up with a corrupt /home, defeating putting /home on raid1. So you probably don't want to do this. You probably want it separate, and if it's separate, you have to configure it separately, you can't do it in anaconda.

Quote:
- for two of the drives, the same RAID1 setup, with md0 unencrypted as /boot and md1 encrypted
Fine. This means any LVs you create on md1 are encrypted, including swap. Conversely you could encrypt the LVs, meaning you encrypt just /home. Or /home and swap. And not rootfs. It's an option.

Quote:
using the 3rd drive to create a logical volume (encrypted) in its own volume group (?) for /backup
You'll need to deal with it separately, but this is possible. In this case I would not partition the disk, just make the whole disk a PV, add it to a VG, create your LV(s), encrypt them. That way if you want to expand or migrate, you can use pvmove to move encrypted extents, rather than having to do a file copy from one encrypted volume to another. It's a minor point. You don't even really have to use LVM here, it's not a major plus unless you expect to expand this in the near future.

Quote:
I understand LVM can support some raid functionality on its own now, but it's pretty new...
It works fine, and has been around for quite a while actually. The thing that would favor LVM mirroring is if reads work like md raid i.e. if you read two files at the same time, md raid will order the LBAs for those files from different disks, increasing performance. Last I checked (couple years old) LVM only reads from one drive at a time until there's a failure. I think you'll find more information and assistance with disaster recovery with md raid.

---------- Post added at 03:56 AM ---------- Previous post was at 02:11 AM ----------

Yeah so I just tested this and it's possible, but it's really non-obvious how to get anaconda to cooperate. And the only possible way I could find to do it is separately encrypted LVs, rather than an encrypted md1 (root, home, swap).

Basically I partitioned both disks GPT, creating: swap, md0(boot), md1(all) using gdisk. Identical sizes for all of them.

I used mkswap on the two swap partitions, and swapon for both. Normally swap goes in LVM by default, but for raid1 if you ever do scrub checks, you get a pile of errors because the swap mirrors aren't ever the same (read man 4 md). So I left them out of LVM, on their own. (I didn't try this but it looked like in anaconda once they're auto added, I can choose to reformat them as swap, and check encryption. That's the thing to do.)

Then created md1 with mdadm. Made it a PV. Created a VG. Created two LVs, home and root. If they aren't created in advance, I get messed up behavior from anaconda. If no LV is created, it would sometimes not see the VG, and even if it did I could only create one LV of any size then it claimed no more space was left. If I created one, I could reuse it, but couldn't create another one in the remaining space. So I just created both of them with lvcreate.

All of this is done in shell before passing the first (language) page of anaconda.

Next, Installation Destination, choose the two disks, customize partitioning. The LVs show up under +Unknown. I click each, check to reformat them, specify a mount point in the Mount Point field, and check encryption for home (I was able to encrypt root if I wanted also).

At some point the two swaps were added to +New Fedora 18 Installation on their own, yet they also appear as partitions under +Unknown. Confusing but manageable.

I clicked on each boot, and then minus to delete them. Then + to make a new boot of the same size, +customize to change from Standard to RAID device type, and uncheck optimize and check redundancy, click apply. I do this because anaconda won't respect prior RAIDs, it forces them to be remade regardless...unless they have LVM on them already.

Finish Partitioning. Error. I forgot to create BIOS Boot. I went back to Manual Partition to do that, and I can reduce the size of Boot to make a BIOS Boot partition. Turns out it makes only one BIOS Boot partition, and when anaconda tries to install grub to both drives, it succeeds on sda, but fails on sdb and then crashes. Bug 895644.

Since grub2-mkconfig is called by anaconda after grub2-install, a grub.cfg is not created. So the system isn't yet bootable. I chrooted /mnt/sysimage, and used grub2-mkconfig. Rebooted, and it is bootable.

And the encrypted home mounts also, asking for a password during startup. So it all worked, but it was a bit of work figuring out what was needed in advance and in what order.
Reply With Quote
  #3  
Old 9th February 2013, 04:16 PM
drinkingcoffee Offline
Registered User
 
Join Date: Jun 2009
Posts: 46
linuxchrome
Re: upgrade form f16 to f18 with LUKS/LVM/RAID

Quote:
Originally Posted by chrismurphy View Post
Fedora 18's GUI installer doesn't have a way to do LVM on RAID. I think it will recognize, I think it will see VG/LV's on md raid if you configure them both in advance from command line. I'm going to guess you're at least semi-comfortable with this, because if you had a drive meltdown, you'd need to use CLI to do recovery.
Yeah, after reading about the F18 installer with regards to LVM+RAID+LUKS, I've come to the conclusion that some manual setup will be required. That's no problem, though. I seem to spend half my waking life working on the command-line. The various mdadm/luks/lvm commands are just things that I don't have to use often enough (thankfully) that I need to look them up each time.

But, it's been a few years since I've set this up, so I now know enough to see that I made mistakes on the details last time. I want to do it well enough this time that I don't need to worry about it again for at least that long.

Quote:
To get LVM-like and raid1 features together in the GUI, you'd need to use btrfs, actually.
That's actually very interesting. I haven't looked much at btrfs (GUI tools or not). Is btrfs generally considered 'safe enough' to use at this point?
I'm a bit unclear on the ordering of LUKS and btrfs, though, if I'm using the 'raid1' built-in to btrfs. I'd want to luksFormat (say) /dev/sda1 and /dev/sda2, then create a btrfs 'raid1' device using those two encrypted partitions?

ie:

Code:
mkfs.btrfs -d raid1 /dev/mapper/encrypted_sda1 /dev/mapper/encrypted_sda2
from https://btrfs.wiki.kernel.org/index...._encryption.3F I see that this might result in reduced performance compared to a md-raid 1 setup, where the encryption is on top of the raid. It sounds like this would mean data is encrypted separately for each drive as it's written to disk?
This is what the installer seems to do, right? (that's what it looks like, anyway, from playing with it in a vm)

Does running LUKS encryption on top of btrfs make more sense? It's not clear (for me, at least) from that FAQ if that's a good or a bad thing to try (though I can see that it makes sense to turn compression off in this case).

Quote:
Yeah so I just tested this and it's possible, but it's really non-obvious how to get anaconda to cooperate. And the only possible way I could find to do it is separately encrypted LVs, rather than an encrypted md1 (root, home, swap).
Thanks so much for this information (and for actually trying it out!). That's extremely helpful.
I'm going to try playing around doing an install in a VM right now to see what happens.

After some more research, I'm pretty tempted to use btrfs now. As long as I keep my /backup partition in LVM+ext4, I'm not too worried about running something EXPERIMENTAL (if I wanted safe/stable, I'd be running redhat/centos instead of the latest fedora, anyway ;-)

Some new questions related to btrfs...

- is /boot on btrfs (raid1, but unencrypted) supported? Will this give me headaches with GRUB?
- is putting / on a btrfs raid0 (encrypted, even if it's not strictly necessary) worth the increased risk?
- what about swap. is a btrfs-raid0 swap partition useful compared to two separate swap partitions or does the system do some fancy load-balancing with multiple swap partitions already?


cheers and thanks again!
Reply With Quote
  #4  
Old 9th February 2013, 10:35 PM
chrismurphy Offline
Registered User
 
Join Date: May 2010
Posts: 969
macoschrome
Re: upgrade form f16 to f18 with LUKS/LVM/RAID

Quote:
Originally Posted by drinkingcoffee View Post
Is btrfs generally considered 'safe enough' to use at this point?
It's considered stable on stable hardware, but it's also still considered experimental. But it's included in Oracle's Unbreakable Kernel. But users do continue to find edge cases for various problems on the btrfs list. I don't know how else to answer the question, haha. Btrfs is still probably less tolerant in edge cases, like bad RAM or a bad sector or unclean shutdown, etc. Most of these sorts of issues including write barriers are fixed, but you get some combination of events and you can have problems. In 3+ years, I never have. But I do read linux-btrfs@ and see the edge cases roll in.

If you keep judicious backups, then I'd say benchmark it for your use case and see if it meets with your expectations compared to what you've been using. Chances are it's faster than two layers of dmcrypt for all use cases!

Quote:
I'm a bit unclear on the ordering of LUKS and btrfs, though
Oh yeah right, I spaced out the encryption part, and I haven't tested that combo, so I'm not sure that anaconda gets this right.

It would need to create /boot on ext4 on md raid1. The rest of the space on the disks would first be partitioned, encrypted, and then the encrypted dm devices added to create a btrfs raid1 volume.

Quote:
I see that this might result in reduced performance compared to a md-raid 1 setup, where the encryption is on top of the raid.
I haven't benchmarked the two, so I'm not sure, what the impact is.

Quote:
It sounds like this would mean data is encrypted separately for each drive as it's written to disk?
Yeah it's a case of 1 stream encrypted, then duplicated to the drives; vs that stream duplicated into 2 streams, then encrypted, to the drives. The duplication of an encrypted stream is cheap. But the encryption of two streams is expensive. But you've already been doing this. And depending on your workload you may not even notice the performance hit, until you have a process that needs those CPU cycles that are otherwise consumed with extra encryption.

Quote:
Does running LUKS encryption on top of btrfs make more sense?
Not possible, as it's a dm block device. To encrypt on top of btrfs you'd need file based encryption. There is such a thing, ecryptfs.

Quote:
As long as I keep my /backup partition in LVM+ext4, I'm not too worried about running something EXPERIMENTAL
Yeah what I would do is make one copy on another drive, and shelve it. Then use this 3rd drive you were talking about as your day to day backup, use rsync or something. If you have the space, create two LVs on that drive and have a script that does daily (or hourly, whatever) rsyncs for a week to one LV; then switches and does rync to the other LV. That way you have two backups on one disk with some aging built in (except obviously on switch days you'd have only a day of aging).

Quote:
- is /boot on btrfs (raid1, but unencrypted) supported? Will this give me headaches with GRUB?
Supported? Depends on who you ask. I think for anaconda, they're saying not recommended. However I've tested /boot on btrfs multidevice single (linear), raid0,1,10 with and without encryption, and GRUB finds its files no matter what. It even finds them when /boot is in a btrfs subvolume. GRUB2 basically boots anything these days. By default anaconda does not create partitions for Mount Points when the device type is btrfs, instead it creates one btrfs volume, and makes the Mount Points subvolumes.

Your case is complicated by encryption. Anaconda doesn't configure GRUB for finding /boot encrypted with LUKS. I think this is possible but I haven't tried it and it seems complicated to say the least. So you're better off with /boot on ext4 on md raid1.

Quote:
- is putting / on a btrfs raid0 (encrypted, even if it's not strictly necessary) worth the increased risk?
Same risk as any raid0. Lose a drive, lose everything. Not implemented yet but eventually there will be different raid levels per subvolume on btrfs, so things get really customizable. But right now you'd need to make separate dm-crypt partitions for the raid0 btrfs and raid1 btrfs. I don't know that anaconda will help you with that, I haven't tried it. I think it's willing to make just one btrfs volume.

Quote:
- what about swap. is a btrfs-raid0 swap partition useful compared to two separate swap partitions or does the system do some fancy load-balancing with multiple swap partitions already?
Anaconda doesn't make swapfiles as far as I know. It will put swap on an LV, or on a partition. And it can be encrypted in either case. Presently btrfs doesn't support swapfiles anyway, so that's not something you can do post install either. And I'm spacing out how anaconda deals with swap partition on multiple devices. If you click on "create partitions automatically" in Manual Partitioning, I'm pretty sure you get one swap partition on one disk. I think you can create two swap partitions, and use the button to the right of the big + - buttons at the bottom of the Mount Point list, to choose which drive those partitions go on. So you can create two equal sized swap partitions. Question is whether it matters and I dont know:

If a drive dies while its swap partition is being used, the kernel is going to get pissed. So the other swap isn't really a backup of the first; they have different priorities in that if one is busy the other would get used. If you're really using that much swap, you need more RAM. On the other hand, if you make just one swap partition, now your remaining space on the other drive is mismatched and that space can't be used anyway. You can get fancy and make the 1st swap 500MB smaller since that disk also has a 500MB Boot partition. And then everything lines up.

I can tell you that it's not ideal to mirror a swap partition because md check scrubs will report mismatch errors. man 4 md for more info on why. But anaconda doesn't configure swap this way anyway.

Also, depending on what drives you have, make sure you check their SCT ERC capabilities:
smartctl -l scterc /dev/sdX

If it's not supported or settable below 300 deciseconds, you need to increase the value of the SCSI timeout layer, otherwise any read errors aren't corrected by md or btrfs.

Something like this works:
echo 120 >/sys/block/sdX/device/timeout

---------- Post added at 03:35 PM ---------- Previous post was at 01:36 PM ----------

Oh yeah, some benchmarks of some of this. It's SSD, and not raid 1 but might give you some idea what direction to go in.


http://www.mayrhofer.eu.org/ssd-linux-benchmark
Reply With Quote
Reply

Tags
f16, f18, form, luks or lvm or raid, upgrade

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Upgrade with fedup from 17 to 18 with luks / dm-crypt xonieh Installation, Upgrades and Live Media 0 16th January 2013 04:19 PM
[SOLVED] Installing with RAID, LUKS and LVM Pragmatikf Installation, Upgrades and Live Media 3 8th October 2012 12:20 AM
Boot to command line after upgrade form 4 to 6 noppie Installation, Upgrades and Live Media 2 7th December 2006 06:13 PM
How to install Fedora form ISO form harddisk? Stevenisme Installation, Upgrades and Live Media 3 5th March 2005 03:04 PM
Upgrade form 8.0 to FC2? bj_ EOL (End Of Life) Versions 3 7th November 2004 04:10 AM


Current GMT-time: 06:37 (Thursday, 24-04-2014)

TopSubscribe to XML RSS for all Threads in all ForumsFedoraForumDotOrg Archive
logo

All trademarks, and forum posts in this site are property of their respective owner(s).
FedoraForum.org is privately owned and is not directly sponsored by the Fedora Project or Red Hat, Inc.

Privacy Policy | Term of Use | Posting Guidelines | Archive | Contact Us | Founding Members

Powered by vBulletin® Copyright ©2000 - 2012, vBulletin Solutions, Inc.

FedoraForum is Powered by RedHat