PDA

View Full Version : HowTo: use bcache on a fresh F20 install



l3iggs
23rd December 2013, 08:59 AM
Hello Fedora people.

bcache is cool. It's a recent (3.10) addition to the Linux kernel that allows you to use a ssd to cache reads and writes to/from your hdd in hopes of getting better I/O performance than you'd get from a hdd alone.

I have recently successfully setup bcache in my Fedora version 20 install. I am a brand new Fedora user here so bear with me. I'll try to describe how I did this here.

If your ssd is larger than 60GB, it probably only makes sense to use the first 64GB for your bcache, you'll likely not see any benefit to using a larger amount of ssd cache space and you can use the remaining ssd space for something more useful. Taking care of a larger ssd is done in step 4a below, if your ssd is 60GB or smaller, just skip step 4a.

Disclaimer
You should assume that following this guide will leave you with only zeros in all of the drives attached to your machine. Also note that even if you get things set up properly here, you'll be in an inherantly precarious situation. Following this guide will have you running an experimental file system (btrfs) on an experimental caching system (bcache) which can be impacted by hardware errors on either one of two physical drives.
Assumptions

Your machine is set up for BIOS type booting (not EFI, there are probably simple modifications you can make to get this to work under a EFI boot scenario, but I don't know them)
/dev/sda is the rotating magnetic disk with nothing important on it you want to cache with a ssd
/dev/sdb is your ssd with nothing important on it you intend to use as a cache for /dev/sda

Do this
Step #1: Boot from F20 live media.
Step #2: Choose "Try Fedora" and open a terminal.
Step #3: Install bcache tools and gparted.

sudo yum install bcache-tools gparted
Step #4: Repartition your HDD.

sudo gparted
Then use gparted to:
- Create a new gpt partition table for /dev/sda (msdos works too but it's old and may cause problems later)
- Make a new partition: /dev/sda1 cleared, 2MiB, bios_grub flag set (this is needed to boot with BIOS from gpt disks)
- Make a new partition: /dev/sda2 ext4, 512MiB (this will be your /boot partition)
- Make a new partition: /dev/sda3 swap
- Make a new partition: /dev/sda4 cleared (this will be part of your bcache)
You should figure out the proper sizes for /dev/sda3 and /dev/sda4 yourself
Step #5: Repartition your SSD.
Still using gparted:
- Create a new gpt partition table for /dev/sdb (msdos works too but it's old and may cause problems later)
- Make a new partition: /dev/sdb1 cleared (this will be part of your bcache)
I recommend that if your ssd is larger than 64 GB you make /dev/sdb1 only 64 GB in size (making it bigger is probably a waste). You can use the rest of your ssd for someting else more useful.
Step #5.1: Remove any existing file systems on the targets.
In case you're not following this guide exactly and you have junk left over in your target partitions from something else, clean them up like this. You shouldn't need to do this if you just made them "unformatted" as described in steps 4 and 5 above


sudo wipefs -f -a /dev/sda4
sudo wipefs -f -a /dev/sdb

Step #6: Create your bcache.

sudo make-bcache --wipe-bcache -w512 --discard --writeback -B /dev/sda4 -C /dev/sdb
Your hybrid hdd/ssd bcache will now appear as /dev/bcache0, we'll install Fedora there later.
Step #7: Put a file system in your bcache.

sudo mkfs.btrfs /dev/bcache0
Here I use btrfs because it's new and shiny and I like shiny new things. You can use whatever filesystem you'd like to (ext4 is the more conservative choice here, but if you're following this guide you're not very conservative, are you?). The rest of the guide assumes you chose btrfs.
Step #8: Start the Fedora installer
Step #9: Click some things in the installer

Click the "Installation Destination" button
select bcache0 and sda (black check marks)
click "Full disk summary and bootloader..." in the lower left hand corner
Make sure your hdd (NOT YOUR SSD) has a green check mark indicating that grub will be installed there
Click the Done button (upper left corner)
Choose BTRFS from the Partition scheme dropdown
Select I want to review/modify my disk partitions before continuing.
Click the Continue button
Expand Unknown
Click the sda2 partition, set Mount point as /boot, check Reformat, Click Update Settings button
Click the swap partition, check Reformat, Click Update Settings button
Click the sda1 partition, check Reformat, Click Update Settings button
Click the Plus (+) button in the lower left, select "/" for Mount Point:, click Add mount point button, Choose btrfs... in the Volume: dropdown menu, Click Update Settings button
Click the Plus (+) button in the lower left, select "/home" for Mount Point:, click Add mount point button, Choose Btrfs 1.1 in the Volume: dropdown menu, check Reformat, Click Update Settings button
Click the Done button (upper left corner)
Click the Accept Changes button

Step #11: Finish the install as you would normally.
Step #12: Chroot into your new install.
Once the install has completed,

sudo chroot /mnt/sysimage
Step #13: install bcache-tools

yum install bcache-tools
This puts bcache-tools into your new install.
Step #14: Rebuild initramfs.

yum update kernel
or

dracut -f
This will update the kernel which will trigger a regeneration of your initramfs which will now pull in the proper udev rules (since we've installed bcache-tools) to allow your system to boot properly going forward.
Step #15: Reboot into your new bcached system.

That's it, just 15 easy steps!

You should now have a fully functioning system. All of this should be one-time and you should never have to do any of this again for this install.

Note you can get various performance stats on your bcache like this

bcache-status

Thanks to emesix for the great chroot idea.

Good luck!

emesix
26th December 2013, 12:09 AM
Hi there

Thanks for the guide!!!

I made it work with the KDE version of Fedora 20.
Only thing that didnt work was using anaconda from shell. Got a old beta installer that failed.
After i used the installer from the desktop and finished installing.
I didnt reboot but entered the command: sudo chroot /mnt/sysimage
And skipped step 12 to 15 completely.

Sorry for mine bad english.
Vincent

l3iggs
26th December 2013, 07:34 AM
I'm really glad it worked for you!

I like your idea to chroot to the install directory. I'll update my guide with your suggestions. *done*

Since I setup my bcache install the other day, my system has been running great with no issues. Things seem pretty quick too! (sorry nothing quantitative here)

lsatenstein
28th December 2013, 06:55 AM
Can you describe your system hardware. My system is using sata2 drives, an older INTEL E7300 64bit dual core system.

As well, what size SSD did you use? For bcache, would a 60 gig one would be more than adequate?

l3iggs
28th December 2013, 08:44 AM
Can you describe your system hardware. My system is using sata2 drives, an older INTEL E7300 64bit dual core system.

As well, what size SSD did you use? For bcache, would a 60 gig one would be more than adequate?

I did the install with only two sata drives attached to my ~4 yr old amd64 system running a Core i7 930. /dev/sda is a fairly old 400GB magnetic drive (ST3400620AS) and /dev/sdb is a fairly old 60GB ssd (SAMSUNG MMCRE64G5MXP-MVB (VBM19C1Q)). Both drives are directly connected to the intel x58 based motherboard.

I don't know exactly how bcache decides what to keep on the ssd. I'd imagine it's quite complicated. After 5 days of heavy use (gaming, compiling linux kernel, running several virtual machines), bcache-status shows that I've used 28% of my cache (see below). This seems to be holding steady for the last few days and I've seen the usage fluctuate up and down +/- 3GB with typical use.

I actually think 60GB is just about right for this. An Anandtech article (http://www.anandtech.com/show/4329/intel-z68-chipset-smart-response-technology-ssd-caching-review/2) I was reading on the subject (a similar tech, Intlel's "Smart Response") mentions Intel actually limits their cache to 64GB because "it saw little benefit in internal tests to making the cache larger than that." So if I had a ssd larger than 60 GB, I would modify my above guide by repartitioning the ssd so that the bcache uses only 64 GB of it [done]. I think that using more than 64 GB of ssd space for the bcache is actually a waste since the cache will likely not use more space than that.

Hope that answers your question.

Here's my output of bcache-status

--- bcache ---
UUID 3ab74457-a17b-4303-9d36-d28e835609ab
Block Size 512B
Bucket Size 512.00KiB
Congested? False
Read Congestion 2.0ms
Write Congestion 20.0ms
Total Cache Size 59.63GiB
Total Cache Used 16.70GiB (28%)
Total Cache Unused 42.93GiB (72%)
Dirty Data 1.30MiB (0%)
Evictable Cache 59.63GiB (100%)
Replacement Policy [lru] fifo random
Cache Mode writethrough [writeback] writearound none
Total Hits 409467 (83%)
Total Misses 79029
Total Bypass Hits 255269 (100%)
Total Bypass Misses 0
Total Bypassed 7.60GiB

emesix
29th December 2013, 12:01 PM
Hi there

Well mine system is a AMD Athlon x4 system. So its the slower SATA.
I have a 2TB WD green drive as the magnetic disk and a 160GB OCZ SSD.



Disk /dev/sda: 1.8 TiB, 2000398934016 bytes, 3907029168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: dos
Disk identifier: 0x000f159d

Device Boot Start End Blocks Id System
/dev/sda1 2048 2099199 1048576 83 Linux
/dev/sda2 2099200 69208063 33554432 82 Linux swap / Solaris
/dev/sda3 69208064 3907029167 1918910552 83 Linux


Disk /dev/sdb: 167.7 GiB, 180045766656 bytes, 351651888 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes




--- bcache ---
UUID b0a856c4-5ae4-4df1-bfe5-4df1bd9639fb
Block Size 512B
Bucket Size 512.00KiB
Congested? False
Read Congestion 2.0ms
Write Congestion 20.0ms
Total Cache Size 167.68GiB
Total Cache Used 21.80GiB (13%)
Total Cache Unused 145.88GiB (87%)
Dirty Data 1.70MiB (0%)
Evictable Cache 166.00GiB (99%)
Replacement Policy [lru] fifo random
Cache Mode writethrough [writeback] writearound none
Total Hits 364564 (83%)
Total Misses 70033
Total Bypass Hits 129578 (100%)
Total Bypass Misses 0
Total Bypassed 20.60GiB


It works perfectly :) you would never guess i am using a very slow hard-disk from mine raid system. :)
There is 1 thing thats strange.... The system is complaining about not enough room on /tmp. :confused:
Only had it on fedora with K3b (DVD burning) but had something like that in Arch linux while compiling.

Vincent

lsatenstein
30th December 2013, 02:26 AM
I did the install with only two sata drives attached to my ~4 yr old amd64 system running a Core i7 930. /dev/sda is a fairly old 400GB magnetic drive (ST3400620AS) and /dev/sdb is a fairly old 60GB ssd (SAMSUNG MMCRE64G5MXP-MVB (VBM19C1Q)). Both drives are directly connected to the intel x58 based motherboard.

I don't know exactly how bcache decides what to keep on the ssd. I'd imagine it's quite complicated. After 5 days of heavy use (gaming, compiling linux kernel, running several virtual machines), bcache-status shows that I've used 28% of my cache (see below). This seems to be holding steady for the last few days and I've seen the usage fluctuate up and down +/- 3GB with typical use.

I actually think 60GB is just about right for this. An Anandtech article (http://www.anandtech.com/show/4329/intel-z68-chipset-smart-response-technology-ssd-caching-review/2) I was reading on the subject (a similar tech, Intlel's "Smart Response") mentions Intel actually limits their cache to 64GB because "it saw little benefit in internal tests to making the cache larger than that." So if I had a ssd larger than 60 GB, I would modify my above guide by repartitioning the ssd so that the bcache uses only 64 GB of it. I think that using more than 64 GB of ssd space for the bcache is actually a waste since the cache will likely not use more space than that.

Hope that answers your question.

Here's my output of bcache-status

--- bcache ---
UUID 3ab74457-a17b-4303-9d36-d28e835609ab
Block Size 512B
Bucket Size 512.00KiB
Congested? False
Read Congestion 2.0ms
Write Congestion 20.0ms
Total Cache Size 59.63GiB
Total Cache Used 16.70GiB (28%)
Total Cache Unused 42.93GiB (72%)
Dirty Data 1.30MiB (0%)
Evictable Cache 59.63GiB (100%)
Replacement Policy [lru] fifo random
Cache Mode writethrough [writeback] writearound none
Total Hits 409467 (83%)
Total Misses 79029
Total Bypass Hits 255269 (100%)
Total Bypass Misses 0
Total Bypassed 7.60GiB

Thank you, I am going to stick to a 60giger. Did you really notice performance improvements?

There was a complete tuning article for bcache, for which I posted the links.

And have a happy, healthy and lucky new year

---------- Post added at 08:26 PM ---------- Previous post was at 08:22 PM ----------


Hi there

Well mine system is a AMD Athlon x4 system. So its the slower SATA.
I have a 2TB WD green drive as the magnetic disk and a 160GB OCZ SSD.



Disk /dev/sda: 1.8 TiB, 2000398934016 bytes, 3907029168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: dos
Disk identifier: 0x000f159d

Device Boot Start End Blocks Id System
/dev/sda1 2048 2099199 1048576 83 Linux
/dev/sda2 2099200 69208063 33554432 82 Linux swap / Solaris
/dev/sda3 69208064 3907029167 1918910552 83 Linux


Disk /dev/sdb: 167.7 GiB, 180045766656 bytes, 351651888 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes




--- bcache ---
UUID b0a856c4-5ae4-4df1-bfe5-4df1bd9639fb
Block Size 512B
Bucket Size 512.00KiB
Congested? False
Read Congestion 2.0ms
Write Congestion 20.0ms
Total Cache Size 167.68GiB
Total Cache Used 21.80GiB (13%)
Total Cache Unused 145.88GiB (87%)
Dirty Data 1.70MiB (0%)
Evictable Cache 166.00GiB (99%)
Replacement Policy [lru] fifo random
Cache Mode writethrough [writeback] writearound none
Total Hits 364564 (83%)
Total Misses 70033
Total Bypass Hits 129578 (100%)
Total Bypass Misses 0
Total Bypassed 20.60GiB


It works perfectly :) you would never guess i am using a very slow hard-disk from mine raid system. :)
There is 1 thing thats strange.... The system is complaining about not enough room on /tmp. :confused:
Only had it on fedora with K3b (DVD burning) but had something like that in Arch linux while compiling.

Vincent

/tmp is for small files where there is no security risk. Fedora20 uses /var/tmp (or /var/userid/tmp so that your tmp files are secured.

l3iggs
30th December 2013, 03:37 AM
Did you really notice performance improvements?
I wish I had done some disk I/O testing for both my HDD or SDD alone before setting this up. Unfortunately I don't have any real numbers. Maybe someone else would like to contribute some speed testing results here?

I think a good test would be something like:
(a) Build the linux kernel from the HDD alone
(b) Build the linux kernel from the SDD alone
(c) Build the linux kernel from bcache0
(d) Compare times from a,b and c ;)

I do know that the HDD I have is rather loud. I notice that when I am doing a disk I/O intensive task, I can't hear the HDD at all, so that's a good sign I suppose.

I guess an important question to ask is what kind of overhead is the cache introducing? I don't know the answer to that.

emesix. Regarding running out of space in /tmp I don't think it's related to bcache. I believe tmpfs mounts are stored in ram so maybe it's time to buy some more ram or take some steps to prevent large amounts of data from ending up there. Issue a df command to look at your tmpfs usage to see if you're actually running out.

bobgus
18th February 2014, 11:36 PM
I have a new system with 32GB of ram and a couple of 2TB disks for raid 1. I was thinking of putting a PCIe SSD in between. Based on this thread, I wonder if a 60GB cache would have any value?