SSD drives under Linux - discussion thread

[jpollard]

You are missing the optimization done at the block layer to merge multiple writes into one very large write.

Things like read-ahead/write-behind also affect the actual I/O rates - read-ahead tries to predict data you will want. When it misses the extra reads are dumped, and read-ahead resumes from where you really wanted data. This causes extra work on reading, for a better average response rate. Write behind is part of the optimization to reduce fragmentation. The one place extra writes still occur is when recording access times (atime). This can be stopped by using the "noatime" mount option, but you also loose some meta-data. This meta-data is just the record of when a file was last accessed, which can make it hard to tell if a file has been idle and could be removed (as in put in an off-line archive because it hasn't been used).

I believe tmpfs is part of swap, so some swap activity may be due to I/O for /tmp.

[stevea]

Originally Posted by dd_wizard

I've been curious about something I read, but I can't find the page again. I think it was on thessdreview.com, or maybe anandtech. The article quoted either a Microsoft or Intel engineer who stated that between 50% and 60% of disk I/O operations were 4k (NTFS block size) writes. That may well be the case in Windows, but I'm wondering how accurately it applies to linux. Here's the closest thing I've been able to produce under F15:

Probably 99+% of all modern Linux disk I/O is in 4KB or greater increments.. By default all of the major Linux performance file systems are based on 4KB or larger blocks. Disk interface protocols have used the 512B LBAs (logical block addresses) since dinosaurs roamed the earth. The drive manufacturer consortium are recently considering creating a new standard with larger block sizes.

The claim that 50-60% of operations are 4K+ writes (as opposed to reads) makes little sense. I suspect that SSD articles are suggesting either that random 4KB write tests are an important differentiator between SSDs drives (controllers) *or* that truly random (unaligned) 4KB SSD I/O has a terrific performance penalty.

[...] According to the man page for sar, those sizes are 512 byte sectors.

This is a reporting issue. The POSIX compliant utilities require reports in 512B increments, which isn't particularly human-friendly. Also since disk devices all manage 512B LBAs - this is the most rational unit to measure (not report) disk block I/O in. Aside from bootup it's unlikely that any non-4KB I/O touches your disk, unless you purposely create it. If you align your files-system and swap partition - you should be in excellent shape to avoid alignment penalties.

[...] I'm really curious what other users see when /tmp, /var/tmp, and firefox's cache are on a physical drive.

dd_wizard

Here is an example from a rotating-disk system;:

12:00:01 AM       DEV       tps  rd_sec/s  wr_sec/s  avgrq-sz  avgqu-sz     await     svctm     %util
...
Average:          sda      3.18    193.21     49.26     76.28      0.05     15.04      2.34      0.74
Average:          sdb      2.92     71.81     38.31     37.74      0.04     13.63      1.67      0.49

Incidentally sdb is a backup drive and nearly all the file writes are small incremental diffs. Both drives, all filesystems are mounted 'noatime'. Swap is on sda.

Originally Posted by jpollard

You are missing the optimization done at the block layer to merge multiple writes into one very large write.

Yes. A good deal of filesystem design involves trying to create opportunities for multi-block contiguous I/O. An ext4 fs extents can *potentially* cause a 128MB sequential read/write. ext3 has some more limited clustering features. Also the I/O queues attempt to coalesce contiguous I/O into a single operation.

I believe tmpfs is part of swap, so some swap activity may be due to I/O for /tmp.

I'm not sure how to read that comment. tmpfs is unrelated to swap. It's similar to but distinct from /dev/ram, where tmpfs permits dynamic variability in the allocated size. tmpfs pages can be swapped out, but so can much of DRAM data. The kernel doesn't select tmpfs pages for swap on any special basis. Certainly if there is dross in /tmp (for example file downloaded days ago) then they are likely candidates for any swapping that is necessary.

http://www.kernel.org/doc/Documentat...tems/tmpfs.txt

[jpollard]

I'm not sure how to read that comment. tmpfs is unrelated to swap. It's similar to but distinct from /dev/ram, where tmpfs permits dynamic variability in the allocated size. tmpfs pages can be swapped out, but so can much of DRAM data. The kernel doesn't select tmpfs pages for swap on any special basis. Certainly if there is dross in /tmp (for example file downloaded days ago) then they are likely candidates for any swapping that is necessary.

The only selection the kernel can do is based on least recently used pages... which will tend to be tmpfs. And that is the relationship between tmpfs and swap. I wasn't meaning it is explicitly tied to swap.

Of course, that depends on how much stuff is thrown into tmp - and that can be surprisingly large.

I have seen systems killed due to accumulated garbage in tmpfs. The crashes only stopped when a real disk partition was assigned (30GB at the time, swap was make 36GB, and users were severely limited in processes and memory). I should say this was a server with 128GB of physical memory, 128 CPUs and with about 500 users - with only about 100 concurrent users. Management wasn't happy about it, but they were less happy with the daily crashes.

[stevea]

Originally Posted by jpollard

The only selection the kernel can do is based on least recently used pages... which will tend to be tmpfs. And that is the relationship between tmpfs and swap. I wasn't meaning it is explicitly tied to swap.

Of course, that depends on how much stuff is thrown into tmp - and that can be surprisingly large.

I have seen systems killed due to accumulated garbage in tmpfs. The crashes only stopped ...

This is intended to be a sticky thread about practical use of SSDs under Linux, not a debate on side-topics. Unless you have a question, answer or added information related to SSDs, please start another thread on your topic or PM me.

You already pointed to the tmpfs issue in post 5 and it was addressed in post 8. The use of tmpfs for /tmp and /var/tmp is not specific to SSD drives, does potentially occupy swap, and yes exceeding swap limits is a well characterized admin issue unworthy of 5 posts.. LRU is certainly NOT the sole basis for page swapping - but that is a topic for another thread too,

[hefeweizen]

Thanks for the great information.

Originally Posted by stevea

No It's unclear/doubtful that any file-system atop an LVM block manager
or a RAID block manager can support TRIM at this time.
Therefore you should not use LVM nor RAID (mdadm) atop an SSD for
optimal performance unless you investigate the issue.

I found this link, https://www.redhat.com/archives/dm-d.../msg00035.html, that indicates that LVM atop SSD will be supported in the 2.6.38 kernel of Fedora 15. First, please confirm I'm understanding correctly. But secondly, are there any other changes that should be considered for running F15 with SSDs?

[dd_wizard]

This post isn't about usability, but I've been seeing interesting movement on the pricing front. Supermicro has been OEMing the Micron P300 enterprise class SSD for just over $10/GB for a while now. Today I found the new Hitachi SAS2, SLC enterprise drives for just over $4/GB! That used to be the high end for consumer grade products. A price war would certainly be welcome.

dd_wizard

[stevea]

Originally Posted by hefeweizen

Thanks for the great information.

I found this link, https://www.redhat.com/archives/dm-d.../msg00035.html, that indicates that LVM atop SSD will be supported in the 2.6.38 kernel of Fedora 15. First, please confirm I'm understanding correctly. But secondly, are there any other changes that should be considered for running F15 with SSDs?

Yes. It appears that as of 2.6.37 that devices under the device-mapper do support the discard option.
This is good news.

[synic]

Hi Stevea,

Nice thread. Hope this isn't off topic but:

I have a thread F15 - 64g SSD - partitioning / space usage suggestion and am wondering about the best performance I can get out of a 64g SSD?

I know I need swap, / and /home but am wondering about the location of /var and /usr and /tmp?

Silly question maybe but can I specify to use /tmp in swap? Or is this post over at SSDReviewForums specifying the exact issue? It looks very interesting and I think I understand the basics but I have to admit it is all starting to look double dutch to me. If I am going to specify sending /tmp to /tmpfs, and with only 2 gig ram, then am I going to want to specify a much larger swap than the 3 gig I have already partitioned? Maybe even 4 or 5 gig. I don't really care about the life span of a 64g ssd. I just want to wring as much system speed out of it as possible.

[dd_wizard]

My usage is word processing, firefox with 50 sites open in multiple tab groups, occasional rpm builds, gnumeric, fm radio usually running, etc. With 2GB of ram, 1.5GB swap, and /tmp and /var/tmp in tmpfs, I've never run into RAM problems. This is what top is showing atm, notice the big user is firefox:

top - 14:31:29 up  4:27,  2 users,  load average: 0.26, 0.23, 0.24
Tasks: 168 total,   2 running, 165 sleeping,   0 stopped,   1 zombie
Cpu(s): 19.2%us,  3.7%sy,  0.0%ni, 77.0%id,  0.0%wa,  0.1%hi,  0.1%si,  0.0%st
Mem:   2047896k total,  1818940k used,   228956k free,    45832k buffers
Swap:  1572860k total,   234944k used,  1337916k free,   505368k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                            
 9384 gene      20   0 2392m 672m  34m S 25.7 33.6   7:39.34 firefox                            
 9469 gene      20   0  418m 100m  17m R  5.5  5.0   1:42.99 plugin-containe                    
 1048 root      20   0  183m  20m 8084 S  4.0  1.0   8:02.55 Xorg                               
 5543 gene       9 -11  342m  29m  26m S  3.5  1.5   8:04.64 pulseaudio                         
 2569 gene      20   0 1625m  38m  11m S  2.7  1.9   6:36.54 exaile                             
 2126 gene      20   0  243m 7268 3312 S  2.0  0.4   2:20.27 compiz                             
 2077 gene      20   0  511m  10m 5920 S  1.5  0.5   4:05.94 python                             
 2288 gene      20   0  571m  11m 7044 S  0.7  0.6   0:07.89 gnome-terminal                     
 1967 gene      20   0  424m 9276 5776 S  0.4  0.5   1:09.37 avant-window-na                    
 5904 gene      20   0  508m  18m  10m S  0.3  0.9   0:15.03 awn-applet                         
 2073 gene      20   0  480m 7832 5064 S  0.2  0.4   0:34.44 awn-applet                         
 2074 gene      20   0  523m  10m 6256 S  0.2  0.5   0:43.32 python                             
 2080 gene      20   0  536m  14m 6004 S  0.2  0.7   0:33.36 python                             
 5194 root      20   0     0    0    0 S  0.1  0.0   0:02.98 kworker/u:2                        
12957 gene      20   0 15124 1224  884 R  0.1  0.1   0:00.02 top

dd_wizard

---------- Post added at 02:36 PM ---------- Previous post was at 02:32 PM ----------

P.S. I did run out of RAM once when I had a lisp program generate all the solutions for N-Queens 13 or 14. You wouldn't believe how slow it ran once the return list started being built in swap.

dd_wizard

[synic]

Originally Posted by dd_wizard

My usage is word processing, firefox with 50 sites open in multiple tab groups, occasional rpm builds, gnumeric, fm radio usually running, etc. With 2GB of ram, 1.5GB swap, and /tmp and /var/tmp in tmpfs, I've never run into RAM problems. This is what top is showing atm, notice the big user is firefox:

...
---------- Post added at 02:36 PM ---------- Previous post was at 02:32 PM ----------

P.S. I did run out of RAM once when I had a lisp program generate all the solutions for N-Queens 13 or 14. You wouldn't believe how slow it ran once the return list started being built in swap.

dd_wizard

hi dd_wizard.

Can I please have a geezer at your fstab?

Working SSD fstab's seem to be a little tricky to find.

[leigh123linux]

Here's my fstab, the highlighted line is the ssd partition.

[root@main_pc leigh]# cat /etc/fstab

#
# /etc/fstab
# Created by anaconda on Sun Feb 27 18:03:34 2011
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
UUID=91faae69-0fa6-42d8-8d94-1d0070678d8a /                       ext4    defaults,discard        1 1
tmpfs                   /dev/shm                tmpfs   defaults        0 0
devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
sysfs                   /sys                    sysfs   defaults        0 0
proc                    /proc                   proc    defaults        0 0
none                    /tmp                    tmpfs   defaults        0 0
UUID=14b77b7a-26e3-4dba-953c-b70a7e1d7f2f /var/lib/mock           ext4    defaults        1 2
UUID=6a49ba85-7b5a-495f-ba12-9a82329382dc /home                   ext4    defaults        1 2
UUID=9597b36e-a364-416d-b4d6-c5e36e49474d /home/leigh/qBT_dir     ext4    defaults        1 2
UUID=4ce66800-671d-48e4-9ec6-c63fd1a6b468 /home/leigh/data        ext4    defaults        1 2
UUID=a9b68060-02fe-4235-8883-9be1fc091372 /home/leigh/data1       ext4    defaults        1 2
UUID=cdcf850b-7178-499e-a9ab-2ca8a637236c /home/leigh/development ext4    defaults        1 2
[root@main_pc leigh]#

[synic]

Originally Posted by leigh123linux

Here's my fstab, the highlighted line is the ssd partition.

[root@main_pc leigh]# cat /etc/fstab

#
# /etc/fstab
# Created by anaconda on Sun Feb 27 18:03:34 2011
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
UUID=91faae69-0fa6-42d8-8d94-1d0070678d8a /                       ext4    defaults,discard        1 1
tmpfs                   /dev/shm                tmpfs   defaults        0 0
devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
sysfs                   /sys                    sysfs   defaults        0 0
proc                    /proc                   proc    defaults        0 0
none                    /tmp                    tmpfs   defaults        0 0
UUID=14b77b7a-26e3-4dba-953c-b70a7e1d7f2f /var/lib/mock           ext4    defaults        1 2
UUID=6a49ba85-7b5a-495f-ba12-9a82329382dc /home                   ext4    defaults        1 2
UUID=9597b36e-a364-416d-b4d6-c5e36e49474d /home/leigh/qBT_dir     ext4    defaults        1 2
UUID=4ce66800-671d-48e4-9ec6-c63fd1a6b468 /home/leigh/data        ext4    defaults        1 2
UUID=a9b68060-02fe-4235-8883-9be1fc091372 /home/leigh/data1       ext4    defaults        1 2
UUID=cdcf850b-7178-499e-a9ab-2ca8a637236c /home/leigh/development ext4    defaults        1 2
[root@main_pc leigh]#

You dont have the noatime or data=writeback options implimented? Sorry, but is this covered / incorporated in the discard option or have you just decided not to impliment it?

[leigh123linux]

Originally Posted by synic

You dont have the noatime or data=writeback options implimented? Sorry, but is this covered / incorporated in the discard option or have you just decided not to impliment it?

data=writeback caused issues here.

[synic]

Originally Posted by leigh123linux

data=writeback caused issues here.

I think it gave me issues too. Booted with it in fstab and screen froze for a minute and screen had rainbows before loading up properly. Took it out and had a pretty straight forward boot.

[zenous]

guys i manage to get an SSD drive today
after booting f15 using thumbdrive i want to install it to my SSD
the installation stuck at finding disk
do i have to cr8 a new partion b4 installing f15?