Fedora 16 and Discard (TRIM) support on LUKS/dm-crypt devices

[NoHumanBeing]

Does anyone know whether the cryptography-target for the device-mapper (dm-crypt) used in Fedora 16 will support the "discard" option to pass through the TRIM operation to an underlying device (e. g. a solid state drive)?

We plan to use Fedora 16 on a system with a solid state drive as storage. We need to use full disk encryption (encrypt everything except "/boot") on that system because it stores sensitive data and we want to be able to RMA the disk in case of a disk failure, but need to be sure that the data is not disclosed to the drive manufacturer.

Solid state memory has limited write cycles. You never actually know when a drive failure is soon until it happens and you probably cannot perform a secure-erase on a completely worn-out drive so the only option we have is to use full disk encryption, reliably destroy the key in case of a disk failure and return the drive.

The problem is that solid state drives wear out more quickly and drop in write performance when used without TRIM for prolonged periods of time. So the question is: Will TRIM propagate through the encryption layer? As far as I understand each disk sector is encrypted independently, so theoretically TRIM support should not be that difficult to implement. Just pass the TRIM operation down to the underlying device with the sector number adjusted by a certain offset if neccessary. The drive will erase the sector in question. Afterwards it will contain bytes with its bits all reset (0x00), which will of course decrypt to some garbarge, but that won't matter since the sector in question isn't considered to hold any useful information anyways. I don't see why TRIM support on encrypted devices is all that problematic, but for some reason it didn't get implemented for quite some time.

Anyone knows reliably, whether "discard" will finally work on the dm-crypt target used in Fedora 16? This is quite a critical feature for us.

[Moby1de]

the new ssd are supporting "garbage collection". It works fine for me. You dont need Trim support

[NoHumanBeing]

The problem with garbage collection is that the SSD won't know which (logical) sectors contain information which needs to be preserved and which sectors are "empty", since over time, every sector that has been written to will contain data and the SSD can't know which of this data needs to be preserved (e. g. because it is part of a file) and which data can be discarded because it has no knowledge of the file system. As a block device, the disk will just store "a bunch of bytes", but it doesn't know what these bytes mean, so it will just have to preserve every single bit of information which has been written to it, just like a magnetical hard disk does.

But contrary to a magnetical hard disk, "preserving" data means extra work for the NAND memory, since the data needs to be "refreshed" (data is stored as electrical charge on the floating gate of field effect transistors and over time, this charge will "bleed out" from the gate into the substrate, since no material is a "perfect insulator", so the cells will need to be "recharged" every now and then in order to preserve the state) and "shuffled around" (moved to a new physical location by changing the mapping between logical and physical address space as part of the wear levelling purpose) every now and then. Without TRIM, the memory will "refresh and shuffle around" data, which the host never needs. After all, how should the disk know that the data is no longer needed by the host OS? The disk doesn't have any knowledge about the file system.

TRIM will tell the disk that there is a sector which has some state, which does not need to be preserved, because it contains no useful data at all. As far as I understand the NAND will then no longer take any care to preserve the state of that sector, e. g. not refresh the contents of the cells associated with this sector and not move the data around for wear levelling purpuses. This is additional information from the host that the NAND will never have without TRIM.

I don't know if it is THAT bad, but I understand that this "unneccessary preservation of data" causes the NAND to perform "superfluous write procedures", which take time (and will thus slow down write performance) and wear out the NAND, causing it to fail earlier. The latter is probably not so much of a problem, as the NAND will probably sustain decades of non-stop writing before it will actually fail. You probably won't be using a disk long enough for the NAND to wear out, unless you get a "bad" one (the write count the memory sustains seem to be a statistical value and there have been a few disks out there that failed quite early, which is why I'm doing all this encryption stuff, so that I can RMA a disk which can no longer be securely erased because the NAND is not erasable any more and not have the data on the system disclosed). But I'd like to use TRIM in order to stop the NAND from refreshing cells which contain no useful data, so that it has both "more space" (empty cells can directly be written to, cells containing data first need to be erased before they can be written to, the erase cycle takes additional time, "more empty cells" thus means "faster writes") as well as "more time" (because the NAND doesn't have to waste time "refreshing" cells which contain no useful data if it could use this time for writing new data instead) and will thus write faster.

[Moby1de]

PHP Code:

After all, how should the disk know that the data is no longer needed by the host OS? 


The OS know the free space and try to overwrite it/the old data when files are saved; SSD will use free space instead; when the ssd is not in use it delete the old data...
Read the discription of garbage collection.....
You do not need trim for full speed.

[NoHumanBeing]

Originally Posted by Hangcock

You do not need trim for full speed.

Are you sure? SSDs have 512 KB blocks (usually) but the host uses sectors of 4 KB, so there are 128 sectors in a block. The block is the smallest erasable unit of memory on the SSD. If the contents of a single sector within a block is changed, the disk will copy the entire block into a "scrapspace" buffer, modify the sector, and subsequently write out the whole block to a new location (on the logical layer, 4 KB are written, on the physical layer 512 KB are written, this is called "write amplification"). If the sector was in an empty state before the write operation took place (because it was TRIMmed before), the SSD could just write into it, leaving all the other sectors within the block intact. Guess what's faster?

This is also what I measure. The performance of my SSD (OCZ Vertex 2E) is highest after a secure erase (after which the entire memory is empty and all the space is unmapped) and drops (not dramatically, but it drops a bit, you can benchmark it) as you write data on it, as more and more sectors are "used" (have had data written to it). From what I understand, TRIM would clear these sectors and leave them in an "unused" ("ready to be written") state.

[AdamW]

AIUI, it does, though I'm by no means an expert on the issue. At the least, my desktop uses a standard Fedora-style LVM setup, and I can use wiper.sh on it with apparent success (whereas it chokes on the BIOS RAID on my laptop).

[AdamW]

BTW, RH has people who are very intimately acquainted with this topic, for obvious reasons, and their current standing advice is not to use trim at the mount level unless you're sure it actually improves speed on the specific SSD you're using; they've tested with a wide range of SSDs and found it doesn't actually improve overall performance on a lot of them. The 'safe' approach is to run wiper.sh periodically when the system is not in active use, via cron. There's a few interesting threads in the devel archives on the topic, which will probably show up if you search for trim.

[NoHumanBeing]

Originally Posted by AdamW

AIUI, it does, though I'm by no means an expert on the issue. At the least, my desktop uses a standard Fedora-style LVM setup, and I can use wiper.sh on it with apparent success (whereas it chokes on the BIOS RAID on my laptop).

The problem is the dm-crypt layer. As far as I know, dm-crypt will just "swallow" any "Discard" command it receives from the file system and not pass it through to the underlying drive. As far as I know even the device mapper itself will pass it through, it's just the cryptography target which doesn't have any pass-through option.

This is actually quite a shame since, as far as I understand, it would be quite possible to implement such an option, it's just that some people are overly paranoid and say "ooh, if we enable pass-through for the Discard command, forensics could find out which sectors are holding actual data and which are not", whereas I say "who cares which sectors are holding data if you cannot look at the contents of a sector since it is encrypted?".

If this is right, then my suggestion would be to implement it as an option and have it disabled by default, but please please please, add support for this for the people who know what they are doing and are willing to "take the risk". There is a reason why most of the file systems are having a "Discard" option implemented. I'm not sure if it will result in a performance gain, it just seems to be "the proper way of operating a solid state drive" and disclosure of the identifiers of empty/unused sectors is not regarded a serious privacy concern by me. I actually regard it as more secure to use TRIM. Let's just say an unauthorized programm gains root access to the system. It could just access the device-mapper target and read the plain text data written to the drive. Disk encryption is worth nothing when the encrypted file system is already mounted, it just secures the data while the system is shut down or the file system is unmounted for some other reason. So our hypothetical software reading our drive will probably find lots of remnants of already-deleted files there. If the system had TRIM enabled, these remnants would have been gone and the program could read what's stored on the system NOW, while it can read what was stored on the system "in the past" and has already been deleted (but not overwritten yet) when the system does NOT have TRIM enabled. I think the option of enabling TRIM is actually more secure.

[AdamW]

ah, my disk isn't encrypted, so I don't know about those details.

[NoHumanBeing]

Yes, the point is that we need to use full disk encryption on the drive because potentially sensitive data might be stored on it and EEPROM memory that is worn out can no longer be erased. The 10.000 writes per sector are just a statistical value. It might actually fail at any time and without any warning.

Memory that is worn-out will either become read-only or completely inaccessible. In the latter case, all the clock and data lines on the memory module go high-Z and the IC will no longer respond to anything. The point is that the memory will continue to hold the data physically even after a failure, but can no longer be written to or erased. So the only options are:

- Physically destroy the memory (e. g. by incinerating the ICs, but this is not an option if the drive is still under warrenty and you want to RMA it for replacement)
- Do not store sensitive data on the drive (this is actually what we do, the encrypted data is quite "insensitive" as long as the key is not disclosed in any way)

One might say that the data is secure since a worn-out EEPROM module cannot be read. The problem is that the modules "sometimes" do not fail completely and become read-only or that there might be some adventurous way of "unlocking" modules after they have failed completely. The latter probably involves opening the packaging of the memory IC and shooting some new bondings in. I've got this information directly from a memory manufacturer. They didn't want to get any more specific, but they told me that an "undocumented method of unlocking worn-out EEPROM modules exists", that it "involves opening the packaging" and they "strongly advise against storing sensitive data in clear text on non-volatile solid state memory". I have taken this advice quite seriously since they are effectively mentioning a "weak point" in their own product. They certainly didn't make this statement for marketing purposes. ;-)