Marc's Public Blog - Linux Hacking

Please also read this very good report from Luke on SSD unreliabilities.

Crucial C300 sudden death and full data loss

So, before people tell me "you should have bought intel", I got the C300 from work and installed it because I trusted it had been pre-qualified and was good. Turns out, I was one of the people doing the pre-qualifying :) Later, I did look at intel for my drives, but they were lacking in size capacity when I was looking for 512GB and later for 1TB. Too bad because they've been shown to be the most or amongst the most reliable drives so far.

I wasn't using TRIM with the C300 at the time, because it's not necessarily recommended if you use dm-crypt ( http://asalor.blogspot.com/2011/08/trim-dm-crypt-problems.html ). The C300 likely did not appreciate that I filled it a few times and that it had to use its garbage collection algorithm to reclaim blocks (when it was getting full, it was very very slow, as in 100x slower than a spinning drive at times).

In the end, it died after fewer than 3 months of use, where it just disappeared from the SATA bus. Some magic 'plug the power but don't do data' for 20mn or so, several times, brought it back (after several days of trying) long enough for me to see it could still talk SATA and that all my partitions and data were gone. Even then, I wasn't able to really usefully talk to it anyway.

If you read up on this, and hear Crucial support recommending that you let the drive sit at the bios for 20mn once it a while so that its garbage collection can kick in after 15mn of it not being used, you'll understand that you should run away in the other direction rather than buying one of those drives: http://forums.crucial.com/t5/Solid-State-Drives-SSD/Bios-is-not-recognize-crucail-m4-SSD-64GB/td-p/63835

Maybe the C400 sucks less, but once burnt, twice shy, and the C300 was so bad, that I just don't trust Crucial to write proper firmware at this point.

Samsung 830 512GB severe random IO slowness with linux and windows

So, I then did some reading about SSDs and shopping around. I found that the Intel did not come in 512GB (it was a bit smaller), and that while Intel ruled SSDs early on, it hasn't kept up in performance and offerings. Also, they use older sanforce controllers that do not do well with software encrypted data (exactly what I had).

From online reading, the next best player was Samsung, so I got an 830 512GB drive from them, and was quite hopeful after reading reviews and how they designed and manufacture all the parts themselves.
That decision came back to bite me. I spent too many hours (between 10 and 20) with samsung tech support, doing many benchmarks, and working with the linux community, because its random IO performance was bad. How bad? Well, how does 5 times slower than my hard drive in the same laptop sound?

On time spent with the linux community debugging this with multiple filesystems, laptops, and tweaks, I'm not kidding. See a small sample:

There is one problem that might be linux specific: the drive is very slow for encrypted block device access without:
blockdev --settra 8192 /dev/mapper/cryptroot

But past that, I found no other problems, and replicated the problem with ntfs, and even after putting the SSD in a windows desktop with Windows 7 64bit. Samsung's old Benchmark registered so low on their own Random IO test that the bar graph showing speed was a single line at the '0' mark.
Samsung support had no idea why I saw this. They suggested I get another drive in case mine was bad.
The new drive had identical problems and in the end I had to return both. Yes, the drives had the latest firmware. The test below shows lower block IO because it was on SATA2, not SATA3 like in my laptop but I was only worried about Random IO.

I have no idea why those drives were so unexplainably slow on random IO reads (not even writes) when block IO was fast (500MB/s). I don't know how they work for others, but they failed to perform for me on 3 machines, 4 filesystems, and 3 operating systems.

That was a lot of time wasted, and I'm not impressed with Samsung anymore :(

Samsung 840 Evo 1TB (bought in November 2013)

Non surprisingly, it's the fasted drive of them all, but it's over a year newer. With an upgrade to linux 3.11 in the meantime, encrypted btrfs on that drive is almost as fast as native raw speeds.

OCZ 512GB Vertex 4 vs Samsung 830 512GB vs Samsung 840 EVO 1TB linux tests with btrfs, ext4, and dmcrypt

Update: the drive is fast, it never really lost data (except apparently the last few blocks written before locking up), but I had 3 drives eventually go into a mode where they lock up and my laptop needs to be power cycled under certain workloads (mencoder is one for me). They always recovered, only lost the last bits of data being written when the drive locked up, but that's just no good. I replaced the drive 3 times and eventually switched to a 1TB Samsung 840 Evo.

Old text from July 2012:
After more research still, this drive came highly recommended too. It's the fastest from OCZ and the most expensive (they have slower cheaper ones). And yet, the most expensive one was $100 cheaper than the Samsung, and 36 times faster for random IO reads. Oh, also the OCZ has 5 year warranty instead of 2 for Samsung.

There are only crude tests, but they are the ones that mattered to me to validate that the OCZ didn't have severe problems in my hardware like the Samsung 830 did.
Here are optimizations I used:

The OCZ was upgraded to Firmware Version 1.5, and the Samsung 830 claimed to have the latest firmware as of 2012/08 (that was version CXM03B1Q as reported by smartmontools). The Hitachi Hard drive in my laptop is a slow 5400RPM model.

Long story short, the OCZ is super fast, worked great in my tests, and was cheaper than the Samsung and the Crucial. Win all around.

Scanning 15K inodes with the Vertex 4 (vs Samsung and spinning hard drive)

Summary of results:

                                        OCZ Vertex 4 512GB | Samsung    | Samsung    | Hitachi HD 
                                                           | 830 512GB  | 840 1TB    | 1TB 5400 RPM
---------------------------------------------------------------------------------------------------
Encrypted BTRFS:                                           |            |            |
reset_cache; time du -sh src                               |            |            |
514M        src                                            |            |            |
real                                                0.959s | 25.741s    | 0.53s      | 5.252s
                                                           |            |            |
Unencrypted btrfs                                          |            |            |
mount -o compress=lzo,discard,nossd,space_cache,noatime    |            |            |
reset_cache; time du -sh src                    kernel 3.5 | kernel 3.5 | kernel 3.11| kernel 3.5
514M        src                                            |            |            | 
real                                                0.636s | 24.937s    | 0.11s      | 1.007s
                                                           |            |            |
Unencrypted ext4                                           |            |            |
mkfs.ext4 -O extent -b 4096 -E stride=128,stripe-width=128)|            |            |
mount -o discard,noatime /dev/sda2 /mnt/mnt2               |            |            |
reset_cache; time du -sh src                               |            |            |
519M        src                                            |            |            |
real                                                0.289s | 12.459s    | 0.21s      | 0.380s

Block Reads on Vertex 4 and Samsung 840 EVO (vs Samsung 830 and spinning hard drive)

Summary:

                                   OCZ Vertex 4 512GB | Samsung      | Samsung     | Hitachi HD 
                                                      | 830 512GB    | 840 1TB     | 1TB 5400 RPM
----------------------------------------------------------------------------------------------------------------
dd if=/dev/sda3 of=/dev/null bs=1M count=2000         |              |             |
2097152000 bytes (2.1 GB) copied, 4.65606 s, 450 MB/s | 479MB/s      | 541MB/s     | 104 MB/s
                                                      |              |             |
hdparm -t /dev/sda3                                   |              |             |
/dev/sda3:                                            |              |             |
 1430 MB in  3.00 seconds =               476.12 MB/s | 463.45 MB/s  | 1112.81MB/s | 99.83 MB/s
                                                      |              |             |
Unencrypted ext4 block read:                          |              |             |
reset_cache; dd if=testcopy of=/dev/null bs=1M        |              |             |
2146631680 bytes (2.1 GB) copied, 4.21942 s, 509 MB/s |              | 458MB/s     | 105 MB/s
                                                      |              |             |
Unencrypted btrfs block read:                         |              |             |
reset_cache; dd if=testcopy of=/dev/null bs=1M        |              |             |
2146631680 bytes (2.1 GB) copied, 4.31774 s, 497 MB/s | 495 MB/s     | 542MB/s     | 90.3 MB/s
                                                      |              |             |
Encrypted btrfs block read:                           |              |             |
dd if=w2k-s001.vmdk of=/dev/null bs=1M     kernel 3.5 | kernel 3.5   | kernel 3.11 | kernel 3.5
2146631680 bytes (2.1 GB) copied, 6.51863 s, 329 MB/s | 253 MB/sec   | 533 MB/sec  | 69.1 MB/s

Block writes on Vertex 4 and (vs spinning hard drive)

I didn't benchmark write speed on the Samsung 830 because I couldn't get random reads to be fast, so I didn't care about writes.

Summary:

OCZ Vertex 4 512GB:                           | Samsung EVO  | Hitachi 1TB 
                                              | 840 1TBM     | 5400 RPM
---------------------------------------------------------------------------
Raw device write (write to /dev/sdxx bs=1M)   |  kernel 3.11 |
(2.1 GB) copied, 9.35569 s, 229 MB/s          |  430 MB/s    |  90.9 MB/s
                                              |              |
File write on unencrypted ext4:               |  kernel 3.11 |
(2.1 GB) copied, 11.3984 s, 188 MB/s          |  495 MB/s    |  92.0 MB/s
                                              |              |
File write on unencrypted btrfs:              |  kernel 3.11 |
(2.1 GB) copied, 8.58974 s, 250 MB/s          |  413 MB/s    |  130 MB/s
                                              |              |
File write through encryption layer on btrfs  |  kernel 3.11 |
(2.1 GB) copied, 16.4673 s, 130 MB/s          |  341 MB/s    |  95.6 MB/s

SSD winner and conclusions

The OCZ Vertex 4 512GB won by far obviously. The price is competitive, the speed beats pretty much everything out there, and the 5 year warranty shows that they mean business (most only offer 1, 2, or sometimes 3 years). 5 years is a record in this field.
Well, that was then, turns out OCZ was more confident than good, their drives did lock up, they made bad tradeoffs on other drives as explained by Luke on http://lkcl.net/reports/ssd_analysis.html, and they are now going bankrupt as of Dec 2013.
After about a year of use, I confirmed that the OCZ drive is just unstable with linux/dmcrypt/btrfs on my laptop. I'm not sure what part causes it to hang and require a power cycle every other day or so after a few months of use, but that was quite annoying.

I'm now hopeful that the Samsung EVO 840 will be my new "good drive" for a long longer :)

It sucks that it took me so long to find a good drive, and so many backup/restore cycles, but I think I got it now. Wish me luck! :)

The conf had plenty of talks to see, and great parties. Just a few pictures below to give a taste, and I also have a lot of pictures of my visiting of San Diego.

A few pictures from the keynotes:

kernel panel

A few slides from talks:

And we had some nice parties. The first night was a rooftop party:

The second evening, we had a party on an almost-island, including a small cruise to get there:

The 3rd evening, we had a beach Hawaiian party on Coronado Island, which was quite good:

the long bridge that gets you there

A few random picts from the conf:

It was a good conf and a great time.