Marc's Public Blog - Linux Hacking


All | Aquariums | Arduino | Btrfs | Cars | Cats | Clubbing | Computers | Diving | Dreamstate | Edc | Electronics | Exercising | Festivals | Flying | Halloween | Hbot | Hiking | Linux | Linuxha | Monuments | Museums | Oshkosh | Outings | Public | Rc | Sciencemuseums | Solar | Tfsf | Trips



This page has a few of my blog entries about linux, but my main linux page is here
Picture of Linus

Here is a list of older linux event reports I made before my blog was started, then the rest are below
1996/11/18-21:Linux Pavillion Comdex Fall 1996 (photos only). I've been going since then to help at the linux pavillion.
1997/11/18-21: Linux Pavillion Comdex Fall 1997 (photos only)
1998/05/28-30: Linuxexpo 1998 (photos only)
1998/11/16-20: Linux Pavillion Comdex Fall 1998 (full report)
1998/11/11: Silicon Valley Tea Party (report with pictures)
1999/02/15: Windows Refund Day (report with pictures)
1999/03/20: SVLUG KTEH night (photos only)
1999/03/01-04: LinuxWorld Expo Winter 99 (complete report with many pictures)
1999/03/31: Mozilla Party one year anniversary (photos only)
1999/05/18-22: Linuxexpo 1999 (complete report with many pictures)
1999/06/07: June 99 Balug meeting with Linus
1999/08/09-12: LinuxWorld Expo Summer 99 (complete report with many pictures)
1999/11/15-19: Linux Business Show at Comdex Fall 1999 (full report with pictures)
2000/08/14-17: LinuxWorld Expo Summer 2000 (complete report with many pictures)
2001/01/17-20: Linux.conf.au/LCA 2001 (complete report with pictures)
2001/07/25-28: OLS 2001 (photos only)
2001/08/25: Linux 10th Anniversary (report with pictures)
2001/09/27-30: LinuxWorld Expo Summer 2001 report with pictures)
2001/11/05-10: ALS 2001 (photos only)
2002/06/26-29: OLS 2002 (photos only)
2003/01/20-25: LCA 2003 (photos only)
2003/07/23-26: OLS 2003 (photos only)
2004/01/12-17: LCA 2004 (photos only)
2004/07/21-24: OLS 2004 (photos only)
2005/04/18-23: LCA 2005 (photos only)
2006/01/24-28: LCA 2006 (photos only)
2007/01/17-21: LCA 2007 (photos only)

Here is a list of all the talks I've given:

And below are my blog posts:

Table of Content for linux:

More pages: July 2002 February 2004 March 2004 November 2004 April 2005 August 2005 January 2006 July 2006 August 2007 November 2007 January 2008 October 2008 November 2008 December 2008 January 2009 May 2009 July 2009 August 2009 September 2009 November 2009 December 2009 January 2010 March 2010 April 2010 June 2010 August 2010 October 2010 January 2011 July 2011 August 2011 December 2011 January 2012 March 2012 May 2012 August 2012 December 2012 January 2013 March 2013 May 2013 September 2013 November 2013 January 2014 March 2014 April 2014 May 2014 October 2014 January 2015 March 2015 May 2015 January 2016 February 2016 June 2016 July 2016 August 2016 October 2016 January 2017 September 2017 January 2018 March 2018 December 2018 January 2019 January 2020 May 2020 January 2021 September 2021 March 2023 April 2023 December 2023 June 2024



2012/08/15 The tale of SSDs: Crucial C300 early Death, Samsung 830 extreme random IO slowness, and settling with OCZ Vertex 4
π 2012-08-15 01:01 in Linux
For indexing purposes, I didn't change the title, but I don't recommend the OCZ Vertex 4 anymore, it does lock up under certain workloads. My new drive is the Samsung 840 Evo which performs fine where the 830 absolutely did not for me.

Please also read this very good report from Luke on SSD unreliabilities.

Crucial C300 sudden death and full data loss

So, before people tell me "you should have bought intel", I got the C300 from work and installed it because I trusted it had been pre-qualified and was good. Turns out, I was one of the people doing the pre-qualifying :) Later, I did look at intel for my drives, but they were lacking in size capacity when I was looking for 512GB and later for 1TB. Too bad because they've been shown to be the most or amongst the most reliable drives so far.

I wasn't using TRIM with the C300 at the time, because it's not necessarily recommended if you use dm-crypt ( http://asalor.blogspot.com/2011/08/trim-dm-crypt-problems.html ). The C300 likely did not appreciate that I filled it a few times and that it had to use its garbage collection algorithm to reclaim blocks (when it was getting full, it was very very slow, as in 100x slower than a spinning drive at times).

In the end, it died after fewer than 3 months of use, where it just disappeared from the SATA bus. Some magic 'plug the power but don't do data' for 20mn or so, several times, brought it back (after several days of trying) long enough for me to see it could still talk SATA and that all my partitions and data were gone. Even then, I wasn't able to really usefully talk to it anyway.

If you read up on this, and hear Crucial support recommending that you let the drive sit at the bios for 20mn once it a while so that its garbage collection can kick in after 15mn of it not being used, you'll understand that you should run away in the other direction rather than buying one of those drives: http://forums.crucial.com/t5/Solid-State-Drives-SSD/Bios-is-not-recognize-crucail-m4-SSD-64GB/td-p/63835

Maybe the C400 sucks less, but once burnt, twice shy, and the C300 was so bad, that I just don't trust Crucial to write proper firmware at this point.

Samsung 830 512GB severe random IO slowness with linux and windows

So, I then did some reading about SSDs and shopping around. I found that the Intel did not come in 512GB (it was a bit smaller), and that while Intel ruled SSDs early on, it hasn't kept up in performance and offerings. Also, they use older sanforce controllers that do not do well with software encrypted data (exactly what I had).

From online reading, the next best player was Samsung, so I got an 830 512GB drive from them, and was quite hopeful after reading reviews and how they designed and manufacture all the parts themselves.
That decision came back to bite me. I spent too many hours (between 10 and 20) with samsung tech support, doing many benchmarks, and working with the linux community, because its random IO performance was bad. How bad? Well, how does 5 times slower than my hard drive in the same laptop sound?

On time spent with the linux community debugging this with multiple filesystems, laptops, and tweaks, I'm not kidding. See a small sample:

  • http://www.spinics.net/lists/dm-crypt/msg04403.html
  • http://www.spinics.net/lists/dm-crypt/msg04406.html
  • http://www.spinics.net/lists/dm-crypt/msg04408.html
  • http://www.spinics.net/lists/dm-crypt/msg04436.html
  • http://www.spinics.net/lists/linux-btrfs/msg18009.html
  • http://www.spinics.net/lists/linux-btrfs/msg18168.html
  • http://www.spinics.net/lists/linux-btrfs/msg18204.html
  • http://www.spinics.net/lists/linux-btrfs/msg18238.html
  • http://www.spinics.net/lists/linux-btrfs/msg18240.html
  • http://permalink.gmane.org/gmane.comp.file-systems.ext4/33491
  • http://permalink.gmane.org/gmane.comp.file-systems.ext4/33637
  • There is one problem that might be linux specific: the drive is very slow for encrypted block device access without:
    blockdev --settra 8192 /dev/mapper/cryptroot

  • http://www.spinics.net/lists/dm-crypt/msg04408.html
  • http://www.spinics.net/lists/dm-crypt/msg04436.html
  • But past that, I found no other problems, and replicated the problem with ntfs, and even after putting the SSD in a windows desktop with Windows 7 64bit. Samsung's old Benchmark registered so low on their own Random IO test that the bar graph showing speed was a single line at the '0' mark.
    Samsung support had no idea why I saw this. They suggested I get another drive in case mine was bad.
    The new drive had identical problems and in the end I had to return both. Yes, the drives had the latest firmware. The test below shows lower block IO because it was on SATA2, not SATA3 like in my laptop but I was only worried about Random IO.

    I have no idea why those drives were so unexplainably slow on random IO reads (not even writes) when block IO was fast (500MB/s). I don't know how they work for others, but they failed to perform for me on 3 machines, 4 filesystems, and 3 operating systems.

    That was a lot of time wasted, and I'm not impressed with Samsung anymore :(

    Samsung 840 Evo 1TB (bought in November 2013)

    As explained below, the OCZ drive was fast and didn't have the random IO performance issue that the Samsung 830s had for me, but the Vertex ended up locking up every few days after 3-6 months of use, needing replacement. When I got tired of that, I eventually upgraded to a 1TB Samsung 840 Evo.

    Non surprisingly, it's the fasted drive of them all, but it's over a year newer. With an upgrade to linux 3.11 in the meantime, encrypted btrfs on that drive is almost as fast as native raw speeds.

    OCZ 512GB Vertex 4 vs Samsung 830 512GB vs Samsung 840 EVO 1TB linux tests with btrfs, ext4, and dmcrypt

    Update: the drive is fast, it never really lost data (except apparently the last few blocks written before locking up), but I had 3 drives eventually go into a mode where they lock up and my laptop needs to be power cycled under certain workloads (mencoder is one for me). They always recovered, only lost the last bits of data being written when the drive locked up, but that's just no good. I replaced the drive 3 times and eventually switched to a 1TB Samsung 840 Evo.

    Old text from July 2012:
    After more research still, this drive came highly recommended too. It's the fastest from OCZ and the most expensive (they have slower cheaper ones). And yet, the most expensive one was $100 cheaper than the Samsung, and 36 times faster for random IO reads. Oh, also the OCZ has 5 year warranty instead of 2 for Samsung.

    There are only crude tests, but they are the ones that mattered to me to validate that the OCZ didn't have severe problems in my hardware like the Samsung 830 did.
    Here are optimizations I used:

  • fdisk cylinders were chosen so that they were multiple of 512 to make sure I was aligned with internal block write sizes
  • btrfs was created with mkfs.btrfs and mounted with mount -o compress=lzo,discard,nossd,space_cache,noatime
  • ext4 was created with mkfs.ext4 -O extent -b 4096 -E stride=128,stripe-width=128 and mounted with mount -o discard,noatime
  • cryptsetup luksFormat --align-payload=8192 -c aes-xts-plain /dev/sda2
  • cryptsetup luksOpen --allow-discards /dev/sda4 cryptroot
  • The OCZ was upgraded to Firmware Version 1.5, and the Samsung 830 claimed to have the latest firmware as of 2012/08 (that was version CXM03B1Q as reported by smartmontools). The Hitachi Hard drive in my laptop is a slow 5400RPM model.

    Long story short, the OCZ is super fast, worked great in my tests, and was cheaper than the Samsung and the Crucial. Win all around.

    Scanning 15K inodes with the Vertex 4 (vs Samsung and spinning hard drive)

    Summary of results:

  • Samsung 830 is so slow, it doesn't make sense
  • OCZ Vertex 4 is super fast
  • Samsung 840 Evo 1TB more than a year later is much much faster and easily beats the Vertex 4
  • for reasons that I ignore, encrypted btrfs on my HD was 5x slower than unencrypted in the du -s case.
  • dmcrypt on linux makes btrfs random IO 50% slower
  • random seeks/IO on ext4 in this du test is 2x faster than btrfs on SSD, and 3x faster on HD.
  • At the same time, I'm pretty sure btrfs got faster between kernels 3.5 and 3.11 (this is supported by btrfs now being faster than ext4 when it was slower before
  • OCZ Vertex 4 512GB | Samsung | Samsung | Hitachi HD | 830 512GB | 840 1TB | 1TB 5400 RPM --------------------------------------------------------------------------------------------------- Encrypted BTRFS: | | | reset_cache; time du -sh src | | | 514M src | | | real 0.959s | 25.741s | 0.53s | 5.252s | | | Unencrypted btrfs | | | mount -o compress=lzo,discard,nossd,space_cache,noatime | | | reset_cache; time du -sh src kernel 3.5 | kernel 3.5 | kernel 3.11| kernel 3.5 514M src | | | real 0.636s | 24.937s | 0.11s | 1.007s | | | Unencrypted ext4 | | | mkfs.ext4 -O extent -b 4096 -E stride=128,stripe-width=128)| | | mount -o discard,noatime /dev/sda2 /mnt/mnt2 | | | reset_cache; time du -sh src | | | 519M src | | | real 0.289s | 12.459s | 0.21s | 0.380s



    Block Reads on Vertex 4 and Samsung 840 EVO (vs Samsung 830 and spinning hard drive)

    Summary:

  • raw device speed is about the same for Samsung 830 and OCZ. The 840 is marginally faster
  • encrypted btrfs at 329MB/s is slower, but still is plenty fast. If anyone is doing software encryption faster than that, let me know :)
  • well, a bit over a year later, on the same laptop with the same CPU, encrypted btrfs block speed is about as bad as native block speed. Impressive.
  • unencrypted btrfs block speed is about as fast as block device speed in kernel 3.11. Very nice.
  • OCZ Vertex 4 512GB | Samsung | Samsung | Hitachi HD | 830 512GB | 840 1TB | 1TB 5400 RPM ---------------------------------------------------------------------------------------------------------------- dd if=/dev/sda3 of=/dev/null bs=1M count=2000 | | | 2097152000 bytes (2.1 GB) copied, 4.65606 s, 450 MB/s | 479MB/s | 541MB/s | 104 MB/s | | | hdparm -t /dev/sda3 | | | /dev/sda3: | | | 1430 MB in 3.00 seconds = 476.12 MB/s | 463.45 MB/s | 1112.81MB/s | 99.83 MB/s | | | Unencrypted ext4 block read: | | | reset_cache; dd if=testcopy of=/dev/null bs=1M | | | 2146631680 bytes (2.1 GB) copied, 4.21942 s, 509 MB/s | | 458MB/s | 105 MB/s | | | Unencrypted btrfs block read: | | | reset_cache; dd if=testcopy of=/dev/null bs=1M | | | 2146631680 bytes (2.1 GB) copied, 4.31774 s, 497 MB/s | 495 MB/s | 542MB/s | 90.3 MB/s | | | Encrypted btrfs block read: | | | dd if=w2k-s001.vmdk of=/dev/null bs=1M kernel 3.5 | kernel 3.5 | kernel 3.11 | kernel 3.5 2146631680 bytes (2.1 GB) copied, 6.51863 s, 329 MB/s | 253 MB/sec | 533 MB/sec | 69.1 MB/s



    Block writes on Vertex 4 and (vs spinning hard drive)

    I didn't benchmark write speed on the Samsung 830 because I couldn't get random reads to be fast, so I didn't care about writes.

    Summary:

  • my HD is slow enough (even though 95MB/s isn't actually slow) that encryption doesn't make it slower for block writes
  • on SSD however, it looks like I may hit an encryption bottleneck, speed goes down by almost 2x. Still 130MB/s is fast enough for writes, and I really mostly care about ultimate read speed, which I do have.
  • Somehow with kernel 3.11 on the Samsung 840 EVO, encrypted writes are now 80% of unencrypted speed. Very nice.
  • OCZ Vertex 4 512GB: | Samsung EVO | Hitachi 1TB | 840 1TBM | 5400 RPM --------------------------------------------------------------------------- Raw device write (write to /dev/sdxx bs=1M) | kernel 3.11 | (2.1 GB) copied, 9.35569 s, 229 MB/s | 430 MB/s | 90.9 MB/s | | File write on unencrypted ext4: | kernel 3.11 | (2.1 GB) copied, 11.3984 s, 188 MB/s | 495 MB/s | 92.0 MB/s | | File write on unencrypted btrfs: | kernel 3.11 | (2.1 GB) copied, 8.58974 s, 250 MB/s | 413 MB/s | 130 MB/s | | File write through encryption layer on btrfs | kernel 3.11 | (2.1 GB) copied, 16.4673 s, 130 MB/s | 341 MB/s | 95.6 MB/s

    SSD winner and conclusions

    SSDs are not hard drives. They are faster for sure when they work well, but don't expect them to fail like hard drives. They can fail suddenly with absolutely no warning, and chancees of recovery after that are slim to none. Recovery companies know how to deal with hard drive platters, but most aren't setup to deal with SSDs yet. Moral of the story: have backups (I did, and now I actually backup my SSD on the 2nd HDD in my laptop and can boot on the HDD if the SSD craps out while I'm on a trip).

    The OCZ Vertex 4 512GB won by far obviously. The price is competitive, the speed beats pretty much everything out there, and the 5 year warranty shows that they mean business (most only offer 1, 2, or sometimes 3 years). 5 years is a record in this field.
    Well, that was then, turns out OCZ was more confident than good, their drives did lock up, they made bad tradeoffs on other drives as explained by Luke on http://lkcl.net/reports/ssd_analysis.html, and they are now going bankrupt as of Dec 2013.
    After about a year of use, I confirmed that the OCZ drive is just unstable with linux/dmcrypt/btrfs on my laptop. I'm not sure what part causes it to hang and require a power cycle every other day or so after a few months of use, but that was quite annoying.

    I'm now hopeful that the Samsung EVO 840 will be my new "good drive" for a long longer :)

    It sucks that it took me so long to find a good drive, and so many backup/restore cycles, but I think I got it now. Wish me luck! :)

    2012/08/29 Linuxcon 2012 in San Diego
    π 2012-08-29 01:01 in Linux
    I was invited this year again to speak at Linuxcon 2012. This year, I gave a talk about sleep monitoring with arduino which went quite well.

    The conf had plenty of talks to see, and great parties. Just a few pictures below to give a taste, and I also have a lot of pictures of my visiting of San Diego.

    A few pictures from the keynotes:



    kernel panel
    kernel panel


    A few slides from talks:



    And we had some nice parties. The first night was a rooftop party:





    The second evening, we had a party on an almost-island, including a small cruise to get there:







    The 3rd evening, we had a beach Hawaiian party on Coronado Island, which was quite good:

    the long bridge that gets you there
    the long bridge that gets you there






    A few random picts from the conf:




    It was a good conf and a great time.

    See more images for Linuxcon 2012 in San Diego

    More pages: July 2002 February 2004 March 2004 November 2004 April 2005 August 2005 January 2006 July 2006 August 2007 November 2007 January 2008 October 2008 November 2008 December 2008 January 2009 May 2009 July 2009 August 2009 September 2009 November 2009 December 2009 January 2010 March 2010 April 2010 June 2010 August 2010 October 2010 January 2011 July 2011 August 2011 December 2011 January 2012 March 2012 May 2012 August 2012 December 2012 January 2013 March 2013 May 2013 September 2013 November 2013 January 2014 March 2014 April 2014 May 2014 October 2014 January 2015 March 2015 May 2015 January 2016 February 2016 June 2016 July 2016 August 2016 October 2016 January 2017 September 2017 January 2018 March 2018 December 2018 January 2019 January 2020 May 2020 January 2021 September 2021 March 2023 April 2023 December 2023 June 2024

    Contact Email