The text below is a comparison I did between Linux Software Raid and Hardware Raid in August 2001 while working at VA Linux Systems. The quick version is that software Raid on a single 80MB/s was almost twice as fast as the hardware raid card for raid 5 ext3 / lvm (lvcreate -i 5 -I 32) -> Striping ---------- Output ---------- ----------- Input ---------- 47290 KB/sec 69% CPU 63052 KB/sec 52% CPU ext3 / lvm over md0 (raid5) ---------- Output ---------- ----------- Input ---------- 32313 KB/sec 49% CPU 62755 KB/sec 65% CPU ext3 / Mylex DAC1164P / 5 QUANTUM ATLAS_V_36_SCA (2.2.18) ---------- Output ---------- ----------- Input ---------- 23309 KB/sec 24% CPU 24145 KB/sec 22% CPU The original test follows: ---------------------------------------------------------------------------- After a couple of claims here that lvm over sw raid would be slow, and hopes from me that more recent $2000+ mylex cards would have somewhat decent performance, I ran a series of tests. Here are the results. Enjoy, Marc ################################################################################ Goal: ---------------------------------------------------------------------------- Get real numbers on the current performance of software raid 5 and lvm, ext2 versus the current ext3, comparisons with NFS performance, and hardware raid Bonnie isn't the end to all all benchmarks, nor was it the only interesting thing to test, but I was just interested in rough throughput numbers, not specialized file operation tests. Marc 2001/08/08 Executive summary: ---------------------------------------------------------------------------- - Just like when I did similar tests two years ago when I was trying to show Dan Jones that Mylex cards where much slower than software raid, I found again, that software raid, in the worst case (lvm on top of sw raid 5) is more or less twice as fast as hardware raid on a fairly expensive DAC1164 (2.6 faster on reads and 1.4 faster on writes) - lvm with striping (no redundancy) is more than twice as fast as a DAC1164 - lvm over sw raid5 is actually a tad faster than straight sw raid 5 and almost as fast on reads - the 2.4 sw raid code apparently pauses or throttles a rebuild during heavy disk access so that bonnie saw no speed degradation whatsoever. the same obviously can't be said of mylex raid, were performance goes way down during a rebuild (sorry, I didn't want to fail a file server to get exact figures :-D) - ext3 is a still a bit slower than ext2 on writes (differences on reads are negligeable), but it's now 86% to 99% as fast. Apparently, when the block subsystem is slower to start with, the difference between ext2 and ext3 narrows to almost nothing (1% difference in the lvm over sw raid 5 case) - If you look at the full bonnie bench results per block size, you'll notice that ext3 performance is much smaller for blocks smaller than 4K, and when you have 4K or better, it goes back to being about as fast as ext2 (actually, the magic number is somewhere between 512B and 4K) - straight lvm without striping is of course slow since it's bound by the bandwidth of one disk, but lvm with striping is reasonably fast. What's nice is that you select striping at the time you create a logical volume (i.e. a partition), but the shortcoming (outside of the obvious absence of redundancy) is that once you do that, you can't extend the volume group to use more physical volumes anymore. For this reason, lvm over sw raid 5 arrays is a clear win, and this is what I intend to use from now on. - For NFS serving, with either sw raid or hw raid with ext3, I can saturate my 100Mbit connection (I get about 10MB/s both read and write). I guess it means that putting a gigabit ethernet card in file servers is not necessarily a bad idea anymore. Specs: ---------------------------------------------------------------------------- 5 IBM DRHS36D drives 36G VA 2250 used with onboard AIC controller and SCA backplane Bus speed unfortunately limited to 80MB (probably the fault of the SCSI chip) Dual 600Mhz PIII, 512MB Ram Software raid configured with: raid 5 chunk-size 32 parity-algorithm left-asymmetric magic:~# cat /proc/sys/dev/raid/speed_limit_{max,min} 81920 1024 Filesystems formatted with: mke2fs -b 4096 -R stride=8 /dev/md0 Kernel: linux 2.2.7-ac3 ext3 0.9.5_247ac3 lvm 0.9.1b8 Mylex card and driver version (running on VA 2.2.18): ***** DAC960 RAID Driver Version 2.2.9.1 of 22 January 2001 ***** Copyright 1998-2000 by Leonard N. Zubkoff Configuring Mylex DAC1164P PCI RAID Controller Firmware Version: 5.08-0-83, Channels: 3, Memory Size: 32MB PCI Bus: 3, Device: 8, Function: 0, I/O Address: Unassigned PCI Address: 0xF4210000 mapped at 0xC0009000, IRQ Channel: 19 Controller Queue Depth: 128, Maximum Blocks per Command: 128 Driver Queue Depth: 127, Scatter/Gather Limit: 33 of 33 Segments Stripe Size: 64KB, Segment Size: 8KB, BIOS Geometry: 255/63 SAF-TE Enclosure Management Enabled Client for NFS tests is running 2.2.18pre11-va2.9-servers1 too, and had to go through a couple of switches to reach the NFS server. Disclaimer: ---------------------------------------------------------------------------- I didn't do professional benchmarks with multiple runs, averaging and so forth, this was done quickly (well, it still took 6 hours), and the idea was to get ballpark figures, so don't rely on those for anything serious :-) Summary of bonnie results for File Size: 1000 and block size: 16384 ---------------------------------------------------------------------------- ext2 over md0 (raid5) during raid rebuild ---------- Output ---------- ----------- Input ---------- 32008 KB/sec 53% CPU 66535 KB/sec 61% CPU ext2 over md0 (raid5) after raid rebuild ---------- Output ---------- ----------- Input ---------- 32434 KB/sec 54% CPU 66715 KB/sec 63% CPU ext3 over md0 (raid5) ---------- Output ---------- ----------- Input ---------- 29314 KB/sec 49% CPU 65683 KB/sec 63% CPU ext2 / lvm ---------- Output ---------- ----------- Input ---------- 17948 KB/sec 14% CPU 19174 KB/sec 13% CPU ext3 / lvm ---------- Output ---------- ----------- Input ---------- 16935 KB/sec 23% CPU 18622 KB/sec 14% CPU ext2 / lvm (lvcreate -i 5 -I 32) ---------- Output ---------- ----------- Input ---------- 54404 KB/sec 47% CPU 63346 KB/sec 51% CPU ext3 / lvm (lvcreate -i 5 -I 32) ---------- Output ---------- ----------- Input ---------- 47290 KB/sec 69% CPU 63052 KB/sec 52% CPU ext2 / lvm over md0 (raid5) ---------- Output ---------- ----------- Input ---------- 32460 KB/sec 55% CPU 65093 KB/sec 67% CPU ext3 / lvm over md0 (raid5) ---------- Output ---------- ----------- Input ---------- 32313 KB/sec 49% CPU 62755 KB/sec 65% CPU ext3 / Mylex DAC1164P / 5 QUANTUM ATLAS_V_36_SCA (2.2.18) ---------- Output ---------- ----------- Input ---------- 23309 KB/sec 24% CPU 24145 KB/sec 22% CPU ext3 / lvm over md0 (raid5) / NFS v3 ---------- Output ---------- ----------- Input ---------- 9479 KB/sec 15% CPU 10113 KB/sec 19% CPU ext3 / Mylex DAC1164P / 5 QUANTUM ATLAS_V_36_SCA / over NFS v3 (2.2.18) ---------- Output ---------- ----------- Input ---------- 9428 KB/sec 13% CPU 9749 KB/sec 24% CPU Full bonnie results: ---------------------------------------------------------------------------- ext2 over md0 (raid5) during raid rebuild ********************************* File Size: 1000 Block Size: 512 ---------- Output ---------- ----------- Input ---------- 34870 KB/sec 66% CPU 65994 KB/sec 81% CPU File Size: 1000 Block Size: 4096 ---------- Output ---------- ----------- Input ---------- 32466 KB/sec 54% CPU 66813 KB/sec 61% CPU File Size: 1000 Block Size: 16384 ---------- Output ---------- ----------- Input ---------- 32008 KB/sec 53% CPU 66535 KB/sec 61% CPU ext2 over md0 (raid5) after raid rebuild ********************************* File Size: 1000 Block Size: 512 ---------- Output ---------- ----------- Input ---------- 31990 KB/sec 63% CPU 66158 KB/sec 80% CPU File Size: 1000 Block Size: 4096 ---------- Output ---------- ----------- Input ---------- 32291 KB/sec 53% CPU 66625 KB/sec 61% CPU File Size: 1000 Block Size: 16384 ---------- Output ---------- ----------- Input ---------- 32434 KB/sec 54% CPU 66715 KB/sec 63% CPU ext3 over md0 (raid5) ************* File Size: 1000 Block Size: 512 ---------- Output ---------- ----------- Input ---------- 18043 KB/sec 87% CPU 41524 KB/sec 84% CPU File Size: 1000 Block Size: 4096 ---------- Output ---------- ----------- Input ---------- 31182 KB/sec 49% CPU 64054 KB/sec 68% CPU File Size: 1000 Block Size: 16384 ---------- Output ---------- ----------- Input ---------- 29314 KB/sec 49% CPU 65683 KB/sec 63% CPU ext2 / lvm ********** File Size: 1000 Block Size: 512 ---------- Output ---------- ----------- Input ---------- 18071 KB/sec 23% CPU 19217 KB/sec 19% CPU File Size: 1000 Block Size: 4096 ---------- Output ---------- ----------- Input ---------- 17963 KB/sec 14% CPU 19146 KB/sec 13% CPU File Size: 1000 Block Size: 16384 ---------- Output ---------- ----------- Input ---------- 17948 KB/sec 14% CPU 19174 KB/sec 13% CPU ext3 / lvm ********** File Size: 1000 Block Size: 512 ---------- Output ---------- ----------- Input ---------- 13633 KB/sec 53% CPU 18806 KB/sec 33% CPU File Size: 1000 Block Size: 4096 ---------- Output ---------- ----------- Input ---------- 17154 KB/sec 24% CPU 18788 KB/sec 15% CPU File Size: 1000 Block Size: 16384 ---------- Output ---------- ----------- Input ---------- 16935 KB/sec 23% CPU 18622 KB/sec 14% CPU ext2 / lvm (lvcreate -i 5 -I 32) ******************************** File Size: 1000 Block Size: 512 ---------- Output ---------- ----------- Input ---------- 48107 KB/sec 65% CPU 63203 KB/sec 71% CPU File Size: 1000 Block Size: 4096 ---------- Output ---------- ----------- Input ---------- 54386 KB/sec 52% CPU 62776 KB/sec 52% CPU File Size: 1000 Block Size: 16384 ---------- Output ---------- ----------- Input ---------- 54404 KB/sec 47% CPU 63346 KB/sec 51% CPU ext3 / lvm (lvcreate -i 5 -I 32) ******************************** File Size: 1000 Block Size: 512 ---------- Output ---------- ----------- Input ---------- 22171 KB/sec 88% CPU 46748 KB/sec 88% CPU File Size: 1000 Block Size: 4096 ---------- Output ---------- ----------- Input ---------- 34149 KB/sec 48% CPU 61607 KB/sec 58% CPU File Size: 1000 Block Size: 16384 ---------- Output ---------- ----------- Input ---------- 47290 KB/sec 69% CPU 63052 KB/sec 52% CPU ext2 / lvm over md0 (raid5) ******************* File Size: 1000 Block Size: 512 ---------- Output ---------- ----------- Input ---------- 33761 KB/sec 65% CPU 64416 KB/sec 82% CPU File Size: 1000 Block Size: 4096 ---------- Output ---------- ----------- Input ---------- 32071 KB/sec 54% CPU 65174 KB/sec 63% CPU File Size: 1000 Block Size: 16384 ---------- Output ---------- ----------- Input ---------- 32460 KB/sec 55% CPU 65093 KB/sec 67% CPU ext3 / lvm over md0 (raid5) ******************* File Size: 1000 Block Size: 512 ---------- Output ---------- ----------- Input ---------- 19112 KB/sec 90% CPU 43864 KB/sec 88% CPU File Size: 1000 Block Size: 4096 ---------- Output ---------- ----------- Input ---------- 31201 KB/sec 47% CPU 62380 KB/sec 67% CPU File Size: 1000 Block Size: 16384 ---------- Output ---------- ----------- Input ---------- 32313 KB/sec 49% CPU 62755 KB/sec 65% CPU ext3 / Mylex DAC1164P / 5 QUANTUM ATLAS_V_36_SCA (2.2.18) ********************************************************* File Size: 1000 Block Size: 512 ---------- Output ---------- ----------- Input ---------- 17889 KB/sec 38% CPU 28082 KB/sec 31% CPU File Size: 1000 Block Size: 4096 ---------- Output ---------- ----------- Input ---------- 23425 KB/sec 25% CPU 23548 KB/sec 23% CPU File Size: 1000 Block Size: 16384 ---------- Output ---------- ----------- Input ---------- 23309 KB/sec 24% CPU 24145 KB/sec 22% CPU ext3 / lvm over md0 (raid5) / NFS v3 **************************** File Size: 1000 Block Size: 512 9668 KB/sec 25% CPU 10678 KB/sec 31% CPU File Size: 1000 Block Size: 4096 ---------- Output ---------- ----------- Input ---------- 9743 KB/sec 17% CPU 10398 KB/sec 26% CPU File Size: 1000 Block Size: 16384 ---------- Output ---------- ----------- Input ---------- 9479 KB/sec 15% CPU 10113 KB/sec 19% CPU ext3 / Mylex DAC1164P / 5 QUANTUM ATLAS_V_36_SCA / over NFS v3 (2.2.18) *********************************************************************** File Size: 1000 Block Size: 512 ---------- Output ---------- ----------- Input ---------- 9862 KB/sec 23% CPU 9513 KB/sec 32% CPU File Size: 1000 Block Size: 4096 ---------- Output ---------- ----------- Input ---------- 10028 KB/sec 17% CPU 9816 KB/sec 22% CPU File Size: 1000 Block Size: 16384 ---------- Output ---------- ----------- Input ---------- 9428 KB/sec 13% CPU 9749 KB/sec 24% CPU ################################################################################