I’m getting some odd numbers with 4 disks in RAID 10. hdparm -t gives between 102MB/s and 120MB/s with 4 WD10EADS Green 1TiB drives to /dev/md3.
The individual drives give me 75MB/s. Shouldn’t RAID 10 speed this up substantially?
Well it is RAID 10 and not RAID 0. RAID 10 as You probably know incorporates striping and mirroring. so You won’t get 4 times greater speed. More like 2 times. Also software RAID is never as good as hardware one.
That’s not true, Bender
Software RAID can be as fast as some hardware RAID solutions. If you have enough CPU power, the CPU can usually calculate faster than some of the cheaper hardware ASICS. I also have a significant amount of system memory for caching.
As such, I am not looking for 4x throughput. And RAID1 is fast in its own right for reads, though it suffers a write penalty. However, RAID 10 should be faster than a single drive, and should be decently fast. RAID 10 incorporates elements of both 0 and 1, and should be about as fast as 3 drives in this case. I am wondering if I should be checking any parameters such as chunk size or read ahead.
My chunk size now is 512K and read ahead is 8192K. I tried dbench and got 200MB/s but on 8 procs and when I increase the procs I get significantly lower throughput. Should I disable NCQ? Should I rebuild the array with 1024K chunk size?
The file system is JFS on this volume and I know it is deprecated but it works so well for many things and is faster for deletes than XFS.
I think that the chunk size is to great :)?? Normally (i only used RAID 0) the default chunk size was 32kB. Are there any benefits in having such big chunk size?? I’ve read some posts and people didn’t see any benefits with chunks greater than 128kB.
Second: Software RAID will NEVER be as good as hardware RAID. The cheap ones are in fact software RAID (you can tell when You get a disk with drivers).
Real hardware RAID doesn’t need any disks. You could search for some benchmarks and see that software RAID isn’t too good but at least not worse than fake RAIDs?
XFS after some tweaking (there are many tutorials out there) can be very good, at least with big files, performance with small files sucks noodles.
P.S. I’ve read in many places that NCQ isn’t really beneficial (at least on normal desktop PC)
If i’m not wrong NCQ improves things when there is a long queue and apparently on a desktop it even makes things worse from what i read but hey, better to check it personally right as everyone has different needs 
A few things:
This is not a desktop, this is a home server. I’m using this mostly for banging around on, file serving, maybe openvpn and hyperic after a while.
Also, the chunk size is the “Chunk” of data that is divided up on the different disks. If you are using more linear/sequential reads (Like in a file server with large files vs small files) a larger chunk size means more contiguous data from one drive. I also set the read ahead (caches in data past the entry point on read request so that it is already available when requested) to 8192K but I have also tried as low as 512K and as high as 16384K.
One more point. Software RAID often gets ‘poo pooed’ as it is many times not as robust as hardware RAID. The RAID in the linux kernel is pretty robust. It offers features that many software controllers do not, and as this is not a desktop, the CPU power will pretty much be wasted if not used. The features, scalability, reliability and fault tolerance of a cheap $20 “RAID” card do not match that of the kernel based RAID. Also, the tools are pretty tops as well as the ability to ensure that you can recovery by using a different machine as opposed to needing “Card XYZ1A firmware 1.9” where if you use “Card XYZ2A firmware 2.1” your data will never be recovered.
Sure, if we are talking $1900 HBAs with built on RICS coprocessors and battery cache with write through support, we may have something to talk about, but in the sub $900/card market, the software RAID on Linux, Solaris, etc are pretty good if you can spare the cycles.
I also don’t really need much performance on small files. I will likely be using small files at some point, but I can use Ext3 or Reiser for those. Most of the files I will be hosting will be large data blocks in the neighborhood of 400MB+
I wouldn’t trust benchmarks you used. It said on my PC that every disk performed with speed 75MB/s while in reality (i got 2x160GB and 2x500GB) the small one doesn’t exceed ~56MB/s and the big one ~75MB/s. So i wouldn’t trust it and i’d check it simply by copying some files somewhere.