in the testing lab

Thought a few people might like to see this. A new JackRabbit (JR4) being built. We are testing one of its RAIDs. Dealing with some MB issues, but otherwise, back onto stable ground. This is a single RAID card with 8 drives, 7 in a RAID6 + 1 hot spare. Large sequential streaming read and write. 4x larger than RAM in the machine. Caching is not relevant to this. No tuning. At all. Not even load balancing IRQs.

root@jr4s:~# !dd
dd if=/data/big.file of=/dev/null bs=16M count=1k ...
1024+0 records in
1024+0 records out
17179869184 bytes (17 GB) copied, 30.7842 s, 558 MB/s
root@jr4s:~# dd if=/dev/zero of=/data/big.file bs=16M ...
1024+0 records in
1024+0 records out
17179869184 bytes (17 GB) copied, 27.7003 s, 620 MB/s

More to the point, we had to specifically de-tune this motherboard, and the Linux kernel a bit in order to get stability. What we are seeing is remarkable, even if these numbers don’t seem stellar on the surface. With RAID6, you lose 2 drives to parity. This is just the breaks, part of the cost you pay for more resiliency. So of the 7 drives in the RAID6, 5 are (effectively) for data. Now look at those sustained data rates again. Reads off of 5 platters, with ECC going on (RAID6) at 558 MB/s or 111.6 MB/s sustained. Now look at the writes. Writes to the 5 platters, with ECC going on (RAID6) at 620 MB/s or 124 MB/s sustained. Now, for a gedanken experiment, imagine a server with, I dunno, 24 of these drives, in 2 groups of 12. 1 drive in each group for hot spare, 11 in a RAID6. Assuming nearly linear scaling (its a gedanken experiment, please do allow me that lattitude), we should expect writes at about … oh 2.2 GB/s sustained, and reads at oh, about 2.0 GB/s. This is with the box and kernel specifically detuned to increase stability. Now imagine, if you will, a box with 48 of these drives. With 3 groups of 16. 1 drive each for hot spare, 15 for RAID6. Hmmmm …. On a related note, we will be taking both of these boxes to the test track, and opening the throttle. Wide.