Upped sustained speeds on new JackRabbit unit

I forgot to mention this. Odd.
Our updated JackRabbit (JR4 f.k.a JRM) unit being burnt in over the last few days for a customer. Putting obscene loads on it. Trying hard to crash it. Really.

From fio (apart from the 4M buffer size issue, I really like fio)

streaming-write: (groupid=0, jobs=1): err= 0: pid=12270
write: io=6,397GiB, bw=1,498MiB/s, iops=365, runt=4479113msec
clat (msec): min=1, max=4,560, avg= 2.73, stdev=10.33
bw (KiB/s) : min=    0, max=2427840, per=101.23%, avg=1552322.90, stdev=2228
30.41
cpu          : usr=0.09%, sys=8.08%, ctx=3264682, majf=0, minf=256


Let me walk you through the parsing of this. 1 MiB = 1024 KiB = 1048576 B. So 1498 MiB/s = 1571 MB/s or 1.57 GB/s.
Which as you might notice, was sustained for 4479 seconds. A bit more than an hour. With a CPU load of 0.09% in user space, 8% in system space, and a few million context switches. You might also notice the maximum bandwidth measured during an interval. 2.48 GB/s. We didn’t sustain it, but it burst to that. Remember, this is direct IO, uncached IO. So this isn’t cache speed.
For each of the 9 storage disks in each raid (ok 11 disks, but 9 of them are in use for data storage in RAID6), this represents about 138 MB/s. We see some instances of these drives bursting this high, but not sustaining there. This is from tests outside of this environment. We see typically 87 MB/s sustained write speed, though often on more modern firmware revs, we are seeing 110 MB/s. These drives, we are seeing close to that on writes. So the fluctuations were really fluctuations and not stable measurements … not a signal per se.
Of course, reads went up as well.

streaming-read: (groupid=1, jobs=1): err= 0: pid=12422
read : io=6,400GiB, bw=1,593MiB/s, iops=388, runt=4212362msec
clat (msec): min=1, max=2,977, avg= 2.57, stdev= 3.13
bw (KiB/s) : min= 4894, max=1845493, per=100.18%, avg=1634342.17, stdev=94968.36
cpu          : usr=0.08%, sys=13.23%, ctx=3277615, majf=0, minf=1273


Again, working on our parsing …
1593 MiB/s = 1670 MB/s = 1.67 GB/s.
Based upon other measurements (crude dd bits) we have been talking about 1.2 GB/s writes and 1.5 GB/s reads. I think I can comfortably say that we are measuring (sustained) 1.57 GB/s writes and 1.67 GB/s sustained reads, uncached (e.g. straight to/from disk). But this is RAID6, not JBOD, and is configured in customer deployable configurations.

7 thoughts on “Upped sustained speeds on new JackRabbit unit”

1. Joe, what is the significance of your using RAID 6 vs JBOD? Is there a desire for JBOD among storage customers? Why? And is it generally slower?
thanks,
Peter

2. @Peter
We occasionally get customers sending us performance numbers from a competitor that took 48 drives and ran their tests as JBOD configured drives. It wastes everyone’s valuable time to have to deal with this … very few people would run their units as JBOD without some sort of RAID atop it. Yet we are stuck in that customers look at the “performance data” and conclude that 2GB/s is greater than 1.67 GB/s, though 2GB/s is in an unusable configuration, and 1.67 GB/s is one of several usable configurations.

3. OK, so JBOD can give better test results, but it is not a practical setup for a customer to use, which is why you test in real-world redundancy configurations.

4. I am no expert on this, but how about publishing RAID1 or RAID10 speeds? Since there are no parity calcs, I would expect speeds == JBOD for writes, >=JBOD for reads. With the disks cheaper and cheaper, quite a few companies went from RAID5/6 to RAID10.

RAID10 would be a little less than JBOD/2 speed for writes. With careful layout, we can get it to JBOD – δ for reads.
RAID6 is preferred for large streaming storage, RAID10 for databases.

6. Joe,
could you post some link for a quality read on RAIDs? I would love to read good article <10 (OK 20) pages. I am especially interested in RAID1 performance with hw- (like 3ware) and sw-raid controllers. I am also interested in high-end 2/4 CPU machines.
FYI: My impression and reading so far (Wikipedia, I do not like their RAID article too much) that w hw RAID1 controller the write-penalty compared to writing to a standalone disk was negligible as the incoming signal was split electrically (analog).

7. CORRECTION:
I had clicked “Submit” too quickly: In my last comment please ignore the end:
“…as the incoming signal was split electrically (analog).”
Sorry for this,