Lab JR5 quickie benchmarks

I’ve seen some clustered file system results a few months ago where the vendor was happy to sustain something like 1.4 GB/s during their IO operations, and called this good. Something like 60 disks. Lustre, and some other bits.
Their approach (and most people’s approach) in this space is to start with a bunch of demonstratably slow servers/disks, and aggregate them. Which eventually gets you to the performance you are looking for, albeit with low performance density, large expenditure of capital, large investment in space/power/cooling.
There is a benefit … a profound and unassailable benefit … to better performance. Not simply as an enabling technology (though it is that), but as a mechanism of cost reduction. If you can do the same job in less power, less footprint, and therefore less acquisition and operational cost, why would you even consider alternatives ?
So we’ve built an in-lab JR5 machine for us and our customers to use for testing.
Very quick tests, in a non-optimal, non-tuned configuration, using our previous generation RAID (newer ones are significantly faster), we are seeing streaming writes of 2.3 GB/s and streaming reads of 3 GB/s, for 192GB of data. The unit has 24 GB ram.

[root@jr5-lab ~]# fio sw.fio
Run status group 0 (all jobs):
WRITE: io=192GB, aggrb=2,295MB/s, minb=2,351MB/s, maxb=2,351MB/s, mint=85611msec, maxt=85611msec
[root@jr5-lab ~]# fio sr.fio
Run status group 0 (all jobs):
READ: io=192GB, aggrb=3,013MB/s, minb=3,086MB/s, maxb=3,086MB/s, mint=65213msec, maxt=65213msec


I can’t emphasize enough, that this is a single box, these are RAID6’s in a reasonable configuration (we’ve seen youtube videos and read reports of people using 48-72 drives in a JBOD or in RAID0 to meet/beat these numbers, but we aren’t interested in artificial and unlikely to ever be used scenarios … only realistic drive aggregations need apply).
The hardest thing we’ve had to deal with has been FUD from other vendors, the ones reading off vendor spec sheets. Way back in the Sun Thumper/Thor days, we regularly heard of 3 GB/s being thrown at us as competitive. We’d have to pry their use case out, and then get it tested. It was never … ever … even remotely close to 3 GB/s. That was a marketing number. Real world numbers people experienced were 300-600 MB/s in their best case.
While SSD/Flash and other tools such as RAM drives are becoming more visible, we are certainly pleased that this unit represents one of the fastest measured spinning rust based storage systems, with some of the highest performance density possible per rack. Of course, we do have Flash and SSD based units, but it will be a while before those are as economical as this unit is, or even providing similar storage density.
We will re-run with our new generation of RAID fairly soon.