On the test track with a new model JackRabbit

This is going out to a customer in about a week, though we have a little time to run tests. Tuning these units is like tuning an engine … and you like to tweak and tune, you eventually learn where the parameters of this engine, from tuning, hit good efficiency. You get a feel for it. You tune one parameter, the vibration pattern changes slightly. You tune another parameter, you get another overtone. Eventually you get a feel for where this engine is strongest, and you start tuning with that. You put the unit on a test track and put out some great times for your benchmark cases. But you know its not enough. And you continue to evolve this. And you keep working at it. The response changes in subtle ways, until you hit that one area. Most folks won’t get this unless you’ve done it before. You have a sense that you just hit a whole new vista of performance. So you take it out to the test track. Crack the throttle a little. You hear and see smooth power, unlike you’ve seen quite before, over your smaller case. Ok, its time for the real tests. On the race track. Crack the throttle hard. Lets see what this critter can do. This is what happens.

Basic specs: 24x 2TB drive JR4 unit, 48 GB RAM, 12 processor cores. Drives in RAID6 configuration. Not RAID0. Our kernel/tools/driver configs. Tests: 128 GB streaming write, and then 128 GB streaming read. Previous records we have on this: 1800+ MB/s read, 1400+ MB/s write. These results:

Run status group 0 (all jobs):
  WRITE: io=128GB, aggrb=1,661MB/s, minb=1,701MB/s, maxb=1,701MB/s, mint=78759msec, maxt=78759msec
    READ: io=128GB, aggrb=2,208MB/s, minb=2,261MB/s, maxb=2,261MB/s, mint=59261msec, maxt=59261msec

Yes, we really did just sustain 1.7 GB/s writes and 2.2 GB/s reads. The simpler dd tests also show this:

[root@jr4-1 ~]# dd if=/dev/zero of=/data/big.file ...
2450+0 records in
2450+0 records out
82208358400 bytes (82 GB) copied, 53.8856 seconds, 1.5 GB/s
[root@jr4-1 ~]# dd of=/dev/null if=/data/big.file ...
2450+0 records in
2450+0 records out
82208358400 bytes (82 GB) copied, 36.7853 seconds, 2.2 GB/s

Not my favorite benchmark, but bonnie++ results

[root@jr4-1 ~]# bonnie++ -u root -d /data -s 96g:1024k -f
Using uid:0, gid:0.
Writing intelligently...done
Rewriting...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine   Size:chnk K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
jr4-1        96G:1m           1068893  59 578061  47           1663432  66 245.1 145
Latency                         256ms     639ms               585ms     100ms
Version  1.96       ------Sequential Create------ --------Random Create--------
jr4-1               -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16   651  11 +++++ +++   541   7   673  11 +++++ +++   477   7
Latency               213ms     258us     258ms     287ms      27us     338ms
1.96,1.96,jr4-1,1,1278051551,96G,1m,,,1068893,59,578061,47,,,1663432,66,245.1,145,16,,,,,651,11,+++++,+++,541,7,673,11,+++++,+++,477,7,,256ms,639ms,,585ms,100ms,213ms,258us,258ms,287ms,27us,338ms

More tests to come, then the long burn-in runs. I think we have some more head room on the writes. This said, I do find it amusing at least that some of our data suggests that this single box is faster than some others storage clusters. And this single box isn’t that expensive. Just imagine a rack full of 10 of these in a nice storage cluster (aka siCluster) running a nice fast parallel file system.