This is going out to a customer in about a week, though we have a little time to run tests.
Tuning these units is like tuning an engine … and you like to tweak and tune, you eventually learn where the parameters of this engine, from tuning, hit good efficiency. You get a feel for it. You tune one parameter, the vibration pattern changes slightly. You tune another parameter, you get another overtone. Eventually you get a feel for where this engine is strongest, and you start tuning with that. You put the unit on a test track and put out some great times for your benchmark cases.
But you know its not enough. And you continue to evolve this. And you keep working at it. The response changes in subtle ways, until you hit that one area. Most folks won’t get this unless you’ve done it before. You have a sense that you just hit a whole new vista of performance. So you take it out to the test track. Crack the throttle a little. You hear and see smooth power, unlike you’ve seen quite before, over your smaller case.
Ok, its time for the real tests. On the race track. Crack the throttle hard. Lets see what this critter can do.
This is what happens.
Basic specs: 24x 2TB drive JR4 unit, 48 GB RAM, 12 processor cores. Drives in RAID6 configuration. Not RAID0. Our kernel/tools/driver configs.
Tests: 128 GB streaming write, and then 128 GB streaming read.
Previous records we have on this: 1800+ MB/s read, 1400+ MB/s write.
Run status group 0 (all jobs): WRITE: io=128GB, aggrb=1,661MB/s, minb=1,701MB/s, maxb=1,701MB/s, mint=78759msec, maxt=78759msec READ: io=128GB, aggrb=2,208MB/s, minb=2,261MB/s, maxb=2,261MB/s, mint=59261msec, maxt=59261msec
Yes, we really did just sustain 1.7 GB/s writes and 2.2 GB/s reads.
The simpler dd tests also show this:
[root@jr4-1 ~]# dd if=/dev/zero of=/data/big.file ... 2450+0 records in 2450+0 records out 82208358400 bytes (82 GB) copied, 53.8856 seconds, 1.5 GB/s [root@jr4-1 ~]# dd of=/dev/null if=/data/big.file ... 2450+0 records in 2450+0 records out 82208358400 bytes (82 GB) copied, 36.7853 seconds, 2.2 GB/s
Not my favorite benchmark, but bonnie++ results
[root@jr4-1 ~]# bonnie++ -u root -d /data -s 96g:1024k -f Using uid:0, gid:0. Writing intelligently...done Rewriting...done Reading intelligently...done start 'em...done...done...done...done...done... Create files in sequential order...done. Stat files in sequential order...done. Delete files in sequential order...done. Create files in random order...done. Stat files in random order...done. Delete files in random order...done. Version 1.96 ------Sequential Output------ --Sequential Input- --Random- Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size:chnk K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP jr4-1 96G:1m 1068893 59 578061 47 1663432 66 245.1 145 Latency 256ms 639ms 585ms 100ms Version 1.96 ------Sequential Create------ --------Random Create-------- jr4-1 -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 651 11 +++++ +++ 541 7 673 11 +++++ +++ 477 7 Latency 213ms 258us 258ms 287ms 27us 338ms 1.96,1.96,jr4-1,1,1278051551,96G,1m,,,1068893,59,578061,47,,,1663432,66,245.1,145,16,,,,,651,11,+++++,+++,541,7,673,11,+++++,+++,477,7,,256ms,639ms,,585ms,100ms,213ms,258us,258ms,287ms,27us,338ms
More tests to come, then the long burn-in runs. I think we have some more head room on the writes. This said, I do find it amusing at least that some of our data suggests that this single box is faster than some others storage clusters. And this single box isn’t that expensive. Just imagine a rack full of 10 of these in a nice storage cluster (aka siCluster) running a nice fast parallel file system.