A little JackRabbit on a test track

I hadn’t mentioned it, perhaps I should have. We had been building/testing a unit, now sold and scheduled for shipping, which we wanted to see what it could do if we did some tuning.
We tweaked, we measured, we tuned, we listened. Did some trial runs.
Then we cracked the throttle wide open and let ‘er rip.

root@jr1:~# ./simple-write.bash
start at Sun Jan 27 13:31:49 EST 2008
100+0 records in
100+0 records out
13421772800 bytes (13 GB) copied, 15.0744 seconds, 890 MB/s
stop at Sun Jan 27 13:32:04 EST 2008
root@jr1:~# ./simple-read.bash
start at Sun Jan 27 13:33:25 EST 2008
100+0 records in
100+0 records out
13421772800 bytes (13 GB) copied, 17.441 seconds, 770 MB/s
stop at Sun Jan 27 13:33:42 EST 2008

This is a single RAID card, sub $10,000 machine. 890 MB/s is within 14% of theoretical maximum to disk for this unit. We may be able to get a little more, but tuning for that is hard.
This is, BTW, RAID6 with one hot spare. A real configuration that someone might use.
We ran this for several days, reading and writing multiple TB of data to see if we could crash this faster setting. Wasn’t able to, though it looks like I am seeing some odd page/buffer cache behavior that needs tuning. That is, for some of the tests, we were seeing absolutely ferocious 1000 MB/s sustained performance for long intervals, while this was blasting data to disk (again, far outside disk cache), but it wasn’t “stable” performance, in that OS jitter effects would then seem to blast it down to 600-700 MB/s. Average performance is where we reported it previously, with semi-stable performance plateaus at about 890 MB/s and 1000 MB/s, though the latter gets unstable after a few seconds, and reverts to one of the other lower plateaus.
These units (3U JackRabbits) will ship in our standard tuning mode until we can find a region of performance stability above that.
BTW: What I talk about stable, I do not mean the OS crashed, or the application aborted. I mean constant IO rates that do not change more than a few percent one way or the other. If you take the writing to mean that the unit is unstable, you have misread or misunderstood it. The unit is rock solid stable under weeks of maximum load from our testing. Stable refers to which particular region of performance we are observing, and whether or not we can remain in that region for extended periods of time.