Response to the 11+ GB/s unit was … incredible …

We showed two large speedometers, one with Bandwidth, one with IOPs. These measured their data right off the hardware (via the device driver and block subsystem mechanisms).

First, we ran a fio test with 96 threads reading 1.1TB of data in total. This took about 100 seconds or so.

Second, we ran a fio test with 384 threads randomly reading 8k chunks of data out of that 1.1TB.

Left them in a loop, with a big speedometer pair on the screen.

The bandwidth speedometer peaked at 14 GB/s or so, and settled to 11+ GB/s during the first test. The bandwidth speedometer peaked around 350k IOPs during the second. We pointed out that, contrary to many of our competitors numbers, what they see there, is what they can measure on their own devices … that these aren’t theoretical guesses as to performance, but actual measurements that they can replicate.

We had a line of people to speak with … it was quite good. Effectively no press about this (an oversight on my part, but my time is a zero sum game). Customers came up to us and asked us if it would work in scenario X, Y, and Z. They gave us some ideas on how to get this into their hands, and we have lots of calls to make now.

A number of our competitors walked in with anything from pessimistic to disdainful expressions on their faces. They didn’t speak to me, but watched as the unit cycled through its demo, and listened as I talked to customers. The emotions I saw play across their (the competitors) faces before they got control of themselves and quietly walked out, ranged from amazement and disbelief, crestfallen to fear. One was visibly ashen as he weaved his way out. Just who the hell do these Scalable Informatics people think they are?

The overall idea is that we came to the data gun fight with a freaking data cannon. And it isn’t even our biggest, baddest, or fastest. Not by any stretch.

We are working on bringing this (or a unit very much like it) to SC12. Project 26th will be there (though someone whose opinion I value and respect had suggested simply calling it “number 6” … takes me back to the old “7 of 9” project for the SGI display). Project Hyperspace will be there as well. And we should have some excellent press out around that time, formally announcing new and important partnerships. With good partners and excellent products in our booth (how cool is that!)

Viewed 113949 times by 5941 viewers


5 thoughts on “Response to the 11+ GB/s unit was … incredible …

  1. I figured you meant a different Number 6.

    If you can get your hands on the LLNL / FusionIO Graph500 code, that might make a fun benchmark. My reference code has only a trivial, slow “out-of-core” mode. I’m quite certain you’d place well. We are supposed to have better ways to slice & dice the results available someday, but I’m also supposed to have the next version done someday. sigh.

  2. Hi Joe:

    Nice! I wouldn’t mind giving Ceph a try on that box. Since we do full journal writes for every data write, the throughput would be limited to a max of half of what you see, but that’s ok because there’s nothing out yet that can push the kind of network throughput that thing can do.

    I bet you are having a lot of fun. 🙂

  3. @Mark

    We are planning for something like at around SC12 (possibly at our booth). Bug me offline if you’d like. We had trouble with the build for RHEL6 base, and we are reworking for a ubuntu 12.04 base for siCluster. When we are done, Ceph will *just work* on it.

  4. @mark

    One important thing: I’d like to see mmap performance on the unit with Ceph. We’ve got customers with huge data requirements, wanting single name spaces. mmap + cluster file systems is painful.

    I’ve run FhGFS on there (Andreas?) and it positively ROARS for TB sized data sets. Haven’t made those benchies public yet, plan to at SC12. Would like to do the same for Ceph.

    But mmap performance with FhGFS isnt that good. Nor is it for most things. But we have a large customer base using that and unable to move for a number of reasons, so we need to make that go faster.

  5. Neat, I’ll be at SC12 so I’ll try to stop by and take a look.

    Regarding mmap performance: Honestly I have no idea. I’ll add it to my list. 🙂

    I did ask Sage about it and he thought it would be good, but didn’t really know either. He did note that mmap isn’t cluster-coherent when you have multiple hosts with the same file mmaped…

Comments are closed.