Benchies: figuring out how to tune this thing …

Design is good, but it looks like we are rate limited on the PCIe gen 2.

128GB read from a single name space. 8 simultaneous threads.

Run status group 0 (all jobs):
   READ: io=126984MB, aggrb=5285.8MB/s, minb=5412.6MB/s, maxb=5412.6MB/s, mint=24024msec, maxt=24024msec

Yes, that is 5.3 GB/s. Still far south of what we can be doing, but I’ve verified that we are rate limited to ~2GB/s per RAID with other tests. This looks like a card issue. This is closer to PCIe-x4 gen2 performance, not the x8 performance we are supposed to be getting. Working on this.

Getting something like 115k IOPs. Again, should be much higher. Starting to think I am hitting some stack driver latencies.

Viewed 30311 times by 4791 viewers

3 thoughts on “Benchies: figuring out how to tune this thing …

  1. Any other QPI traffic other than to/from the raid controllers? 5.3GB/s is starting to get up there. With 6.4GT/s QPI you should still have a lot of headroom, but if someone else is eating into that you might be getting pinched. Of course it sounds like you’ve determined the raid controllers are only getting 2GB/s anyway so you are probably right to start looking there.

  2. @Mark

    Looks like its a card chip issue … it wasn’t designed for lower latency high IOPs. Switching to a different unit bumped the performance up to 170k IOPs. As we have 6 more groups of these drives, our hope is that a reasonable scenario of about 1M+ IOPs at roughly 10GB/s. Actual use may vary of course …

  3. Cool, looking forward to hearing what you are eventually able to get. Test settings are always welcome too. What’s the drive/raid configuration like?

Comments are closed.