Testing iSCSI over 10 GbE, iSER over IB, SRPT over IB, ...

This will be short, no long discussion of benchmarks. Basically we tried JackRabbit as a target for many block oriented protocols. With 10 GbE, and with IB. I though 10 GbE would be badly beaten by IB in performance (real world, no ram disks here).

I think I was wrong. 10 GbE based iSCSI was quite simple to set up, pretty easy to tune, and actually nice to work with. Compare this to building SCST-SRPT or the right version of iSER or the correctly patched OFED for SCSI-TGT, or … I like IB. I really do. Building OFED is a bear. It is not easy. It is not download a tarball, compile for 2 minutes, install a driver kernel module, and bam, fast networks. 10 GbE is. The stack is simple. Setting it up is simple. Using it simple. Simple implies lower barriers to usage. I bet that the IB people could (if they wanted) integrate IPoIB directly into the driver, and make the actual driver builds nearly clean and simple as the 10 GbE. The 10 GbE card I played with is a pre-production unit. I am hoping that we get to play with it some more. It is quite nice. The nice thing was the the 10 GbE performance was comparable (within the same ballpark) as the IB performance for the IO operations. I don’t care if I have double the bandwidth if I never have any hope of ever using that bandwidth. With a real storage system like a JackRabbit, you are not going to be able to pump or pull more than some upper limit from or to the disks in some unit of time. What is nice is that the JackRabbit’s disks allow us to feed something at 10 GbE/IB speeds. What was remarkable to me was how well matched they were. We were seeing in the vicinity of 450-550 MB/s sustained performance to physical disk on writes and a little lower on reads. On pre-release 10 GbE hardware. Again this is a sub-$10k storage unit, and the storage was RAID6 with 1 hot spare (13 drives in the RAID6). Native on disk speed we tamped down to 750 MB/s (slight de-tune to optimize overall storage throughput, and avoid cache-buffer “beat” effects). With a little tuning, I bet we can get somewhat higher iSCSI performance. There always will be overhead, call it ~10% or so for protocol etc. So our numbers aren’t bad at all.