Rumors of Infiniband's (imminent) demise ...

… are greatly exaggerated. John at InsideHPC has an article on Cisco’s move out of Infiniband which refers to another article on this. I have to basically say, I’ll believe 10GbE conquering IB when I see it. Every year is proclaimed to be the year that IB is vanquished by 10GbE. I was reporting on this stuff 2+ years ago, and asked very simple questions just last year. When will 10GbE come into its own in HPC? In storage, it could. There are a few very good 10GbE units out there, and we can use them quite well. 10GbE for storage is IMO, a no brainer, though IB for storage (if you have IB) is also a no brainer … to a degree. 10GbE makes using storage fast, quite easy. As we have some of the fastest and most reasonably priced storage on the market, we need as much network firepower and simplified connectivity as possible. Our 1GB/s NFS result over a single fibre with a single client and a single server is pretty amazing. Considering that the data was too large to be cached for reading/writing should say something about the performance of our units. But that it was done using plain old NFS over 10GbE, should say something about what we can do as well.

There are still a few things wrong with IB. TCP over IB has never received the attention it deserved, and now is a serious Achilles heel relative to 10GbE. You can’t just use TCP over IB as a faster gigabit, because its not that much faster … maybe 2x. 10GbE is about 5x faster than 1GbE for a single process. This in turn makes storage over IB dependent upon verbs or SDP or … which aren’t as common. Block storage using SDP or iSCSI is fine, and we support them, but iSER targets/initiators with IB aren’t trivial to set up. When you are deploying enmasse, trivial to set up is very important. But IB’s other advantages seem to outweigh its current liabilities. IB does have challenges ahead of it. I just don’t see 10GbE kicking IB out any time soon. We are at QDR IB now, with 1us latencies. When will 10GbE get 1us latencies? You need this for MPI codes which are sensitive to latency. Color me skeptical. 10 GbE adoption will happen, slowly, over time. Displacing IB? Some places maybe. Not everywhere, or even in the majority. The rumors of IB’s imminent demise are again spreading, and I believe, as in the past, they are premature.