Half open drivers ... OFED stacks with verbs ABIs that don't match the kernel's verb ABI ...

I just ran through another update exercise. IB cards, OFED stack. GlusterFS atop this. Cards are well known vendors cards. They work pretty well. But … only with very specific kernels. Other kernels need not apply. Our 2.6.32.22 kernel is pretty darned fast (so our customers tell us). Now lets build the OFED 1.5.2 … and see what happens …

To make a long story short, we wound up abandoning that approach. While we were faster in all aspects, the OFED stack wound up … somehow … having a verbs ABI that was mismatched to the kernel verbs ABI. Which meant … ib_send_bw and other things … like any verbs dependent app (like, I dunno, GlusterFS mebbe?) didn’t work. I guess I am confused … how could the OFED stack correctly compile if the ABIs are different? Unless they were doing something funky? Next time you build OFED by hand using install.pl, just for laughs, use it with the -vvv option. You’ll see some nice serious “funkyness”. Or look in the install.pl and you can see the “business logic” encoding to decide what does and does not get built. This is annoying. Maybe this is why Redhat doesn’t use OFED directly, but uses the packages on their own. The drive source that isn’t in the kernel … the half open drivers I talked about in the past … these are a problem. The ABI mismatches … and building against the wrong ABI … yeah, thats a problem too. Wasted nearly a day on a siCluster trying to work around these issues. Finally came up with a solution that works. Not especially happy with it, but hey, it does work.