I might have a point about udev. A reasonable point.
But the problem appears to be one or the other packages we installed. Retried it on my sacrificial machine here at home. No IB cards, but I was able to boot our stable kernel after installing the Mellanox OFED. I’d prefer to use our OFED build, and looks like I’ll be able to do that.
By self inflicted I mean that I ran the installer script one too many times. It is definitely buggy. But I have a good snapshot I can work from. So even if I bork it again, I can recover faster.
Annoying. Udev is fragile, but not spontaneously so. Now I have the workflow down for it, I won’t re-inflict this. Will likely include a disk with this image for the customer. Just in case.