fun with SCSI targets

Had some fun today with our SCSI target. Its a very nice system, very powerful. Not terribly easy to use. But it works well. We have tools we developed around it to make it easy to use.
Creating iSCSI targets works nicely with our target code. It builds the target, sets up the infrastructure. Done with thin provisioning, its pretty fast and mostly painless.
Well, it was until we discovered that the stack, while including /etc/initiators.allow and /etc/initiators.deny support, and even honoring them at a coarse level, had different semantics than the documentation indicated.
Caused us some grief, but the LUN masking was working. Just not the way we thought it was.
LUN masking enables the initiators (the thing requesting the block device) to only see the relevant LUNs for them. But in this case, the LUN masking was moved out of the initiators.allow and initiators.deny.
Ugh. At a coarse access control level, it worked. But at the fine grain LUN masking, we had to use a different approach. Sort of like tcpwrappers for baseline ACL, and a secondary mechanism for finer grain control.
Its fine, just caused us to have to do some things differently. For ~50 targets, our workaround isn’t very fast. Our preferred mechanism should be much faster, but it looks like some things haven’t been as well documented in the new code.
Sadly, this SCSI target isn’t the new one going into the kernel. That one, LIO, has a very different management interface, not to mention a different execution model. I am not at all sure how well they will scale to hundreds, thousands, and more targets. This isn’t a critique of ours or the LIO code. Moreover, our stack provides SRP, and LIO doesn’t. This is a problem, we aren’t ready to give up SRP. Our stack supports RDMA over IB (SRP), while LIO doesn’t yet.
Hopefully this will get fixed soon, and the two teams will work together. Hopefully this won’t be a repeat of the LVM2 – EVMS bit from a few years ago. In retrospect, the wrong stack (LVM2) won there (IMO). Our stack and LIO are both complex, both solving hard problems, in different ways. We can always recode around new management interfaces. Giving up important functionality? Not so much. has an article about this. We are mentioned in it. Search for LIO and SCST. Or Scalable Informatics.

This is important to us, as we have some plans to develop some products, which will hook into the target interface. So having one that works well, is simple to develop/debug, and well supported is critical. Vlad and team have been wonderful at support, and I am not willing to simply ditch them. Nicolas and the rest of the LIO team are very good as well, have an interesting stack.
Decisions. My issue with SCST was a minor annoyance, solved (with a bit of effort), and I’ll try to get some patches to fix the docs. Maybe even implement my RFE with them. Thinking about our development, we will try to build it as a core function, and a target interface. Not sure if we can, but we will look at it.