Taking our lumps and some of our partners lumps while we are at it

We had a recent event that badly irked a customer, and rightly so. It took far too long for us to be able to get a replacement part for them. I want to talk about this a little. I won’t name the customer or partner, or the product. The punchline for the customer was that they got their replacement part more than a month late. For an enterprise shop. This was IMO unacceptable. The customer is pissed at us over this … I understand and accept this, but it annoys the hell out of me, as we are taking lumps for the partner who couldn’t deliver their RMA product. The customer didn’t see any of what happened behind the scenes. So we got to take their lumps for them. Since we included them in the solution, yeah, we deserve to share some of the lumps. That was a judgement failure on our part.

It all started with a failure we noticed while working on servicing the system. The partners' product failed to show the correct status. We recorded the serial numbers from the unit, opened up an RMA for overnight shipping while we were at the customer site, and assumed it would be taken care of. It wasn’t. This partner has exactly one distributor in the US. RMAs have to transit them. This is where the RMA was opened. This partner let it slip. We kept on them, asking where things where, and could we have the tracking number. And nothing. After a week of this, we tried to escalate. This is an enterprise customer, they expect and we are trying to deliver enterprise class service. Why wouldn’t the distributor get back to us with this tracking number and the RMA documentation? Another week. The customer correctly asked where things were. We tried escalating further, still no communication. We pushed on every contact we had with the distributor. Nothing. Not a thing. Another week. The customer is getting frustrated. Again, we pushed on every contact we had with the distributor. Nothing. Not a thing. Another week. The customer is pissed. This is delaying their project. I understand and accept this. Again, we pushed on every contact we had with the distributor. Nothing. Not a thing. Finally, it gets to the point where I pushed every button I had access to on the partners side, and I pushed every button on the distributor side. VPs on the partner side got involved and asked some very pointed questions of the distributor. Nothing happened. Not a thing. So out of desperation, I called the partner and asked them if I could put a new unit on my credit card, and have it overnighted to the customer. I figured I’d lay into the distributor later, they are obviously a no-op, and I would take them to task on this in a huge way later. Lots of business was at risk here for us, and by extension, for all of the folks. This woke some folks up. We started getting call backs from the distributor indicating that they were sorry that it had fallen through the cracks. The partner admitted to us that they really weren’t ready for this market. They didn’t have the bits set up, so the partner and distributor had gotten into something of a race condition, waiting to see which one would respond first. The customer never saw any of this. They weren’t privy to the conversations, the cajoling, the yelling, all the rest of this. They just knew they had something we sold them that didn’t work, and we couldn’t replace it (and they didn’t know why). We couldn’t replace it because the distributor was unwilling to generate an RMA for an enterprise product and enterprise customer for overnight replacement without direction from the partner, who didn’t have this support setup and in place. The partner did have the capability to overnight the replacement, they chose not to during this interval. If there ever was an Epic Fail … sigh. End result 1st: Customer got their replacement system. It took ~2 months to get an overnight shipment. This was ridiculous. I completely agree. 2nd: We apologized profusely to this customer. Both the partner and us. We were stuck in that we couldn’t replace it without the partner … and they got the clue that something was FUBAR when I wanted to buy a new unit (these ain’t cheap) to ship it out as replacement. I told the partner that I was going to have a very difficult to hear conversation with them after this exchange if I had to buy it. They got the message. 3rd: We strongly recommended to the partner that they find additional distributors, as they obviously have a single point of failure. Lest you think that all is fine and dandy with this now … I am pretty sure this customer is still pissed at us, I offered to meet with them on a recent trip, and this offer wasn’t accepted. The project we were part of was shelved, possibly permanently, which likely also foreclosed upon future revenue from this customer. So we took our lumps. Because our partner was unable to overnight ship a sub $10k USD part, we have likely lost far more (order of magnitude +) amount of business. Good job. [notes of bitter sarcasm should be used when you verbalize this for the appropriate effect] And we get to take the partners lumps, with a customer pissed at us. The customer have some valid reasons to be pissed at us, as we couldn’t get the other folks to act in a timely manner. The way we handle this with disk drives and other parts is we find another distributor and buy the replacement and ship it ourselves. Makes life better, allows us to internally hide the distributors issues from the customers, usually with slight delays. Not in this case. There was one distributor. We could not work around their issues. There was one supplier. We could not work around their inability/unwillingness to ship. It was our responsibility to get this done, and because of external issues, we couldn’t get it done in a reasonable time. This is monumentally frustrating to us. I am not making excuses. I am just pissed that their failures appear as ours, even though we were doing everything possible to prevent this failure. But we were responsible. We owned it. And we rode it down to the bitter end, and took ownership, and the lumps. Even when others should have. So we have learned a bit from this. When we do partnerships, one of the critical questions is going to be (on the hardware side) “suppose we need a replacement part for an enterprise customer, and we need it tomorrow … explain your process, and how we need to manage this.” If they look at us blankly and blink rapidly, we may need to exclude them from enterprise deals. If they come back with a solid plan and process … we are going to test it. To make sure it works. No repeats of this. Ever.