[Update] Not all of the issues were with the supplier. I started investigating and found out that we deserved some of the lumps. Me in particular for not paying more attention to the situation as it evolved. I made the assumption that someone else was covering it, and I didn’t need to. As I’ve discovered, this was a mistake on my part.
The story is more annoying than I allude to here. We have some things to fix in working with suppliers, and helping them to manage their processes shipping to our customers, as well as our own processes.
Yeah, its pretty humbling to discover that a core assumption you made was simply not the case.
As I note, the buck stops with me. And now I know the various trajectories, I can make changes to prevent a recurrence.
EC2 wwas taken down by the storms running across the US. Parts of EC2 were anyway. And it took down Netflix and others. Hmmmm. We put our web and mail into EC2 specifically to avoid these sorts of problems. While we are working on getting our second line up on a different technology from our … Read moreWhat to think about cloud outages