'Amusing' benchmarketing ... without ever having run a benchmark!
By joe
- 2 minutes read - 278 wordsImagine you have a product, and you really haven’t measured its performance, but you want to make performance claims. So you take an “easy” way around this. You simply add up all your bandwidth or IOP data. Yeah, thats it, you add it up. No, I’m not kidding. You do this. Is this meaningful in the HPC world? Hell no. Do people do this? Hell yes. Is it wrong? Extremely. Should you call vendors out who do this? Damn right. Here’s a simple example. Imagine, we put say, I dunno, 4x QDR Infiniband ports in our machines. Does this mean we can claim that we have 12.8 GB/s of sustained bandwidth? And then claim we are unique in the world for doing so? And that we are the fastest? No, it doesn’t mean this at all.
Yet some folks are doing this. If you see such folks, please, call them on it? Please? Its hard enough to get meaningful numbers out there, when people’s entire benchmarking fu is reading performance data off the box the card/disk/ssd/flash came in, and multiplying by the number of units. A real HPC person would know better. Far better. They would never make that mistake, willingly. This does tell you something about the organization doing these things. Such numbers are disingenuous at best. Had a bunch of cases recently where we had to work through and explain the difference between a theoretical number and what people actually measure. Its annoying to have to fight through this. This is the case with 6G chassis, with network cards, RAID cards, SSD, Flash, … Amusing. In a shaking his head, can’t believe it sort of way.