The danger in modifying precision built and tuned machines …

… is that they won’t be precision tuned after you are done with them.
And worse, much of this is self-inflicted in various cases.
We try to ship absolutely peak performance machines. Tuned as much as possible, though in some cases, customers make requests that go against high performance. We try to explain the issues, but customers are always right, even when they aren’t.
In a number of cases, customers wipe what we’ve done. For various reasons, ranging from interest in learning how to set them up themselves, through “its our process”. This generally means that our tuning gets lost.
Every now and then, someone says, “hey, lets roll this forward to the new rev”, and we beg them to let us test/tune it in the lab first. Nothing sucks as bad as loading up the latest new shiny and discovering that it is slow as molasses in January.
So this has been happening to Centos/Red Hat 5.x units we’ve been shipping. Some customers want them rolled to 6.x. And as it turns out, 6.x is a pretty serious regression as compared to 5.x. We’ve confirmed with with numerous ISVs whom all have experienced problems. We’ve had discussions with developers who’ve confirmed what we’ve observed.
Its a no win situation. Customer really wants 6.x for some particular feature or functionality, and they have to give up so much performance in the process …
Of course, as I had noted in another post a while ago, we … us … get blamed for the loss of performance. Yeah …
We’ve been able to recover to about 80-85% of 5.x performance with our test kernel, and a helluva-lotta tuning. But from what I’ve found, there are some new and unfortunate performance implications of the various elevators in 6.x, that actually run counter to what we’ve experienced in the past. Worse, one elevator works better for reads, while another works better for writes. And tuned daemon? Best thing to do with it is to remove it entirely. But thats just the beginning.
We try hard to help our customers. And we will keep doing this. But what do you do when problems are, in part, self-inflicted?