Dealing with a severe performance regression

Same hardware, booting into baseline kernel vs an updated version of our kernel. Normal testing on our part has our updated kernel at a significant performance advantage to the baseline.
Imagine my surprise when, while trying to diagnose an issue on a machine we are building, we found baseline to be significantly faster. The only thing that has really changed between this and the older kernel is a newer driver for RAID.
Will fix this … it was a surprise when I found it. Hasn’t impacted customers,. This definitely has convinced me of the value of automated benchmarking systems with reasonable test cases. We are testing a number of other kernels to see if they are affected.