Linux kernel 2.6.26.5 is buggy, 2.6.27-rc6 works

Alrighty. Been struggling to get an operational 2.6.26.5 kernel working for a customer. This is supposed to be the next generation of our supported kernels, replacing the now aging 2.6.23.14 kernel (you think ours is old? look at RHELs). It works fine on a Ubuntu system. All the things we needed built it, do in fact, work. The problem was the immediate kernel panic on a RHEL5.2 system. 11 seconds in, it couldn’t find /dev/root (the root directory). Couldn’t find it by label, by direct access, …

I spent saturday working on figuring it out. I came to the conclusion that there was something buggy in this kernel, or the way it interacted with the RHEL 5.2 system. Fine. I pulled down 2.6.27-rc6 patches, and built it this morning. Applied it on the system. Same .config file (with the make oldconfig of course, and a few more features turned on). See if this is it, or if I have to go through figuring out whats broken in RHEL versus Ubuntu. Got a nice reward for this effort:

[root@ach ~]# uname -a
Linux ach 2.6.27-rc6 #2 SMP PREEMPT Sun Sep 14 11:18:07 EDT 2008 x86_64 x86_64 x86_64 GNU/Linux

Ok, I am feeling better now. Was starting to doubt both my sanity and my kernel-fu. We will be patching this up, waiting for 2.6.27 final, and beating the heck out of it over the next several months. Change in plans. Next kernel we are supporting is 2.6.27. Has lots of nice goodies in it. NFSoverRDMA comes built in (and is enabled) in our kernel. As is iSER, Infiniband, and many other very-useful-for-HPC-and-storage-things. Like xfs and jfs, ocfs2, … . Unlike RHEL 5.x/4.x kernels. Wondering aloud here: if Oracle started on their RHEL rebuild (unbreakable linux) because of the OCFS2 omission (and likely future btrfs omission). RHEL gives some weak arguments against using xfs and other things, when they feel they have competitive technologies. Their product, their choice. We have to go back and fix some of their (IMO mistaken) choices, in order to allow systems to function. I am guessing Oracle got tired of this as well. So 2.6.27 it is.