rebuilding our kernel build system for fun and profit

No, really mostly to clean up an accumulation of technical debt that was really bugging the heck out of me. I like Makefiles and I cannot lie. So I like encoding lots of things in them. But it wound up hardwiring a number of things that shouldn’t have been hardwired. And made the builds brittle. When you have 2 released/supported kernels, and a handful of experimental kernels, it gets hard making changes that will be properly reflected across the set. Basically we want our changes/patches built into the kernel, not post-installed. Especially since our appliances boot diskless, post-installation is sort of a no-op. So I broke out the things I needed to break out. And made saner “objectified” Makefiles. With some includes and other bits. Very modular, very easy to update/manage. Very easy to test. But I made some cut and paste errors, and spent the last hour+ fixing the errors. Ugh. It does look like I’ve fixed them all though. This gets back to another issue, and that is driver development is largely outside of the kernel source. You can often find 2+ year old drivers without specific features/capabilities, or support for newer hardware. Which makes for exciting times. No … not exciting … painful. Our build integrates all theses drivers in. This is, as it turns out, non-trivial as, as noted, a number of these drivers are developed out of kernel. So we have to patch them to build correctly in the kernel. And then … and then … we have to patch them for kernel API changes (yes the )(**&%&$^& API does change). We find these changes with errors in our builds. And then we have to track down where the change was, and then patch. Or write adapter code. And then we beat the hell out of the kernels. Torture tests. Lets write a petabyte or two, while computing PI, e, and other things to a billion digits or so while our packet howitzers are going full steam. The reason we build our own kernels comes down to stuff like this: your friendly neighborhood distro kernel isn’t stable under what we and our customers would consider medium load, never mind what we would consider heavy load. Our kernels need to be rock solid. They need to be fast and mostly tuned out of the box. The build system is part of this. Making it work in a way I am happy with is important. You can’t build awesome performance systems on top of crappy infrastructure … hardware OR software. You shouldn’t even try.