Parallel building debian kernels … and why its not working … and how to make it work

So we build our own kernels. No great surprise, as we put our own patches in, our own drivers, etc. We have a nice build environment for RPMs and .debs. It works, quite well. Same source, same patches, same make file driving everything. We get shiny new and happy kernels out the back end, ready for regression/performance/stability testing.
Works really well.
But …
but …
parallel builds (e.g. leveraging more than 1 CPU) work only for the RPM builds. The .deb builds, not so much.
Now the standard mechanism to build debian kernels involves some trickery including fakeroot, make-kpkg, and other things. These autogenerate Makefiles, targets, etc. based upon the rule sets.
Fine, no problem with this. I like autogenerated things. Actually I often like programmatic generated things better than human generated things, as the latter invariably have crap you really don’t want in there. Not that the others don’t, but there is mysticism around the existence of some things in peoples build environment, versus empirical reality.
The canonical mechanism is to use CONCURRENCY_LEVEL=N for N=some integer.
Fine. Use it in the make file. And …
We have a stubborn single threaded build. It will not change.
Fine, lets capture output and make it verbose. Look for concurrency level in output. See if something is monkeying with it.

scalablekernel@build:~/kernel/3.18$ grep CONCUR out
cd linux-"3.18""" ; export CONCURRENCY_LEVEL=8 ; fakeroot make-kpkg -j8 --initrd --append-to-version=.scalable --added_modules=arcmsr,aacraid,igb,e1000,e1000e,ixgbe,virtio,virtio_blk,virtio_pci,virtio_net --overlay-dir=../ubuntu-package --verbose buildpackage --us

I look to see where that is coming from. Looks like debian/ruleset/targets/ Which is in turn, coming from /usr/share/kernel-package/ruleset/targets/ . Look for concurrency and see this snippet

debian/stamp/build/buildpackage: debian/stamp/pre-config-common
        @test -d debian/stamp      || mkdir debian/stamp
        @test -d debian/stamp/build || mkdir debian/stamp/build
        @echo "This is kernel package version $(kpkg_version)."
ifneq ($(strip $(HAVE_VERSION_MISMATCH)),)
        @echo "The changelog says we are creating $(saved_version)"
        @echo "However, I thought the version is $(KERNELRELEASE)"
        exit 1
        echo 'Building Package' > stamp-building
# work around idiocy in recent kernel versions
# However, this makes it harder to use git versions of the kernel
          dpkg-buildpackage $(strip $(int_root_cmd)) $(strip $(int_us))     \
            $(strip $(int_uc)) -j1 -k"$(pgp)"  -m"$(maintainer) < $(email)>"
        rm -f stamp-building
        echo done >  $@

[starts banging head against desk again]
Force concurrency level to 1, AND then force -j1. Oh dear lord.
Lets see if switching these back to 8 helps (8 core machine).
Why … yes, yes it does …
[Update] The deities of kernel building are not kind. It appears that parallel build in debian actually breaks other things that it should not break. I have a choice of a very slow (1+ hour) kernel + module build that works, or a fast (roughly 5-10 minute) kernel + module build that fails because of a causality violation (e.g. someone couldn’t figure out how to fix their code to run in parallel).
So, if I have time in the next month or so, I am going to find that very annoying serializer, take it out behind the barn, and put it out of my misery.

1 thought on “Parallel building debian kernels … and why its not working … and how to make it work”

  1. Um, make-kpkg has been deprecated for quite some time. Use the deb-pkg target in the kernel source, which has worked for me in parallel (4×8 core). It’s the way documented in the Debian Kernel Handbook. Looks like module-assistant is the “official” way to build out-of-tree modules, but dkms is handling virtualbox, etc. for me (up to version incompatabilities).
    And I don’t recall the gizmo for CONCURRENCY_LEVEL in make-kpkg, but I do recall it was documented…

Comments are closed.