Of course, this means more work ahead

Our client code that pulls configuration bits from a boot server works great. But the config it pulls is distribution specific. Where we need to be is distribution/OS agnostic, and set things in a document database. Let the client convert the configuration into something OS specific. This is, to a degree, a solved problem. Indeed, etcd is just a modern reworking of what we did with the client code … using a fixed client (e.g. no code) and just doing database queries for information. Then they take the key value pairs and they do something with them. There’s nothing terribly special about that. We’ve been doing a more flexible version of this for years (not simply a kv store). What I am envisioning now is more interesting … think a replicated/distributed document database, client code that can read the document database, as well as a transformation database that maps OS specific things and the configuration document database into a consistent and repeatable control mechanism. I’ll likely have to disable chunks of systemd, or work on having it talk to this system and grab units/services from it. Or just work around it, by giving it the bare minimum work to do, and then taking over after its done. Past experience has been that in-distro control planes are often geared towards very different use cases than what we want, and attempting to build in the functionality we need to their control planes is an exercise in futility. A good example is the whole booting process. Each distro has some concept of a “livecd” type boot which is conceptually similar to what we do, though we realized that the overlay mounts could be problematic. So we built real ramdisks, and unpacked a file system into them. Rather than using a squashfs and an overlay. It is possible with the squashfs and overlay to wind up in a situation where your system cannot boot due to an overlay inode full scenario or similar. This is unacceptable. Our ramdisk method will boot given sufficient RAM (currently 8GB or so, but this is our kitchen sink build … as in “everything, including the kitchen sink”). The OS in the ramdisk is mutable, but the mutations are non-durable. So you can inflict tremendous damage to the running OS image. And completely fix it with a reboot. Which again, the durable nature of the overlay makes it possible to have a persistent broken state. Which is unacceptable. Happily our on-disk OS is installed using the same tools as our ramdisk booted version. No config differences apart from durable boot drives. In the short term, I can rewrite the OS specific setup (networking, etc.). In the longer term, I’ll get to work on the config doc architecture and clients.