Finally fixed the day job DNS

This took far longer than it should have. This was in part due to my initial decision (since changed) to use dbndns at two sites (one internal, one in the cloud).
The TL;DR version. dbndns and its parent project, djbdns, are a royal pain to get up, operational, stable. I tried the packaged versions, the source, etc. Several different distributions (CentOS 6.x , Ubuntu 12.04, …). 4 weeks into this mess, I asked myself the critical question. Was this the right tool for the job? Would I or someone I hire, be able to maintain it?
Thats when the creeping realization came over me, that I had decided upon a solution long before I should have. That I needed to research it better. So I started researching. And came across the Wikipedia article. From that, and the table, it looked like my real options were BIND, unbound, and Maradns for my use case. I’ve used dnsmasq (and use dnsmasq) for many things. Its just not up to the complete task I want it to do. Which is a shame, as I know it best.
I’ve struggled in the past with BIND. I don’t like BIND. I don’t want to keep banging my head against BIND.
I want a very simple to setup system. Understandable syntax. Something that you can almost parse in your head by looking at it.
Thats unbound and Maradns.
I played a little with unbound. This is a very powerful system. Far more power and complexity than I need.
I played with Maradns. Built a domain config file for scalable on a kvm server. Did some testing. Darned thing just worked.
Then I had to put it down for a week while we finished the benchmark.
Picked it up last night at 11pm. By 12pm I had a completely working authoritative DNS server.
Ok, now to see how hard it is to spin this up in the cloud. Over to our Joyent account … ok, first I started on EC2, but EC2 no longer seems to like me. I tried buying a t1.micro server reserved instance, and it kept spinning, and spinning, and …
Ok, over to Joyent. Took a while to remember my password, but I got it reset. Purchased a new small instance … hmm pricing has dropped, but config is spot on. Chose a Linux Ubuntu JEOS instance. Spun it up. Logged in 5 minutes later. Updated OS, set password, keys, rebooted. Installed Maradns. Copied config file over, changed a few lines to represent the new IP associated with this. Spun it up. By 12:30am, I had a working secondary DNS in the cloud. Thank you Joyent, for making spinning up instances so bloody easy/quick/painless!
Over to our registrar, add in the new external name server.
Ok, now the acid test. Point the domain’s port 53 over to the new name servers and bite my nails …
Quick change, and … it worked. SOA came back correct, MX correct … everything correct. The websites worked, mail got through.
So then the next task was fixing sidocs. Had to do a little internal surgery, and I’ll ask Doug to fix up the themeing, but darn it, its working nicely now!
All of this to do a split horizon DNS. Maradns on two kvms (one internal, one external).
This worked well because a human being can read/parse the db.domainname file. And the csv2 file for the domain root is 4 lines long, very easy to parse. This worked well because kvm on Linux is so very nice for these sorts of tasks, and we can spin up new instances trivially on the machine. This worked well because Joyent makes great kit, and its painless … really completely painless … to spin up new instances.
I have to fix a few minor things, but its functional, and I’ve not heard complaints, the websites work, mail is being delivered.