Declarative won't necessarily give you cattle. What you want is a continuous automated build process, and immutable images. That way you have a reliable image that has been tested and you know works, you have documented the build steps, and get an error when the build fails. Over time you can make it more declarative, if being imperative is adding a lot of toil.

The important thing to remember is that improvements should happen gradually in small chunks; you don't have to have a perfect cattle system out of the gate, but you should be working your way there.

> You shouldn’t have a compiler on your webserver at all, and building a kernel should happen on a different machine than installing it.

More likely than not, if they've got execution permission on your webserver, they can send down a compiler to run. Plus or minus a c compiler doesn't open up much (anything?) they couldn't have compiled elsewhere or done in the scripting language or shell code or ... Unless you're running a static website or something. Plus, it's presumptious that all my servers are webservers. :p Do my other servers (other than build servers) get to have compilers because they're less naughty?

> And why isn’t the official kernel good enough?

Too many drivers, not enough local patches that are worth having but not worth pushing upstream. (Going upstream is a good metaphor, it often takes a lot of effort)

> Beyond the security surface that’s just asking to brick the machine; if you never make any mistakes it’s because you never tried anything very interesting with it in the first place.

That's what the console is for. Serial and IPMI consoles are way more convenient than vga consoles, but I've certainly screwed up bunches of times and had to use a console to sort things out.

> So basically /etc and UNIX are bad ‘cause they’re not declarative.

So you declare. ;) If it helps, I ran the script from Make, which is declarative (although not really the way we used it), and we mostly ran Erlang, which is claimed to be declaritive in Erlang: The Movie, but seems to be useful, so I'm not sure. I've never quite understood how the goal of declaritive system management matches with reality; it feels like a very leaky abstraction to me; the part that changes the system is necessary complexity, and hiding it away adds additional complexity. Maybe if you build a read-only root filesystem declaritively and run that? Which you can if you want? While you're saying I shouldn't do something the way I did it, my team managed a bunch of machines and a complex system with very few people, so I think the method works.

But I think about things differently than a lot of other people; I'm comfortable hot loading code to change uninterupted services, wheras others like to move traffic to new servers (and hope it all moves). You could transfer existing connections to your new servers, and that would be a lot of fun (and probably require a custom kernel), but nobody does that. Instead you either serve old connections with old code, or forceably terminate old connections. If I'm going to hot load code, it makes an immutable system directly opposed to operability. Why would I be interested in making my system less operable?

Adding on: farmera don't kill their livestock at the first sign of injury or illness; they won't do heroic interventions like you might do with a pet, but they'll certainly do simple fixes on the live cattle in the hopes of returning to health and retaining the value.