Moreover, is there any sensible way at all to encrypt secrets without baking the secret key into your image? I can’t think of any.
I want to deploy an app that makes use of one or more fairly important secrets, but I haven’t found a sensible way to make it auto-scale while keeping the secrets on-premise.
As far as I can see, the only sensible solution is to create in-cloud/off-premise secret keys that can only be accessed by images signed with an on-premise secret key.
So,
1. Create secret key on an offline, on-premise machine
2. Produce application image, transfer to offline machine, sign with on-premise secret key
3. Create off-premise (in-cloud) secret, which can only be accessed by images signed with the on-premise secret key
4. Upload app image and signature to the cloud, allowing only this image access to the in-cloud secret
[1] https://github.com/kelseyhightower/kubernetes-the-hard-way/b...
1. Keep the secret key in memory. This can be annoying though as a system administrator needs to unlock every process as it comes up; automation can help but is rarely deployed. This is what Vault does for example. There is a long design doc that we all worked on with this as an eventual option[1] but it is unclear if there is a ton of utility for users to be unlocking API servers
2. Hide the secret key inside of hardware using something like an HSM or a TPM. This is really the state of the art however it doesn't really work well in many environments and requires good inventory management.
3. Give machines a stable cryptographic identity and exchange that identity for a secret key that is kept in memory. This solution sort of shifts the problem around but is generally a good tradeoff between automation and security. The idea is, similar to SSH host keys, on first boot a machine generates an identity and then a system administrator approves that identity and later gives that identity authorization to do tasks. Kubernetes has a lot of the groundwork in place for this and it is still underway[2].
[1]: https://github.com/destijl/community/blob/3418b4e4c6358f5dc7... [2]: https://github.com/kubernetes/features/issues/43
You can see the initial proof of concept[1]. It isn't secure yet, for a variety of reasons. But it is enough to play around with. Moving to a better encryption scheme will give us the ability to do locks and per-block validation.
[0]: https://github.com/latchset/clevis [1]: https://github.com/npmccallum/clevis/blob/fuse/src/clevis-fu...
Using PGP / GPG sounds like what you need
For instance, if we see that our worker cluster is has reached some consumed capacity threshold (CPU, memory, etc) then we need to add more worker nodes. The way we do this is via Auto Scaling Groups. Auto Scaling Groups work on launch configs that are essentially an AMI and instance user data. Getting an AMI that just has the Kubernetes stuff that can be started at will and told to join clusters at runtime has not been clear from the documentation I've read.
This guide is more of core recipe from which someone can bake automation with config management or custom images.
https://github.com/coreos/coreos-kubernetes/tree/master/mult...
I think one can choose which approach matches one's learning style, and I am very thankful we have both KTHW and coreos-kubernetes.
It takes care of all the annoying parts of Kubernetes and even has services like a full-featured Docker registry with ACLs and so on, a Docker build system and even a centralized logging mechanism (all optional, of course).
Running it in production. Couldn't be happier.
It works pretty well with Cloud Shell, in case you have corporate firewall issues. If your session is interrupted, run the commands that set the region and region zone again.
I can confirm that it costs about $6/day that the machines are provisioned, and is well worth it, but remember to run all the clean-up steps in the last chapter when you're done or if you're not going to finish it right away.
