- Git-crypt and deploying secrets along with binaries
- Hashicorp Vault
- Square Keywhiz
- AWS KMS
- Lyft Confidant
- Roll your own
All seem to have pros and cons depending on use cases and how mission critical the service you are offering is.
So what do you do to solve this problem in your world?
http://kubernetes.io/v1.1/docs/user-guide/secrets.html
I can't remember which issue this was on, but it seemed like there was some discussion on their GitHub project about making pluggable secrets backends (HashiCorp's Vault was mentioned).
Kubernetes' secrets API is still very basic, but I think the fundamental concept is very sound and has a great foundation to continue building on.
If you are running on cloud, you would probably want your cloud provider to give you service secrets and rotate them somehow. AWS/Google Compute metadata service or Azure Key Vault are capable of doing this but I don't think they entirely map the microservices world because ACLs are set on the VM instances, not microservices specifically.
Our service is built on top of docker deployed on CoreOS with fleet and etcd. Most of our secrets & runtime configuration was stored in etcd, which was our attempt to store the config in the environment (http://12factor.net/config).
With Kubernetes, life is much simpler. Gone are the silly fleet configuration files, and the bootstrap scripts I used to configure etcd. Moreover, Kubernetes' secrets volume means I can have my configuration and secrets easily plugged in.
There are definitely other great solutions out there, but I'm sold on Kubernetes.
https://github.com/kubernetes/kubernetes/issues/10439#issuec...
Also on access control, any process with root on any node in your cluster can get access to all your secrets (since the kubelet needs to be able to do so). There are no user access controls either; any cluster admin can dump all the secrets.
This stuff is clearly documented, so it's not an indictment on k8s; I just get the feeling that the feature isn't really ready for production use yet.
That still leaves the secret in plain view on the nodes that run the pod that needs the service. It would be great to be able to umount the secret when not needed anymore.
The project is brand new and we'd love to hear your feedback: https://github.com/asteris-llc/vaultfs
1. create a secret key, store it with the code of your App 2. store the encrypted credentials in env vars
Later on we even launched our own solution for our clients, an app_secrets.yml file, which can be edited via Dashboard. http://help.fortrabbit.com/secrets
The nice thing is, that this file is partly managed by the platform for it's own credentials and partly by the user.
That has been running for a while now. The adaption rate is low until now. It turned out that not everything will fit into that ONE fault. Blackfire.io and NewRelic run as PHP extensions, thus the API-keys are stored with the extension setting.
We have also discussed to implement an some open source "Secret as a Service" but came to the conclusion that this can too easily turn into to be a SPOF.
I am amazed that this topic is getting discussed again and I have learned about many new concepts here.
The initial trust problem boils down to trusting the API that controls the provisioning of your infrastructure. Failing that, you have to ask a human to manually authorize new nodes to retrieve secrets (that's how puppet approves new agent certs).
On the subject, I typically store a file containing env variable export statements on S3. When the box is provisioned, the file is downloaded to it. Since the box has role-based access, there is no point in downloading and deleting the file: any process on the box can download it again from S3 at any time. Basically, I trust that the EC2 instance will remain secure. Then the file is source'd in any context where my application code will run.
For applications outside of AWS, I just keep a local non-version-controlled copy of the secrets, and then upload them to the server when I provision it.
1) Create an S3 bucket. Remove all permissions from it
2) Create an IAM role - give it explicit read permissions to just that bucket (there's a HOWTO at the bottom of this article: http://mikeferrier.com/2011/10/27/granting-access-to-a-singl...). When you start an ec2 instance, you can give it one (and only one) IAM instance role.
3) Put your secrets or configs in a file on that bucket. For example, config.json or whatever format you choose.
4) On your instance or container, use the aws-cli on when your app starts to copy that file down from S3, then read it into memory in your application and then delete it.
It's a bit of a hack but you can now easily restrict access to that secrets bucket, and only your running instances/containers can access it. The secrets only exist in running app memory. Now don't allow SSH access to those instances :)
2) Create a S3 bucket
3) Write a bucket policy that whitelists specific IAM Roles to specific key paths within the bucket.
[0]: https://github.com/TechnologyAdvice/Cryptex [1]: http://technologyadvice.github.io/lock-up-your-customer-acco...
I'm sure each product listed conforms to one from a small set of design patterns. Has any credible analysis of these designs been published? Are competing offerings likely to evolve toward a stable converged solution? Or is there something in this problem that remains fundamentally unsolved?
I appreciate it's turtles all the way down, but I'm wondering if anyone has proven the merits of some approaches over others, or components within the approaches at least.
At startup the container reaches out to the Secretary daemon that holds the master keys, using public key cryptos to authenticate itself. The Secretary deamon uses Marathon to authenticate containers (checking their public keys stored in env vars) and validate that they're authorized for the specific secret in question (checking that the encrypted secret is indeed part of the containers env vars).
Meaning that Marathon is the single source of trust of which container can access what secrets. The problem then becomes controlling who and how changes are made to the Git repo containing the CD config, which is something Github does well with roles, status/deployment API and pull requests.
We had a similar problem as some describe with the distribution of the initial secret (i.e. Vault token) and one time Vault tokens being cumbersome in dynamic scaling envs. We didn't want the cleartext token ending up in config files nor in the https://github.com/meltwater/lighter config we use to drive our continuous delivery pipelines that go into Mesos/Marathon. We also had some other aspects like
* Wanting to keep secrets, app config and code versions promoted together throughout or deployment pipelines. Seeing secrets as another type of app config we wanted to track all config and versions for an app in the same way, in the same place to avoid mismatches or deployment dependencies.
* Wanting to enable our very independent devteams to easily manage secrets for their services, same was as they manage the app config, versions and rollout of their services. And delegate management of what service is authorized for what secrets to devteams (with both automated checks for unencrypted secrets, and some gentle manual coaching post-commit)
* Versioning and rolling upgrades for secrets? E.g. how to roll out a new secret in a Marathon rolling upgrade? Creating and managing versioned keys in Vault seemed somewhat cumbersome.
Perhaps something like that could be used to solve your initial secret distribution problem or even handle the secrets themselves until Vault has solved the initial secret problem..?
http://keyappliance.strongauth.com/
You can secure the root for it with TPM or HSM.
Thanks!
I've been evaluating most of these same options for my use case, but haven't made any decisions yet.
You're putting a lot of faith in a very complex and not- well- tested codebase if you rely on Consul ACLs to protect secrets.
Of the tools listed in the OP, I feel really good about Square Keywhiz; I'm still rolling it out in my first environment, so I can't say for sure, but I appreciate the level of effort that's gone into only doing secret storage and making sure it is exhaustively tested to spec.
Thanks for your input!
https://www.distelli.com/docs/user-guides/securing-your-appl...
Disclaimer: I'm the founder at Distelli
How do you manage key storage securely? Can people at your company see my secrets? If somebody comes with a court order will you give them my secrets and not tell me? What encryption algorithms do you use? What experience do you have in reducing attack surfaces from internal and external threats? Is any of your software open source? Has your software been audited? Is it PCI (or any other standard) compliant?
We provide you with an agent that you install on your own servers and that agent is marked as a key management server. That agent is contacted to do asymmetric key encryption.
Here is a more detailed blog post about this: https://www.distelli.com/blog/keeping-your-application-secre...
Also we use standard encryption algorithms and have not written our own crypto (and never will).
I hate having to manage web.config but I get your point about keeping attackers at bay (and not providing pivot points).
[1]: http://docs.asp.net/en/latest/fundamentals/configuration.htm...
Accessing secrets as needed at runtime instead requires some kind of extremely reliable service nearby. This is what I find most concerning about Vault since it can lock on you if the cluster goes down.
Whatever you choose, make sure you are comfortable with it, it's easy to deploy and work with.
Since our stack isn't on AWS, it kind of throws out AWS KMS and Lyft Confidant (since it is built on AWS). I'll keep digging into Vault and the other options put forward in this thread. Thanks again.
