We use Ansible's vault feature to decrypt a few centralized secret files onto machines at deploy time. This lets us commit the encrypted text of the files. (The source of truth for the key is in Trello, IIRC, but it could be anywhere you have to auth in as an employee to view.)
It's modestly annoying (operations like "check what changed in the secret configuration file as a result of a particular commit" are impossible) but seems like a reasonable compromise to ensure that e.g. nobody can insta-create an admin session if they happen to have a copy of our codebase and a working Internet connection.
Secrets are communicated to processes which need them in boring Linux-y ways like "file i/o" and "stuff it into an environment variable that the process has access to." If you're capable of doing file i/o or reading arbitrary memory, we're in trouble. Of course, if you can do either of those on our production infrastructure and also connect to our database, we've already lost, so I don't see too much additional gain in locking down our database password.
If you're starting from the position "I have a Rails app which has passwords in cleartext in database.yml" this is an easy thing to roll out incrementally: move the password from database.yml to ENV['RAILS_DB_PASSWORD'], spend ~5 minutes getting your deployment infrastructure to populate that from an encrypted file (details depend on your deployment infrastructure -- I am liking ansible, a lot, for this), verify it works, then change passwords. Voila; Github no longer knows your database password and your continuous integration system no longer knows your production credentials. One threat down; zero coordination required with any other system you use or any other team at the company. You can standardize on this across your entire deployment or not, your call, and it's exactly as easy to back out of as it was to get started.
A side project I'm working on comes under that domain (medical data), By any chance do you have any recommendations of books on this kind of stuff? I do all the usual OWASP/best practice stuff but most of my day job is LoB stuff and while security is important it's not quite the same as losing potentially thousands of peoples medical data.
The only plausible way this can make sense to me is if he said "Running your own key service is a pain, use Amazon KMS". But that's a simple service question, probably wouldn't have taken up as much space.
You don't need 3rd party code hosting on Github, just use Gitlab or JIRA. You don't need some external CI service, run your own Jenkins node. Chat and email should also be internal (we use XMPP, a local Mattermost instance would be an alternative) and SSL-only.
You can do all of this with basically 1 docker command per install on your own dedicated hardware with a fairly underpowered machine.
And this prevents leaking of all sorts of information, not just production database passwords. If you don't trust your engineers, you have bigger problems, as another poster pointed out, if they can modify your software to simply report the password back to them, or just login to production and decrypt it, you're dead in the water.
External service SLA can be joke. It's always aftermath thought. Damage control is always on the customer side because customer has to rotate / invalidate leaked credentials, so first step for me is to have a process to invalidate credentials as often as possible.
I agree that using AWS KMS is the same architecture as using some other KMS that you run yourself, you just garner the benefit of their software and operational capacity and you buy service; this is the same as any other PaaS service at Amazon or some other vendor.
What's the value of Cryptex, though? Why not just store KMS-wrapped secrets in your config file and have Amazon unwrap them? Then you wouldn't be dependent on any local crypto implementation and you could use other KMS features, such as AEAD.
All my side reading recently has been on writing high(er, I follow best practices with my other stuff) security systems across the entire stack, it still frightens me but I see a real need for the side project so I'm going to do everything I can to make it as secure as possible and take a shot.
I'm still going to speak to the company solicitor though just for belts and braces.
Oh and on the hosting, I won't be using any cloud services, Physical server out of a a state of the art DC a few miles up the road that is certified to my UK Gov standards as a provider, they pretty much tick every box I'd ask for though not cheap I can get an insanely powerful machine and they have a superb reputation, looking at approx 75 quid ($110) per month for a dual core i3-4160/8GB RAM w/1TB RAID or £145 ($210) a month for a Xeon 1231 with 32GB RAM and 2TB of RAID storage (that one has dual power supply, n/c) which if it's used isn't that expensive at all.
To address that you need process isolation between the storage of the cyphertext and the manipulation and use of cleartext. This eliminates the browser since for all intents and purposes it is not an isolated process. (You could still use the browser, but provide your tools as an extension that would, presumably, be inspected by users when it updated.)
That said, your solution takes care of a lot of other threat models, but it doesn't really protect users from you.
(Besides, one would assume has been backdoored by the Amazon staffers anyway)
For an organization requiring the highest available security, the ideal solution would be a privately operated hardware security module kept off the DMZ. However, that, as well as the idea of self hosting (and maintaining) the entire dev, test, deploy, and prod stack suggested by another commenter, isn't always within reach of a small, agile team looking to focus on their core competencies.
One could argue that it's possible for Amazon to have falsified the description of KMS as an HSM, or the certifications[0] they were granted for it, but I'd retort that an organization in a position to seriously question those claims shouldn't be using a remote solution anyway.
So, making the more rational assumption that such claims by Amazon can be trusted, their offering is quite secure: the HSM does not allow the export of any key, and exposes only the ability to load encrypted data into the device and have it produce the decrypted result over a secure channel, and vice versa.
Looking at the docs, it looks like the master key source is pluggable, so you don't have to use Amazon's KMS... but none of the other options inspire confidence (local file, fetch from URL, plaintext password, or no password).
At the very least, I'd like to see a plugin for using a key stored on a local TPM chip -- which almost any modern bare-metal server would be equipped with.
