* no detailed analysis of how the attack was undertaken. Its not even clear how the attacker managed to get in (was it a publicly exposed Jenkins? vulnerable bastion? what?)
* no analysis of what the existing matrix.org security perimeter looked like or how it could be made better.
* repetition of security tropes. Use VPN. Use Github Enterprise (wait wtf? Why not private repos in Github?). Don't use Ansible, use salt.
Ridiculous. I was looking forward to a nice long read about how this breach was undertaken. Hugely disappointed.
But yes, publicly exposed jenkins and repos lead to the compromise, not an uncommon story unfortunately.
Perimeter - I didn't see much evidence of one existing and I didn't go probing their networks to find out.
Security tropes are real for a reason, you don't have to believe me though.
Private repos in GitHub are still publicly hosted and are orders of magnitude easier to get into than having an in perimeter repo. They've leaked before and they'll keep on leaking. GitHub even made it harder for people to fork private repos to their own public accounts but it still happens.
Can you provide some actual instances of this happening? Genuinely curious, as my org is currently migrating from enterprise to cloud.
You mean the past-tense verb led, not its metallic homonym lead. :)
Firstly having a private network for your infrastructure isn't a one stop solution for keeping attackers out.
Secondly using Github Enterprise or self hosted GitLab doesn't make up for storing secrets in Git.
Looking forwards to the proper write up.
And yes, using GHE or self hosted GitLab doesn't make up for storing secrets, but it at least keeps them out of the public eye so the effects are less brutal. Its still bad to store secrets in a code repository.
My whole point is that you can reduce risks easily, yet some people don't for some reason.
The sales-pitch for Salt (against Ansible) is ridiculous and misguided.
I just checked out the Salt SSH module and even if they used salt they would still have this issue. Then answer here is to not use the default /etc/ssh/sshd_config value of #AuthorizedKeysFile .ssh/authorized_keys .ssh/authorized_keys2. Uncomment and remove authorized_keys2.
I feel like this is an important part of the story for anyone looking for teachable infosec moments.
When I say the attacker was persisted via CM, I'm pointing at his own notes, nodding to broken CM, the requirements of supporting the CM and availability of the config files.
I also sanity checked the sshd_config file on my systems, they're all set to a sane default:
"AuthorizedKeysFile .ssh/authorized_keys"
FWIW I prefer to treat CM data as "valuable" information for this reason.
Either way, a DNS hijack is not great, but not nearly as bad as the initial compromise.
"The API key was known compromised in the original attack, and during the rebuild the key was theoretically replaced. However, unfortunately only personal keys were rotated, enabling the defacement."
We will write this up in a full postmortem in the next 1-2 weeks.
We’ll publish our own full post-mortem in the next 1-2 weeks.
> One of the more interesting pieces of this was how Ansible was used to keep the attacker in the system.
Fwiw the infra that was compromised was not managed by Ansible; if it had been we would likely have spotted the malicious changes much sooner.
You should not be running package managers on production servers. Or any of the other things salt, ansible, chef, puppet can do.
CVE-2019-1003001, CVE-2019-1003002 -> Anyone with read access to Jenkins can own the build environment.
CVE-2019-1003000 -> I didn't get a lot of the details on this but it basically looks like "broken sandboxing, you can run bad scripts".
This is also a good resource: https://packetstormsecurity.com/files/152132/Jenkins-ACL-Byp...
Don't deploy a public-facing Jenkins, especially if it has credentials attached to it. It's really hard to secure, especially if pull-requests can run arbitrary code on your agents.
Jenkins / CI is the sudo access to most organizations.
Example: https://twitter.com/R1CH_TL/status/1118559239084158977
So any reasonably competent hacker can re-validate the entire forum's content and votes, reasonably quickly reimplement the whole thing, and/or fork the forum at any time.
What can you learn from the compromise? Never use an agent. Kill it with fire^H^H^H^H -9.
2) SSH agent often starts automatically, frequently without user interaction (even if you specify `-i keyfile`). SSH client and DBus are both culprits here; there's also other culprits too.
3) There are often multiple different agents installed on Linux desktop systems. For example, ssh-agent, gnome-keyring, seahorse, gpg-agent ... the list goes on. Good luck auditing that.
4) Without `-i keyfile`, SSH will present to the remote server all keys, in sequence, cached in your agent (and will cause trouble with active firewalls from too many authentication attempts)
5) If the keyfile you specified in `-i keyfile` does not authenticate, then SSH will fall-back to using keys cached in your agent. That's especially frustrating since you might want to know that the key you specified was rejected!
6) Removing the executable flag from ssh-agent is not a permanent solution: updates will often overwrite the program with a new file and reset the executable bit. Obviously the same goes for renaming the program (that one causes a hell of a lot more noise in logs btw; programs seem to complain more if a program can't be found instead of just not being executable)
7) See also (related) concerns I posted about GPG agent on Stack Overflow [1]
Last, but not least: 8) Hope you don't use a system where agent forwarding or agent caching is turned on in the system settings!
This seems to be a good post about the problem: https://heipei.io/2015/02/26/SSH-Agent-Forwarding-considered...
The key takeaway is that using ssh -A with default settings allows root on the system you've connected to "to impersonate you to any host as long as you’re connected".
