"As part of our efforts to enhance our security and in response to an incident published on status.heroku.com, we wanted to inform you that we will begin resetting user account passwords on May 4, 2022. We recommend that you reset your user account password in advance here and follow the best practices below:
Minimum of 16 characters Minimum complexity of 3 out of 4: Uppercase, Lowercase, Numeric, Symbol Don't just add a letter or a 1 digit number to the existing password while changing Passwords may not be duplicated across accounts If you do not reset your password and your user account password is reset by Heroku on May 4, 2022, your existing password will no longer work. To log in to Heroku, you must reset your password by accessing the Heroku login page and clicking the "Forgot your password?" link . Please be aware that you may be required to reset your passwords again in the future. "
For those of you that haven't been following, Heroku has been adding non-update updates to this security thread over the last couple of weeks, which began with the announcement that some (or maybe all) of their GitHub granted access tokens had been compromised: https://status.heroku.com/incidents/2413
Now, weeks later, we're hearing that all account passwords are being reset, and for some reason if you have been using an HTTPS-style log drain that you should reset any secrets related to it as well.
Heroku needs to come out and clearly state what they know about this situation, and more importantly what they don't know -- which is starting to sound like the answer is "a lot". It's not even clear they know how this all happened -- whatever door was left open might still be open. So if you've gone and rotated all of your application secrets (which you probably should do), be prepared to rotate them again when this is all over.
I know it's small, but some will skip the email because they don't use Salesforce software directly and wouldn't anticipate emails from a parent company.
[1] https://devcenter.heroku.com/articles/github-integration
> A statement that confirms whether or not config variables and secrets were accessed, or that you're not sure, needs to be sent out.
To which they replied:
> We currently have no evidence that Heroku customers’ secrets stored in config Var were accessed. If we find any evidence of unauthorized access to customer secrets, we will notify affected customers without undue delay.
Take that as you will, but it doesn't fill me with confidence.
> Law of No Evidence: Any claim that there is “no evidence” of something is evidence of bullshit.
Resetting passwords implies something else may have been compromised (passwords, either hopefully encrypted), but is a pretty scary ask for them to make without providing more context here.
Trainwreck indeed.
I'm still not entirely sure if I've reset/rotated everything I need to, what is "any credentials used with"? Neither the email nor any docs it linked to was clear about exactly what they are suggesting be rotated.
That message wasn't very specific, while also they're not providing the context about the breach that one could use to fill in the gaps.
At this point it kinds of sounds like... everything there is was compromised?
This is the most concerning part of that email, as it implies more than an "out of an abundance of caution", but rather that they suspect their password DB has been compromised.
Thinking about it, it does sound the most likely as they were probably the same DB the customer oAuth tokens were stored in that were used to access Github repositories. But if they already knew the data was stored together why wait till now to reset passwords?
This means that assuming the DB is using proper salt+hash, the main differentiator is the strength of your password. If it's a relatively short one that can be brute-forced/found via dictionary+small mutation, then attackers could possibly log in as you. If it's a strong password from a password manager, then that will likely have kept them from being able to crack your password.
Of course all this only has value if we assume that only the password db was breached. If they managed to access the place your env-var/secrets are stored, then all bets are off.
It's what happens when the product visionaries get bored and leave. Such a shame.
A wow, and thanks for the praise but the credit goes to way more than me. The team around in the early days was unbelievable, I learned a ton on building products and developer experience from James/Adam (founders) in particular, though Heroku wouldn't have gotten there without Orion (the other founder) as well. Byron, PvH, Mark, Noah, Morten, Oren were absolutely huge in so many ways to the leadership and direction of Heroku. And I'm sure I'm going to get messages from 50 others there in the early days that I didn't name drop them, the collective team was an awesome team and pushed each other really well.
At around 2015 it did feel like there was attrition and the technical leadership and vision started to fade. It wasn't me, it was a lot of us moved on to the next thing. At the time it wasn't Salesforce taking control or one person, we'd all put a lot into it and various folks moved on. Adam/Orion/James gave an incredibly amount and were understandably ready to recharge. Still very proud of what we created at that time, what it did for developer experience, and personally (along with that original Heroku Postgres team) trying to do what I describe as unfinished business for creating the amazing developer experience of Postgres.
This concerns me. How are they checking that no other account has the same password? Wouldn't that imply a strong possibility they're either hashing with the same salt across all accounts or not hashing at all?
Hopefully some of the smarter folks here can tell me why I'm way off base...
More seriously, it ought to be possible to create a data structure that, with some degree of security, will efficiently state either that a password is probably in a set or that it’s probably not. A bloom filter can do this, for example. I would not do anything of this sort in production, since, even if it has the best possible level of cryptographic security, any such capability could be abused to precompute a dictionary containing at least a massively higher than average density of Heroku passwords.
IOW, the mere fact that Heroku can tell if a password is or was in use with decent accuracy is, in and of itself, a dangerous oracle.
Related, I often see people trip on the sense of MAY in RFCs (https://www.ietf.org/rfc/rfc2119.txt), because in everyday use it has become "passive-agressive must".
edit: I'm ESL so might be reading way to much into this.
Also while "MAY" in RFCs per that document mean something, press releases aren't RFCs and there may be significant amounts of money riding on the exact interpretation of what those words mean.
And "reading way too much into" is the bread and butter of lawyers, technologists, and mathematicians. Don't worry about it!
I'm at a startup that's been on Heroku since its inception and this breach has got my team thinking about moving to AWS for the first time, even if our scale doesn't necessarily demand it yet.
Does anyone that has done this transition in the past have any advice?
It all depends on your company resources. In my cases the companies were no longer startups and could afford to staff Platform/Infra teams and build out a bespoke solution on AWS. This was an area the company wanted to invest in.
If you're still a legit startup where money/dev resources is tight, I would recommend first looking into one of the Heroku clones/competitors.
Given the availability of tools like Coherence (www.withcoherence.com) that deliver a great developer experience in your own cloud, it makes sense to evaluate them alongside modern PaaS if you're migrating off Heroku today.
(disclosure, I'm a cofounder of Coherence).
I received this email. The reset password link in it is NOT https. If I manually change the http to https it turns out that the server, click.msg.salesforce.com, is returning a certificate that is only valid for click.s10.exacttarget.com
So I think the lack of 's' in http in the original link doesn't matter.
If I'm wrong, could you please explain why (I am reasoning as best I can, and am not an expert, and keen to learn).
However, Heroku should also be smart enough to figure out that all links should be https and servers should have valid certificates.
For a company to make such basic security-related mistakes while in the middle of a bad security incident doesn't look good, to put it politely.
It sort of explains how they got where they are.
Obviously, you've checked the URL and seen that it's legit after, but realistically you should expect a legit email to not be feeding you a potentially insecure link.
Salesforce isn't a small new startup where this can be understood as a cost of doing business, they are a tier-1 provider to the largest companies in the world. Unacceptable (but still sending #Hugops to the team dealing with this mess!)
^ Note: I use to work for Aptible
Porter
The initial issue was supposedly a breach of customer repositories. Which sounds bad, but if you’re not storing credentials in code, then the worst case is that a potential hacker had access to download your code. Not great, but certainly not as catastrophic as some breaches have been.
Since then, Heroku has been acting beyond strange. Everyday they update the incident with essentially the same non-update, but written differently, with vague references to the same information they’ve sent 14 days in a row.
Now, they send another very ominous and strange update about “some” customers having their passwords reset. However, based on this thread and my own experience, it seems like every customer is getting this message.
What does this have to do with the initial issue? Were actual Heroku accounts compromised?
This behavior is either extreme incompetence wrt customer communication, or they’re preparing to announce a truly insane breach that may include everyone that has ever used Heroku.
They need to get their shit together and quickly.
There’s a simple reason for this: their incident response policy requires they not leave customers un-updated more than x hours/days. Their tooling likely reminds someone if it isn’t done.
I don't have the password reset email, and logging in just asks me to add MFA, and then to accept 2020 terms.
And heroku doesn't seem to acknowledge that at all.
I use a private matrix server to send passwords from my laptop/desktop password managers to myself in E2EE channels, then delete the message. On occasion, if a password is only 18 or 20 characters (like an admin account or a single user service account) i'll type it by hand. And i use all lowercase "easy to say" passwords whenever possible. I do not believe (and never have) that adding an ampersand and a 7 to a password makes it more secure than adding "rb" to the end of a password.
But at password generation time, it's much easier. You don't need to tell the user all these rules about length, complexity, reuse, etc. The server generates the password in a way that's sufficient, and the user just saves it. It also avoids the mess of password manager generated passwords not meeting some criteria, or passwords meeting client side checks but failing server side checks.
When you've got so many sites asking for passwords >=10, >=16 characters in length, is anyone actually able to memorize a truely unique password per-site any more?
Besides, a server generated password has known entropy, so you can model the security much more accurately. This push for really long passwords is to guard against low entropy passwords that users create, although users seem to respond predictably: [password] too short? I'll try [password]123 or [password][password].
I'd recommend storing server generated passwords in the same way a user generated password should be stored on the backend: bcrypt or scrypt.
Do they not have any fraud screening or rate limiting on these things, 2FA on unknown devices or something?
I've had my same password forever (I'm old) on google without issue) that is scary short. However, when I login from a new device or every 30 days or go to an admin / security page it asks me to stick my yubico key into my computer. Works fine.
Is this same password used for high volume API stuff without anything else? Normally that's with an access token type system (at least on AWS). Those, yes, should be highly complex. The AWS access / secret key combo is ridiculous frankly in length (mixed case + numbers + symbols) * 40.
"Scary short" and "same password" are terrifying to read. Get a password manager (I recommend 1Password) and use it. My dad is 78 and uses a password manager -- so can you.
Regardless, stop reusing your password please! This is the most basic security measure possible.
Coupled with a Yubico and I feel pretty darn secure.
I don't re-use the password, because it doesn't rotate I can also remember it.
Assume they've been pwned end-to-end based on this alert.
I've went ahead and just deleted the whole account - it's at the point where I don't want to do business with this company, full stop.
I sent heroku an email about a week or bit more ago - complaining that I had gotten a very well crafted phishing email - which was seriously very clever in it's timing (or just blind lucky for them, as I was changing servers and dns and things within 48 hours of this) - that made it look I needed to login and do something with server credentials or something similar.. and the link to click to fill info went to a heroku app with a name attached to it..
So I emailed them and asked them to look into it to see if it was overly obvious that user XY was getting clicks and if they had set up some sort of official looking form to phish server accounts or something like that - memory a bit foggy in the midst of so many changes past couple weeks.
Never heard back from them, but maybe my email helped them realize some things were not being used as they should of been authorized to be used for. Maybe not.
This whole things has been an absolute s*tshow. Now I'm worried that there is a much bigger issue at play, and we might actually be in a bit of trouble.
As a general thing, I like the idea of a "swift, well written" email to your entire userbase saying "please reset your password, now."
That seems like a perfectly reasonably response, and request of your user-base.
(2) just saw that the password reset will invalidate all API tokens. Arrrrggh. Annoying!
Because passwords are shared secrets, the relying party (here Heroku) can lose them and then will predictably act like somehow this is your problem when it happens, see also "Identity theft" aka "We're incompetent and our money got stolen but it's convenient to pretend that's actually your money and ask when you'll fix it".
Use WebAuthn. If (when) outfits like Heroku incompetently lose the credentials nothing interesting happens.
> On April 7, 2022, a threat actor obtained access to a Heroku database and downloaded stored customer GitHub integration OAuth tokens. Access to the environment was gained by leveraging a compromised token for a Heroku machine account. According to GitHub, the threat actor began enumerating metadata about customer repositories with the downloaded OAuth tokens on April 8, 2022. On April 9, 2022, the attacker downloaded a subset of the Heroku private GitHub repositories from GitHub, containing some Heroku source code.
...
> Separately, our investigation also revealed that the same compromised token was leveraged to gain access to a database and exfiltrate the hashed and salted passwords for customers’ user accounts. For this reason, Salesforce is ensuring all Heroku user passwords are reset and potentially affected credentials are refreshed. We have rotated internal Heroku credentials and put additional detections in place. We are continuing to investigate the source of the token compromise.
Hobbyists and Developers who are looking such a platform should expect they are least technical to solving harder software problems then they are, and that’s worth paying the premium for.
May the 4th be with you.
NOTE: A password reset will also invalidate your API access tokens. As a result, any automations you’ve built to integrate with the Heroku Platform API that use these tokens may result in 403 forbidden errors . To avoid downtime you will need to re-enable direct authorizations by following the instructions here and update your integrations to use your newly generated token.
Was this a system wide event?
We have been on Heroku for ~5 years and the last 18 months have been far less reliable. Sad.
hmm, how do they know if those passwords are same across accounts?
Either they save them in plaintext or they can't check if I have same password as some other account.
