Looks like the database that stores pipeline-level config variables for both Review Apps and Heroku CI were compromised.
Per Heroku, "...any secrets you set in Review Apps and Heroku CI config may have been compromised and should be rotated".
This...is really messed up :/
At Salesforce, we understand that the confidentiality, integrity, and availability of your data is vital to your business, and we take the protection of your data very seriously. We value transparency and wanted to notify you of an issue affecting your account. Based on current progress, we plan to complete our investigation by May 30, 2022. We are continuing with remediation activities and plan to publish additional information about the incident once it’s resolved.
As reported on status.heroku.com, on April 7, 2022, a threat actor obtained access to a Heroku database and downloaded stored customer GitHub integration OAuth tokens. On that same day, the threat actor downloaded data from another database that stores pipeline-level config vars for Review Apps and Heroku CI. This was identified on May 16, 2022, after further forensic investigation. We have no evidence of any unauthorized access to Heroku systems since April 14, 2022.
As a result, any secrets you set in Review Apps and Heroku CI config vars may have been compromised and should be rotated. In addition, any Heroku tokens stored in these pipeline config vars would potentially have allowed access to your Heroku account between April 7, 2022 and May 5, 2022, when your passwords were reset, invalidating all Heroku tokens as a result.
Please note, these pipeline-level config vars are different from standard app config vars. App config vars were not stored in this database and we have no evidence to suggest app config vars were compromised.
Hey Bob, why didn't you tell your customers a month ago to rotate their creds just to be safe? This is flat out insulting.
Their legal pages[1] are filled to the brim with those ridiculous statements. I never understood why they'd even bother making it sound nice, especially not for B2B.
Customers won't trust the message and likely can't use them in court, and they themselves must surely know they're creating expectations that they can't guarantee to meet.
But it's probably to Render's credit that, in my opinion, the most annoying thing about Render is that it's impossible to google about Render because "render" is such a common word in the tech world!
Their support is good and responsive, and the developer experience was good enough. It has some warts, and there were definitely times I missed Heroku, but their speed of improvement gives me confidence in their future.
Sad to leave Heroku after almost a decade with them. They were far ahead of their time.
Do you have any evidence Render actually takes security seriously?
Not shitting on their platform, I actually never used it, I just think as an industry we should be way past the point we trust platforms by default.
That's a great point and I fully agree.
I'm struggling to come up with reliable ways of checking security of the companies I'm not familiar with. It's not like I can rely on their landing page. And they are likely not on the market long enough to see how they responded to past security incidents.
The only thing I can think of is checking how they handle registration and logins - but it's not that strong of a signal anyway. Does anyone have other ideas?
I was debating between render & fly, which I've also had my eye on and may still try for something else in the future.
It pains me to see even occasional defenders of Heroku. They're not the company they were 10 years ago. They've been gutted and left for dead years ago but the product was so good nobody noticed until now.
They're not to be trusted as your platform. They simply don't have anywhere close to the manpower required to run such a platform. This was a when not if situation.
If you're still on it, make your plans to move away now. Time is ticking until a major outage or another security incident like this one. See my comment history and related threads for more. Specifically this summary: https://news.ycombinator.com/item?id=31374048
It still sucks that they are parceling out the information, but the claim that they outright lied is not true.
> We also wanted to address a question regarding impact to environment variables. While we confirmed that the threat actor had access to encrypted Heroku customer secrets stored in config var, the secrets are encrypted at rest and the threat actor did not access the encryption key necessary to decrypt config var secrets.
https://status.heroku.com/incidents/2413
Nowhere in that did it clarify it was speaking of app but not pipeline env vars. They had plenty of time to author that post too. Make sure you rotate those app env vars anyways as this somehow appears to be getting worse by the week.
Plenty of folks I respect absolutely love fly.io--I have less hands-on experience there, but they've got a fantastic crew, too.
I don't have experience with any other PaaS's so I can't recommend one, but what you say is what I commonly hear.
I guess that was a lie?!
> Additionally, we have no evidence that the attacker has accessed any customer accounts or decrypted customers’ environment variables.
which, as pointed out in its HN thread, means "we now know they got access to encrypted vars, and we don't know yet if they could have decrypted them." in BS-speak.
The title "We've Heard Your Feedback" is also a red herring, usually means "we know we fucked up bad and we still have no idea of the whole impact of the breach".
Why? Commercialism.
Founders sell to the highest bidder to make their exit worthwhile for themselves, not caring about the future of the product (and customers).
It's a no-brainer that a commercial company like Salesforce (it's in their name!) doesn't have what it takes to build AAA software, but focuses on maximizing their profit. They drove away their best staff, focused on the wrong features, and are seemingly overwhelmed by maintaining their purchased software, all while probably not even realizing their demise.
We should all come to the agreement that takeovers of fundamental software by incompetent companies should be seen as a hostility towards every current user of said software.
That feels like a angsty-tinted view. I recall the day it happened. The Ruby dev shop I was at was optimistically nervous. As Heroku had been a shiny new thing and only deployed Ruby. Acquisition allowed them to expand and support other languages. They didn't even have pipelines!
https://techcrunch.com/2010/12/08/breaking-salesforce-buys-h...
"Trust is our Number 1 value."
1. company all-hands meetings, which are basically pep rallies with no actual content
2. when someone working at Salesforce brings up a glaring problem and says "if Trust is our number 1 value, why don't we do something about this huge problem?", which is usually met with either silence and bureaucratic obstacles or with excuses, usually something like "customers trust us to spend the money they pay us building the features and products they want", which is like...exactly not the definition used at any of the pep rallies.
Is the impact limited to specific customer accounts, or are they just not updating me anymore?
> We value transparency and wanted to notify you of an issue affecting your account.
My guess is they sent it to users with pipelines that have env vars. It's funny since this sentence demonstrates they don't value transparency by not telling the other users more information about the hack.
They updated Heroku Status but surprisingly failed to mention anything about CI or pipelines.
are there any mystery hacks occurring yet?
is this database known to have been spread anywhere?
