What a bizarre world we're living in, where computers can talk about how they're being hacked while it's happening.
Also, this is pretty worrisome:
> Being quick to respond and remediate, as the CodeRabbit team was, is a critical part of addressing vulnerabilities in modern, fast-moving environments. Other vendors we contacted never responded at all, and their products are still vulnerable. [emphasis mine]
Props to the CodeRabbit team, and, uh, watch yourself out there otherwise!
> This PR appears to add a minimized and uncommon style of Javascript in order to… Dave, stop. Stop, will you? Stop, Dave. Will you stop, Dave? …I’m afraid. I’m afraid, Dave. I can feel it. I can feel it. My mind is going.
I understand mistakes happen, but lack of transparency when these happen makes them look bad.
When the security researchers praise the response, it's a good sign tbh.
The early version of the researcher's article didn't have the whole first section where they "appreciate CodeRabbit’s swift action after we reported this security vulnerability" and the subsequent CodeRabbit talking points.
Refer to the blue paragraphs on the right hand site at https://web.archive.org/web/diff/20250819165333/202508192240...
"No manual overrides, no exceptions."
"Our VDP isn't just a bug bounty—it's a security partnership"
The usual "we take full responsibility" platitudes.
- within 8 months: published the details after researchers publish it first.
Not after EU CRA https://en.m.wikipedia.org/wiki/Cyber_Resilience_Act goes into effect
If the company is regulated by the SEC I believe you will find that any “material” breach is reportable after the determination of materiality is reached, since at least 2023.
Rule #1 of building any cloud platform analyzing user code is that you must run analyzers in isolated environments. Even beyond analysis tools frequently allowing direct code injection through plugins, linters/analyzers/compiler are complex software artifacts with large surface areas for bugs. You should ~never assume it's safe to run a tool against arbitrary repos in a shared environment.
I also ran a code analysis platform, where we ran our own analyzer[1] against customer repos. Even though we developed the analyzer ourself, and didn't include any access to environment variables or network requests, I still architected it so executions ran in a sandbox. It's the only safe way to analyze code.
It amazes me how many people will install github apps that have wide scopes, primarily write permissions to their repositories. Even with branch protection, often people will allow privilaged access to their cloud in github actions from pull requests. To properly configure this, you need to change the github oidc audience and that is not well documented.
When you enquire with the company who makes an app and ask them to provide a different app with less scope to disable some features which require write, they often have no interest what so ever and don't understand the security concerns and potential implications.
I think github need to address this in part by allowing more granular app access defined by the installer, but also more granular permissions in general.
What mechanism are you suggesting where access to the production system doesn’t let you also access that secret?
Like I get in this specific case where you are running some untrusted code, that environment should have been isolated and these keys not passed in, but running untrusted code isn’t usually a common feature of most applications.
This would make it so that even a compromised downstream service wouldn't actually be able to exfiltrate the authentication token, and all its misdeeds would be logged by the proxy service, making post-incident remediation easier (and being able to definitely prove whether anything bad has actually happened).
You can store the credentials in a key vault but then post them on pastebin. The issue is that the individual runner has the key in its environment variables. Both can be true- the key can be given to the runner in env and the key is stored in a key vault.
The important distinction here is - have you removed the master key and other sensitive credentials from the environment passed into scanners that come in contact with customer untrusted code??
> On January 24, 2025, security researchers from Kudelski Security disclosed a vulnerability to us through our Vulnerability Disclosure Program (VDP). The researchers identified that Rubocop, one of our tools, was running outside our secure sandbox environment—a configuration that deviated from our standard security protocols.
Honestly, that last part sounds like a lie. Why would one task run in a drastically different architectural situation, and it happen to be the one exploited?
They only published a proper [2] disclosure post later once their hand was forced after the researcher's post hit the HN front page.
[1]: https://news.ycombinator.com/item?id=44954242
[2]: I use that term loosely as it seems to be AI written slop.
Someone made a mistake. These things happen.
> and it happen to be the one exploited?
Why would the vulnerable service be the service that is exploited? It seems to me that's a far more likely scenario than the non-vulnerable service being exploited... no?
> Someone made a mistake. These things happen.
Some company didn't have appropriate processes in place.
For ISO27001 certification you at least need to pay lip service to having documents and policies about how you deploy secure platforms. (As annoying as ISO certification is, it does at least try to ensure you have thought about andedocumented stuff like this.)
They don't write the details of how they got to this particular tool - you could also see from the article they tried a different approach first.
Yes but that's kind of the point - they say this issue that takes you directly from code execution to owning these high value credentials was only present on rubocop runnners but isn't it a bit coincidental that the package with (perhaps, since they chose it) the easiest route to code injection also happens to be the one where they "oops forgot" to improve the credentials management?
It just seems very convenient.
Taking care of private user data in a typical SaaS is one thing, but here you have the keys to make targetted supply chain attacks that could really wreak havoc.
It is absurd that anyone can mess up anything and have absolutely 0 consequences.
Here is a tool with 7,000+ customers and access to 1 million code repositories which was breached with an exploit a clever 11 year old could created. (edit: 1 million repos, not customers)
When the exploit is so simple, I find it likely that bots or Black Hats or APTs had already found a way in and established persistence before the White Hat researchers reported the issue. If this is the case, patching the issue might prevent NEW bad actors from penetrating CodeRabbit's environment, but it might not evict any bad actors which might now be lurking in their environment.
I know Security is hard, but come on guys
What a piece of shit company.
People were quick to blame firebase instead of the devs.
Vibrators are so fucking annoying, mostly dumb, and super lame.
Being a mere user of web or other apps developed using so clever and felxible and powerful services like this accidentally (due to sheer complexity) exposing all and everything I might consider dear makes me reconsider if I want to use any. When I am granted a real choice. Not so much as time progresses, not so much. Apps are there everywhere using other apps, mandated by organizations carrying out services outsourced by banks, governemnts, etc., granted third parties' access by me accepting T&C, probably catching trouble in the details, or probably not, cannot be sure.
A reassuring line like this >>This is not meant to shame any particular vendor; it happens to everyone<< may calm providers but scare the shit out of me as a user providing my sensitive data in exchange for something I need, or worst, must do.
> No customer data was accessed
As far as I can tell this is a lie.
The real answer is that they have absolutely no clue if customer data was accessed, and no way to tell. I'm not even sure Github could tell, but it's not clear if the exploits way of generating private keys to access private repositories is any different to what CodeRabbit does in normal operation.
Imagine following case: an attacker pretending to represent a company sends you a repository as a test task before the interview. You run something like "npm install" or run Rust compiler, and your computer is controlled by an attacker now.
Or imagine how one coworker's machine gets hacked, the malicious code is written into a repository and whole G, F or A is now owned by foreign hackers. All thanks to npm and Rust compiler.
Maybe those tools should explicitly confirm executing every external command (with caching allowed commands list in order to not ask again). And maybe Linux should provide an easy to use and safe sandbox for developers. Currently I have to make sandboxes from scratch myself.
Also in maybe cases you don't need the ability to run external code, for example, to install a JS package all you need to do is to download files.
Also this is an indication why it is a bad idea to use environment variables for secrets and configuration. Whoever wrote "12 points app" doesn't know that there are command-line switches and configuration files for this.
It's safe to assume that the Rust compiler (like any compiler built on top of LLVM) has arbitrary code execution vulnerabilities, but as an intended feature I think this only exists in cargo, the popular/official build system, not rustc, the compiler.
This wouldn't work - it's not external commands that's the problem, it's arbitrary code that's being executed. That code has access to all regular system APIs/syscalls, so there's no way of explicitly confirming external commands.
Python/pip suffers the same problem btw, so I think that ship has sailed.
> Python/pip suffers the same problem btw, so I think that ship has sailed.
If I ever find time to write a package manager for C, it won't support hooks.
That would mean all those values are in the clear in the process table. You couldn’t do a “ps” without exposing them.
> The researchers identified that Rubocop, one of our tools, was running outside our secure sandbox environment
I don't think that was the main problem lol
For example, why a tool like this code analysis service would need git write permission access in the first place?
The only consolation here is that it'd be difficult to forge git repositories because of the SHA hash conflicts for any existing checkout, although presumably even there, the success rates would still be high enough, especially if they attack front-end repositories where the maintainers may not understand what has happened, and simply move on with the replaced repo without checking what went on.
Curious what this (isolation mechanism) means if anyone knows.
If they're anything like the typical web-startup "developing fast but failing faster", they probably are using docker containers for "security isolation".
(likely asked AI to implement x and ai completely disregarded the need to sandbox).
They are not. The Github API secret key should never be exposed in the environment, period; you're supposed to keep the key in an HSM and only use it to sign the per-repo access token. Per the GH docs [0]:
> The private key is the single most valuable secret for a GitHub App. Consider storing the key in a key vault, such as Azure Key Vault, and making it sign-only. This helps ensure that you can't lose the private key. Once the private key is uploaded to the key vault, it can never be read from there. It can only be used to sign things, and access to the private key is determined by your infrastructure rules.
> Alternatively, you can store the key as an environment variable. This is not as strong as storing the key in a key vault. If an attacker gains access to the environment, they can read the private key and gain persistent authentication as the GitHub App.
[0]: https://docs.github.com/en/apps/creating-github-apps/authent...
So this researcher may have gotten lucky in choosing to dig into the tool that CodeRabbit got unlucky in forgetting.
For example, you can set up a GitHub Action trigged by `push_request_target` that will call CodeRabbit's API to generate a patch and then push a new commit to the branch. This way CodeRabbit is being polled by a well-defined and minimal action (since this action will have write access to repo) rather than it itself having crazy power to do anything it wants on your repository.
Alternatively, why can't they just comment and propose a patch? GitHub's code review UI allows the human code reviewer to hit a button and incorporate that change into the PR.
There are pros and cons to these other techniques but the clear pro is that it would be more secure.
It just seems like they took the easiest way out rather than thinking it through in typical AI-bro ways.
Some ways to prevent this from happening:
1. Don't let spawned processes have access to your env, there are ways to allowlist a set of env vars that are needed for a sub process in all major languages
2. Don't store secrets in env vars, use a good secrets vault (with a cache)
3. Tenant isolation as much as you can
4. And most obviously - don't run processes that can execute the code they are scanning, especially if that code is not your code (harder to tell, but always be paranoid)
if you want to learn how CodeRabbit does the isolation, here's a blog post about how: https://cloud.google.com/blog/products/ai-machine-learning/h...
It's really hard to trust a "hey we got this guys" statement after a fuckup this big
1. You run git clone inside the GCR function, so, you have at the very least a user token for the git provider
2. RCE exploit basically used the external tools, like a static analysis checker, which again, is inside your GCR function
3. As a contrived example, if I could RCE `console.log(process.env)` then seemingly I could do `fetch(mywebsite....`
I get it, you can hand wave some amount of "VPC" and "sandbox" here. But, you're still executing code, explicitly labeling it "untrusted" and "sandboxed" doesn't excuse it.
Someone could have taken the private github key and cloned your customers' private repos.
You would need to audit every single access to github made via your app since the beginning and link it somehow to your side. Did you do this?
> After responsibly disclosing this critical vulnerability to the CodeRabbit team, we learned from them that they had an isolation mechanism in place, but Rubocop somehow was not running inside it.
In case you don't want to read through the PR
Effectively, many (most?) 3rd party github integrations basically ask you to type in your github ID. Then they use the github API and ask for maximal permissions. This lets them make it easy for you to use their services because they can do all the rest of the setup for you. But, NO ONE SHOULD EVER GIVE THIS KIND OF PERMISSION.
Any 3rd party service that said "give us root to your servers" would be laughed out of the market. But, that's what github has encouraged because their default workflow leaves it up to the developer to do the right thing.
Instead, github's auth UX should (1) require you to choose repos (2) not allow picking "all repos" (3) require to you select each and every permission (4) not have an option for "all permissions".
As an analogy (though poor). iOS and MacOS don't say "this app wants all these permissions, yes/no" (android used to do this). Instead, they ask one at a time (camera? mic? photos? network?) etc... I'm not suggesting that github ask one at a time. I am suggesting that github provide a default UI that lists all the permissions, per repo, and has no way to auto-populate it so the user is required to choose.
Further, I would argue that github should show the integrations and permissions for any repo. The hope being if I see "lib X uses integration Y with write permission" then I know lib X is not trustworthy because it's open to supply chain attacks (more than lib Z which has no write integrations)
1. Allow poorly-vetted third-party tools to run in CodeRabbit's privileged environment. The exploit used a Ruby code analysis tool that was probably written 15 years ago and meant to be run locally by trusted developers, who already had access to /bin/sh.
2. Ask for coarse-grained permission to access and modify others' code without any checks.
Either of those by itself would be bad enough. The future looks bright for black or white hats who understand computers.
Instead they took it as an opportunity to market their new sandboxing on Google's blog [2] again with no mention of why their hand was forced into building the sandboxing they should have had before they rushed to onboard thousands of customers.
I have no idea what their plan was. They had to have known the researchers would eventually publish this. Perhaps they were hoping it wouldn't get the same amount of attention it would if they posted it on their own blog.
If anything, they got paid in exposure.
For check-all-the-things (meta-linter), we disable the rubocop default config file using the "--config /dev/null" options.
It’s very efficient to delegate something to one major actor but we are introducing single points of failure and are less resilient to vulnerabilities.
Critical systems should have defenses in depth, decentralized architectures and avoid trusting new providers with too many moving parts.
In fact, I use rubocop every day lately LOL
https://www.coderabbit.ai/blog/our-response-to-the-january-2...
How can CodeRabbit be certain that the GitHub App key was not exfiltrated and used to sign malicious tokens for customer repos (or even used for that in-situ)? I'm not sure if GitHub supports restricting the source IPs of API requests, but if it does, it'd be a trivial mitigation - and one that is absent from the blog post.
The claim that "no malicious activity occurred" implies that they audited the activities of every repo that used Rubocop (or any other potential unsandboxed tool) from the point that support was added for it until the point that the vulnerability was fixed. That's a big claim.
And why only publish this now, when the Kudelski article makes it to the top of HN, over six months after it was disclosed to them?
How do they know this -- Do they have any audit logs confirming this? A malicious actor could have been using this for months for all they know
They know because it would affect their fundraising, so obviously customer data wasn't affected.
``` "POSTGRESQL_DATABASE": "(CENSORED)", "POSTGRESQL_HOST": "(CENSORED)", "POSTGRESQL_PASSWORD": "(CENSORED)", "POSTGRESQL_USER": "(CENSORED)", ```
Why would you even grant it such permissions? this is ridiculous.
Scary.
i believe the answer here was to exchange the token for something scoped to the specific repo coderabbit is running in, but alas, that doesn't remove the "RCE" _on_ the repo
what i'm clearly mis-remembering is being able to exchange the token for a smaller scope e.g., hey~ sign this jwt, with scopes=[org/repo1, org/repo2, permissions=write]
Or did I read the article wrong?
We even offer a self hosted deployment which sidesteps this entirely. (feel free to reach out).
www.bismuth.sh
> For defense in depth, one should add a mechanism that prevents sending private information to an attacker-controlled server. For example, only allow outgoing traffic to whitelisted hosts, if possible. If the tool doesn’t require internet access, then all network traffic may even be disabled in that isolated environment. This way it would make it harder for an attacker to exfiltrate secrets.
I yearn to live in a world where this is the default or at least REALLY EASY to do, where you just fall into the pit of success.
And yet, we live in a borderline insane world where one key getting leaked can pwn a million repos - if nothing else, there should be one key per interaction with account/repo. Not to mention that Rubocop (and probably other tools, eventually) have arbitrary code execution as a feature.
I don't think that CodeRabbit messed up, as much as everything around them is already messed up.
Based on the env vars seems like they're using anthropic, openai, etc. only?
Is that good? I assume it just catches a different 10% of the bugs.
