I was just setting up a new project, and things behaved weirdly. My laptop ran out of RAM, it looked like a forkbomb was running.
I've investigated, and found that a base64 encoded blob has been added to proxy_server.py.
It writes and decodes another file which it then runs.
I'm in the process of reporting this upstream, but wanted to give everyone here a headsup.
It is also reported in this issue: https://github.com/BerriAI/litellm/issues/24512
1. Looks like this originated from the trivvy used in our ci/cd - https://github.com/search?q=repo%3ABerriAI%2Flitellm%20trivy... https://ramimac.me/trivy-teampcp/#phase-09
2. If you're on the proxy docker, you were not impacted. We pin our versions in the requirements.txt
3. The package is in quarantine on pypi - this blocks all downloads.
We are investigating the issue, and seeing how we can harden things. I'm sorry for this.
- Krrish
- Impacted versions (v1.82.7, v1.82.8) have been deleted from PyPI - All maintainer accounts have been changed - All keys for github, docker, circle ci, pip have been deleted
We are still scanning our project to see if there's any more gaps.
If you're a security expert and want to help, email me - krrish@berri.ai
It is so much better than, you know... "We regret any inconvenience and remain committed to recognising the importance of maintaining trust with our valued community and following the duration of the ongoing transient issue we will continue to drive alignment on a comprehensive remediation framework going forward."
Kudos to you. Stressful times, but I hope it helps to know that people are reading this appreciating the response.
Write a detailed postmortem, share it publicly, continue taking responsibility, and you will come out of this having earned an immense amount respect.
Were you not aware of this in the short time frame that it happened in? How come credentials were not rotated to mitigate the trivy compromise?
It's very much not production grade. It might miss sneaky ways to install litellm, but it does a decent job of scanning all my conda, .venv, uv and system enviornments without invoking a python interpreter or touching anything scary. Let me know if it misses something that matters.
Obviously read it before running it etc.
the developer has made a new github account and linked their new github account to hackernews and linked their hackernews about me to their github account to verify the github account being legitimate after my suggestion
Worth following this thread as they mention that: "I will be updating this thread, as we have more to share." https://github.com/BerriAI/litellm/issues/24518
also how are we sure that docker images aren't affected?
Was your account completely compromised? (Judging from the commit made by TeamPCP on your accounts)
Are you in contacts with all the projects which use litellm downstream and if they are safe or not (I am assuming not)
I am unable to understand how it compromised your account itself from the exploit at trivvy being used in CI/CD as well.
Otherwise people will naysay and detract from the cause. "It worked before" they will say. "Why don't we do it like before?"
DISA STIG already forbids use of the EPEL for Red Hat Enterprise Linux. Enterprise software install instructions are littered with commands to turn off gpgcheck and install rpm's from sourceforge. The times are changing and we need cryptographically verifiable guarantees of safety!
https://github.com/calebfaruki/tightbeam https://github.com/calebfaruki/airlock
This is literally the thing I'm trying to protect against.
The problems you mentioned resonated a lot with me and why I'm building it, any interest in working to solve that together?: https://github.com/smol-machines/smolvm
I haven't kept up with the recent exploits, so a side question: Have any of the recent supply chain attacks or related exploits included any escapes from basic dev containers?
The harder problem is that CI pipelines routinely grant scanner processes more credential access than they need. Trivy needed read access to the repo and container layers; it didn't need PyPI publish tokens. Scoping CI secrets to the minimum necessary operation, and injecting them only for the specific job that needs them rather than the entire pipeline, would have contained the blast radius here.
This is the interesting part. What kind of UI or other mechanisms would help here? There's no silver bullet for detecting and crashing on "something bad". The adversary can test against your sandbox as well.
Happily sandboxing almost all third-party tools since 2025. `npm run dev` does not need access to my full disk.
Then we talk about containment like anyone actually looked at that dep list.
> We just can't trust dependencies and dev setups.
In one of my vibe coded personal projects (Python and Rust project) I'm actually getting rid of most dependencies and vibe coding replacements that do just what I need. I think that we'll see far fewer dependencies in future projects.Also, I typically only update dependencies when either an exploit is known in the current version or I need a feature present in a later version - and even then not to the absolute latest version if possible. I do this for all my projects under the many eyes principal. Finding exploits takes time, new updates are riskier than slightly-stale versions.
Though, if I'm filing a bug with a project, I do test and file against the latest version.
I see people going in the opposite direction with "dump everything in front of my army of LLMs" setups. Horribly insecure, but gotta go fast, right?
On a personal note, I have been developing and talking to a clanker ( runs inside ) to get my day to day work done. I can have multiple instances of my project using worktrees, have them share some common dependencies and monitor all of them in one place. I plan to opensource this framework soon.
But mention that on HN and watch getting downvoted into oblivion: the war against general computation, walled gardens, locked down against device owners...
I made this tool for macos systems that helps detect when a package accesses something it shouldn't. it's a tiny go binary (less than 2k LOC) with no dependencies that will mount a webdav filesystem (no root) or NFS (root required) with fake secrets and send you a notification when anything accesses it. Very stupid simple. I've always really liked the canary/honeypot approach and this at least may give some folks a chance to detect (similar to like LittleSnitch) when something strange is going on!
Next time the attack may not have an obvious performance issue!
I always wanted to mess with building virtual filesystems but was unwilling to venture outside the standard library (i.e. libfuse) for reasons wonderfully illustrated in this thread and elsewhere. Somehow the idea of implementing a networked fs protocol and leaving system integration to the system never crossed my mind.
I'm glad more people are taking this stance. Large centralized standard libraries and minimal audited dependencies is really the only way to achieve some semblance of security. There is simply no other viable approach.
Edit: What's the license for this project?
I updated my global configs to set min release age to 7 days:
~/.config/uv/uv.toml
exclude-newer = "7 days"
~/.npmrc
min-release-age=7 # days
~/Library/Preferences/pnpm/rc
minimum-release-age=10080 # minutes
~/.bunfig.toml
[install]
minimumReleaseAge = 604800 # secondsNext up, we're going to advise a minimum-release-age of 14 days, cause most other projects use 7 days.
I would expect better spam detection system from GitHub. This is hardly acceptable.
I scrolled through and clicked a few profiles. While many might be spam accounts or low-activity accounts, some appeared to be actual GitHub users with a history of contributions.
I’m curious how so many accounts got compromised. Are those past hacks, or is this credential steeling hack very widespread?
Are the trivy and litellm hacks just 2 high profile repos out of a much more widespread “infect as many devs as possible, someone might control a valuable GitHub repository” hack? I’m concerned that this is only the start of many supply chain issues.
Edit: Looking through and several of the accounts have a recent commit "Update workflow configuration" where they are placing a credential stealer into a CI workflow. The commits are all back in february.
Even still though, we can't really trust any open-source software any more that has third party dependencies, because the chains can be so complex and long it's impossible to vet everything.
It's just too easy to spam out open-source software now, which also means it's too easy to create thousands of infected repos with sophisticated and clever supply chain attacks planted deeply inside them. Ones that can be surfaced at any time, too. LLMs have compounded this risk 100x.
This is why software written in Rust scares me. Almost all Rust programs have such deep dependency trees that you really can't vet them all. The Rust and Node ecosystems are the worst for this, but Python isn't much better. IMO it's language-specific package managers that end up causing this problem because they make it too easy to bring in dependencies. In languages like C or C++ that traditionally have used system package managers the cost of adding a dependency is high enough that you really avoid dependencies unless they're truly necessary.
However, the broader idea of supply chain attacks remains challenging and AI doesn’t really matter in terms of how you should treat it. For example, the xz-utils back door in the build system to attack OpenSSH on many popular distros that patched it to depend on systemd predates AI and that’s just the attack we know about because it was caught. Maybe AI helps with scale of such attacks but I haven’t heard anyone propose any kind of solution that would actually improve reliability and robustness of everything.
[1] Fully Countering Trusting Trust through Diverse Double-Compiling https://arxiv.org/abs/1004.5534
You're right though. There's been talks of a big global hack attack for a while now.
Nothing is safe anymore. Keep everything private airgapped is the only way forward. But most of our private and personal data is in the cloud, and we have no control over it or the backups that these companies keep.
While LLMs unlock the opportunity to self-host and self-create your infrastructure, it also unleashes the world of pain that is coming our way.
In the end we need fully deterministic, 100% verifiable, chains. From the tiny boostrapped beginning, to the final thing.
There are people working on these things. Both, in a way, "top-down" (bootstrapping a tiny compiler from a few hundred bytes) and "bottom-up" (a distro like Debian having 93% of all its packages being fully reproducible).
While most people are happy saying "there's nothing wrong with piping curl to bash", there are others that do understand what trusting trust is.
As a sidenote although not a kernel backdoor, Jia Tan's XZ backdoor in that rube-goldberg systemd "we modify your SSHD because we're systemd and so now SSHD's attack surface is immensely bigger" was a wake-up call.
And, sadly and scarily, that's only for one we know about.
I think we'll see much more of these cascading supply chains attack. I also think that, in the end, more people are going to realize that there are better ways to both design, build and ship software.
The possibilities within a good threat could be catastrophic if we assume so, and if we assume nation-states to be interested in sponsoring hacking attacks (which many nations already do) to attack enemy nations/gain leverage. We are looking at damage within Trillions at that point.
But I would assume that Linux might be safe for now, it might be the most looked at code and its definitely something safe.
LLVM might be a bit more interesting as it might go a little unnoticed but hopefully people who are working at LLVM are well funded/have enough funding to take a look at everything carefully to not have such a slip up.
Basically it forkbombed `grep -r rpcuser\rpcpassword` processes trying to find cryptowallets or something. I saw that they spawned from harness, and killed it.
Got lucky, no backdoor installed here from what i could make out of the binary
how do you do that? have Activity Monitor up at all times?
I've been through SOC2 certifications in a few jobs and I'm not sure it makes you bullet proof, although maybe there's something I'm missing?
This threat actor seems to be very quickly capitalising on stolen credentials, wouldn’t be surprised if they’re leveraging LLMs to do the bulk of the work.
Since they all seem positive, it doesn't seem like an attack but I thought the general etiquette for github issues was to use the emoji reactions to show support so the comment thread only contains substantive comments.
> It also seems that attacker is trying to stifle the discussion by spamming this with hundreds of comments. I recommend talking on hackernews if that might be the case.
Do the labs label code versions with an associated CVE to label them as compromised (telling the model what NOT to do)? Do they do adversarial RL environments to teach what's good/bad? I'm very curious since it's inevitable some pwned code ends up as training data no matter what.
I assume most labs don't do anything to deal with this, and just hope that it gets trained out because better code should be better rewarded in theory?
> ### Software Supply Chain is a Pain in the A*
> On top of that, the room for vulnerabilities and supply chain attacks has increased dramatically
AI Is not about fancy models, is about plain old Software Engineering. I strongly advised our team of "not-so-senior" devs to not use LiteLLM or LangChain or anything like that and just stick to `requests.post('...')".
[0] https://sb.thoughts.ar/posts/2025/12/03/ai-is-all-about-soft...
Run all your new dependencies through static analysis and don't install the latest versions.
I implemented static analysis for Python that detects close to 90% of such injections.
LiteLLM wouldn't be my top choice, because it installs a lot of extra stuff. https://news.ycombinator.com/item?id=43646438 But it's quite popular.
Configure the CI to make a release with the artefacts attached. Then have an entirely private repo that can't be triggered automatically as the publisher. The publisher repo fetches the artefacts and does the pypi/npm/whatever release.
https://docs.npmjs.com/generating-provenance-statements
https://packaging.python.org/en/latest/specifications/index-...
some will even audit each package in there (kind crap job but it works fairly well as mitigation)
or pyproject.toml (not possible to filter based on absence of a uv.lock, but at a glance it's missing from many of these): https://github.com/search?q=path%3A*%2Fpyproject.toml+%22%5C...
or setup.py: https://github.com/search?q=path%3A*%2Fsetup.py+%22%5C%22lit...
One thing not in that writeup is that very little action was needed for my engineer to get pwnd. uvx automatically pulled latest litellm (version unpinned) and built the environment. Then Cursor started up the local MCP server automatically on load.
https://inspector.pypi.io/project/litellm/1.82.8/packages/fd...
It would be interesting if Python, NPM, Rubygems, etc all just decided to initiate an ecosystem-wide credential reset. On one hand, it would be highly disruptive. On the other hand, it would probably stop the damage from spreading.
We are looking at similar attack vectors (pth injection), signatures etc. in other PyPI packages that we know of.
Now I am not worried about the Ai Api keys having much damage but I am thinking of one step further and I am not sure how many of these corporations follow privacy policy and so perhaps someone more experienced can tell me but wouldn't these applications keep logs for legal purposes and those logs can contain sensitive information, both of businesses but also, private individuals perhaps too?
But, one of the arguments that I saw online from this was that when a security researcher finds a bug and reports it to the OSS project/Company they then fix the code silently and include it within the new version and after some time, they make the information public
So if you run infrequently updated versions, then you run a risk of allowing hackers access as well.
(An good example I can think of is OpenCode which had an issue which could allow RCE and the security researcher team asked Opencode secretly but no response came so after sometime of no response, they released the knowledge in public and Opencode quickly made a patch to fix that issue but if you were running the older code, you would've been vulnerable to RCE)
https://github.com/crewAIInc/crewAI/commit/8d1edd5d65c462c3d...
The package was directly compromised, not “by supply chain attack”.
If you use the compromised package, your supply chain is compromised.
it does a lot of CPU intensive work
spawn background python
decode embedded stage
run inner collector
if data collected:
write attacker public key
generate random AES key
encrypt stolen data with AES
encrypt AES key with attacker RSA pubkey
tar both encrypted files
POST archive to remote hostOh boy supply chain integrity will be an agent governenace problem, not just a devops one. If you send out an agent that can autonomously pull packages, do code, or access creds, then the blast radius of compromises widens. That's why I think there's an argument for least-privilege by default--agents should have scoped, auditable authority over what they can install and execute, and approval for anything outside the boundaries.
This stupidity is squarely on GitHub CI. Trivy is also bad here but the blast radius should have been more limited.
They also seem to be spilling into HN [1].
Runaway AI agents? A meme I'm to old to understand?
[1] https://ramimac.me/teampcp/#spam-flood-litellm [2] https://ramimac.me/teampcp/#discussion-flooded
I guess I am lucky as I have watchtower automatically update all my containers to the latest image every morning if there are new versions.
I also just added it to my homelab this sunday, I guess that's good timing haha.
Also the repo is so active that it's very hard to understand the state of issues and PRs, and the 'day 0' support for GPT-5.4-nano took over a week! Still, tough situation for the maintainers who got hacked.
EDIT: here's what I did, would appreciate some sanity checking from someone who's more familiar with Python than I am, it's not my language of choice.
find / -name "litellm_init.pth" -type f 2>/dev/null
find / -path '/litellm-1.82..dist-info/METADATA' -exec grep -l 'Version: 1.82.[78]' {} \; 2>/dev/null
rg litellm --iglob='*.lock'no i don't let it connect to web...
What is the source of compromise?
Does anyone have a list of other compromised projects?
Basically, have all releases require multi-factor auth from more than one person before they go live.
A single person being compromised either technically, or by being hit on the head with a wrench, should not be able to release something malicious that effects so many people.
Domains might get added to a list for things like 1.1.1.2 but as you can imagine that has much smaller coverage, not everyone uses something like this in their DNS infra.
https://github.com/krrishdholakia/blockchain/commit/556f2db3...
- # blockchain
- Implements a skeleton framework of how to mine using blockchain, including the consensus algorithms.
+ teampcp owns BerriAIThere is also the need for data sanitation, because the attacker could distribute compromised files through user’s data which will later be run and compromise the host.
https://greyhaven.co/insights/how-greywall-prevents-every-st...
https://github.com/Nayjest/lm-proxy
``` pip install lm-proxy ```
Guys, sorry, as the author of a competing opensource product, I couldn’t resist
This would also disable site import so not viable generically for everyone without testing.
As described in https://docs.python.org/3/library/site.html :
> Lines starting with import (followed by space or tab) are executed.... The primary intended purpose of executable lines is to make the corresponding module(s) importable (load 3rd-party import hooks, adjust PATH etc).
So what malware can do is put something in a .pth file like
import sys;exec("evil stringified payload")
and all restrictions are trivially bypassed. It used to not even require whitespace after `import`, so you could even instead do something like import_=exec("evil stringified payload")
In the described attack, the imports are actually used; the standard library `subprocess` is leveraged to exec the payload in a separate Python process. Which, since it uses the same Python environment, is also a fork bomb (well, not in the traditional sense; it doesn't grow exponentially, but will still cause a problem)..pth files have worked this way since 2.1 (comparing https://docs.python.org/2.1/lib/module-site.html to https://docs.python.org/2.0/lib/module-site.html). As far as I can tell there was no PEP for that change.
What about reducing the number of dependencies ? Integrating core functionalities in builtin language libraries ? Avoiding frequent package updates ? Avoiding immature/experimental packages from developers of unknown reliability ?
Those issues are grave. I see no future when those get rarer, and I am afraid they may wipe the open-source movement credibility.
if you have tips i am sure they are welcome. snark remarks are useless. dont be a sourpuss. if you know better, help the remediation effort.
Migration guide: https://llmgateway.io/migration/litellm
This was taught in the 90s. Sad to see that lesson fading away.
First Trivy (which got compromised twice), now LiteLLM.
That's why I'm building https://github.com/kstenerud/yoloai
I'm sensing a pattern here, hmm.
The Python ecosystem provides too many nooks and crannies for malware to hide in.
I hope that everyone's course of action will be uninstalling this package permanently, and avoiding the installation of packages similar to this.
In order to reduce supply chain risk not only does a vendor (even if gratis and OS) need to be evaluated, but the advantage it provides.
Exposing yourself to supply chain risk for an HTTP server dependency is natural. But exposing yourself for is-odd, or whatever this is, is not worth it.
Remember that you are programmers and you can just program, you don't need a framework, you are already using the API of an LLM provider, don't put a hat on a hat, don't get killed for nothing.
And even if you weren't using this specific dependency, check your deps, you might have shit like this in your requirements.txt and was merely saved by chance.
An additional note is that the dev will probably post a post-mortem, what was learned, how it was fixed, maybe downplay the thing. Ignore that, the only reasonable step after this is closing a repo, but there's no incentive to do that.
Programming for different LLM APIs is a hassle, this library made it easy by making one single API you call, and in the backstage it handled all the different API calls you need for different LLM providers.
It's pretty disappointing that safetensors has existed for multiple years now but people are still distributing pth files. Yes it requires more code to handle the loading and saving of models, but you'd think it would be worth it to avoid situations like this.
Hundreds of downvoted comments like "Worked like a charm, much appreciated.", "Thanks, that helped!", and "Great explanation, thanks for sharing."
# add any dependency file patterns
osv-scanner -r .
as your projects mature, add osv-scanner as a blocking step to fail your installs before the code gets installed / executed.An actual infosec audit would have rigorously enforced basic security best practices in preventing this supply chain attack.