I mean I'd prolly be okay paying yearly fee for access to such a registry.
More seriously, automated scanners seem to do a good job already of finding malicious packages. It's a wonder that npm themselves haven't already deployed an automated countermeasure.
That's not true. This latest incident was detected by an individual researcher, just like many similar attacks in the past. Time and again, it's been people who flagged these issues, later reported to security startups, not automated tools. Don't fall for the PR spin.
If automated scanning were truly effective, we'd see deployments across all major package registries. The reality is, these systems still miss what vigilant humans catch.
> It started with a cryptic build failure in our CI/CD pipeline, which my colleague noticed
> This seemingly minor error was the first sign of a sophisticated supply chain attack. We traced the failure to a small dependency, error-ex. Our package-lock.json specified the stable version 1.3.2 or newer, so it installed the latest version 1.3.3, which got published just a few minutes earlier.
[1] https://blogs.microsoft.com/blog/2024/05/03/prioritizing-sec...
2) Real chances for owners to notice they have been compromised
3) Adopt early before that commons is fully tragedy-ed.
oh, you can use commas too.
and if you're still not thinking this is fun, here's a quote from Wikipedia "But keep in mind that "PT36H" is not the same as "P1DT12H" when switching from or to Daylight saving time."
just add a unit to your period parameters. sigh.
/s
This for instance will only install packages that are older than 14 days:
uv sync --exclude-newer $(date -u -v-14d '+%Y-%m-%dT%H:%M:%SZ')
It's great to see this kind of stuff being adopted in more places.
It would also be nice to have this as a flag so you can use it on projects that haven't configured it though, I wonder if that could be added too.
https://research.swtch.com/vgo-mvs#upgrade_timing
MVS makes tons of sense that you shouldn't randomly uptake "new" packages that haven't been "certified" by package maintainers in their own dependencies.
In the case of a vulnerable sub-dependency, you're effectively having to "do the work" to certify that PackageX is compatible with PackageY, and "--minAge" gives industry (and maintainers) time to scan before insta-pwning anyone who is unlucky that day.
To achieve my goal, would this approach work:
- Pin all of my package.json versions (no prefacing versions with ~ or ^)
- Routine installation of packages both on my local and on CI servers will be done using `npm ci`
- `npm install <package_name> --save-exact/--save-dev` would be used only at the time of adding a package to package.json, followed by an `npm ci`
- Rely on tooling like GitHub Dependabot and CodeQL to inform the team when a dependency should be updated for security reasons and then manually update only the dependency with the desired version using `npm install lodash@4.17.21 --save-exact`, for example
EDIT: Thinking about this more, we would have to forbid deleting the package-lock.json and regenerating it with `npm install` and forbid the use of `npm update` so that package-lock.json would stay stable.
On the flipside sometimes you get lucky and being on an old version of a package means you don't have the vulnerability in the first place.
libyear is a helpful metric for tracking how much of this debt you might have.
Does the JS ecosystem really move so fast that you can’t wait a month or two before updating your packages?
In the JVM ecosystem it's quite common to have Dependabot or Renovate automatically create PRs for dependency upgrades withing a few hours of it being released. If it's manual it highly irregular and depends on the company.
The main deciding factors were the process and frequency it was released / upgraded by us or our customers.
The on-prem installs had the longest delay because once it was out there it was harder for us to address issues. Some customers also had a change freeze in place once things have been approved which was a pain to deal with if we needed to patch something for them.
Products that had a shorter release or update cycle (e.g. the mobile app) had a shorter delay (but still a delay) because any issue could be addressed faster.
The services that were hosted by us had the shortest delay on the order of days to weeks.
There were obviously exceptions in both directions but we tried to avoid them.
Prioritisation wasnt really an issue - a lot of dependencies were increased on internal builds so we had more time to test and verify before committing to it once it reached our stability rules.
Other factors that influenced us: - Blast radius - a buggy dependency in our desktop/server applications had more chance to cause damage than our hosted web application so it rolled a little slower for dependencies.
- Language (more like ergonomics of the language) - updating our C++ deps was a lot more cumbersome than JS deps)
Normally old major or minor packages don't get an update, only the latest.
E.g. 4.1.47 (no update), 4.2.1 (yes got update).
So if the problem is in 4.1 you must "upgrade" to 4.2.
With "perfect" semver, this shouldn't be a problem, cause 4.2 only add new features... but... back to reality, the world is not perfect.
Suppose you have a package P1 with version 1.0.0 that depends on D1 with version ^1.0.0. The “^” indicates a range query. Without going into semver details, it helps update D1 automatically for minor patches or non-breaking feature additions.
In your project, everything looks fine as P1 is pinned to 1.0.0. Then, you install P2 that also uses D1. A new patch version of D1 (1.0.1) was released. The package manager automatically upgrades to 1.0.1 because it matches the expression ^1.0.0, as specified by P1 and P2 authors.
This can lead to surprises. JS package managers use lock files to prevent changes during installs. However, they still change the lock file for additions or manual version upgrades, resolving to newer minor dependencies if the version range matches. This is often desirable for bug fixes and security updates. But, it opens the door to this type of attack.
To answer your question, yes, the JS ecosystem moves faster, and pkg managers make it easy to create small libraries. This results in many “small” libraries as transitive dependencies. Rewriting these libraries with your own code works for simple cases like left-pad, but you can’t rewrite a webserver or a build tool that also has many small transitive dependencies. For example, the chalk library is used by many CLI tools to show color output.
I don't think people are having major versions updated every month, it is more really like 6 months or once a year.
I guess the problem might be people think auto updating minor versions in CI/CD pipeline will keep them more secure as bug fixes should be in minor versions but in reality we see it is not the case and attackers use it to spread malware.
The problem is that "should" assumes that point releases never introduce regressions (whether they be security, performance, or correctness). Unfortunately, history has shown that regressions can and do happen. The best practice for release engineering (CI/CD, if you will) is to assume the worst, test thoroughly, and release incrementally (include bake time).
Delaying updates isn't just a backstop against security vulnerabilities; it's useful for letting the dust settle after an update of any kind that can adversely impact the application. The theory is that someone will find it before you, report it, and that a fix will be issued.
Really depends on the context and where the code is being used. As others have pointed out most js packages will use semantic versioning. For the patch releases (the last of the three numbers), for code that is exposed to the outside world you generally want to apply those rather quickly. As those will contain hotfixes including those fixing CVEs.
For the major and minor releases it really depends on what sort of dependencies you are using and how stable they are.
The issue isn't really unique to the JavaScript eco system either. A bigger java project (certainly with a lot of spring related dependencies) will also see a lot of movement.
That isn't to say that some tropes about the JavaScript ecosystem being extremely volatile aren't entirely true. But in this case I do think the context is the bigger difference.
> then again, we make client side applications with essentially no networking, so security isn’t as critical for us, stability is much more important)
By its nature, most JavaScript will be network connected in some fashion in environments with plenty of bad actors.
In 2 months, a typical js framework goes through the full Gartner Hype Cycle and moves to being unmaintained with an archived git repo and dozens of virus infected forks with similar names.
I've also had cases where I've found a bug in a package, submitted a bug report or PR, and then immediately pulled in the new version as soon as it was fixed. Things move fast in the JavaScript/npm/GitHub ecosystem.
I know it would take time for packages to adopt this but it could be implemented as parameters when installing a new dependency, like `npm i ping --allow-net`. I wouldn't give a library like chalk access to I/O, processes or network.
You might be able to do this around install scripts, though disk writing is likely needed for all (but perhaps locations could be controlled).
Yeah, it needs work from the language runtime, but I think even a hacky, leaky 'security' abstraction would be helpful, because the majority of malware developers probably aren't able to break out of a language-level sandbox, even if the language still allows you to do unsafe array access.
Then we can iterate.
It's too bad, it would be useful in this situation
- Reviewing the source code in the actual published package
- Tooling that enable one to easily see a trusted diff between a package version and the previous version of that package
- Built-in support in the package manager CLIs to only install packages that have a sufficient number of manual reviews from trusted sources (+ no / not too many negative reviews). With manual review required to bypass these checks.
There are enough users of each package that such a system should not be too onerous on users once the infrastructure was in place.
"waiting a length of time doesn’t increase security, and if such a practice became common then it would just delay discovery of vulnerabilities until after that time anyways"
https://github.com/npm/rfcs/issues/646#issuecomment-12824971...
Large tech companies, as with most industry, have realized most people will pay with their privacy and data long before they'll pay with money. We live in a time of the Attention Currency, after all.
But you don't need to be a canary to live a technology-enabled life. Much software that you pay with your privacy and data has free or cheap open-source alternatives that approach the same or higher quality. When you orient your way of consuming to 'eh, I can wait till the version that respects me is built', life becomes more enjoyable in myriad ways.
I don't take this to absolute levels. I pay for fancy pants LLM's, currently. But I look forward to the day not too far away where I can get today's quality for libre in my homelab.
There's an open discussion about adding something similar to bun as well^
minimumReleaseAge doesn't seem to be a bulletproof solution so there's still some research/testing to be done in this area
Go is one of the few packing systems that got these right.
Two alternatives:
- The occasional alert from `npm audit` that you have to carefully, deliberately, and thoughtfully upgrade your way out of.
- The shifting sands of 100s or 1000s of towering deps that change literally every time you `pnmp install`.
The second one is the current situation and it is madness.
There should be no package lock because package.json should be the package lock.
Though pnpm does have a setting to help with this too: https://pnpm.io/settings#resolutionmode time-based, which effectively pins subdependencies based on the published time of the direct dependency.
Thank you, I'll check it that setting!
We should strive for software to do one thing well (or at least be made up of modular parts that do one thing well) and prize backwards compatability, so that it does not require constant churn.
The sane middle ground between "constant upgrades" and "never upgrade" is to upgrade when there is an actual vulnerability found in a dependency. Instead of churn for no reason, you update only with a good reason.
pnpm config set -g minimumReleaseAge 1440
Does that work as well? I can't tell if the global settings are the same as workspace settings, and it lets me set nonsense keys this way, so I'm not sure if there is a global equivalent.
Basically we severed connection to the public npm registry completely earlier in the week whilst this worm plays out.
Unfortunately there wasn't a way to do this without taking our cached "good" public packages down as well, so we later replicated the good cached packages into a new standalone private registry to be the new upstream.
The bit that was not obvious in the moment but self evident once we realised is that the registry we're using took the copy time as the publish time, and therefore our new 2 week delay is rejecting the copied packages...
So sample size of one, but the registry we're using is definitely using upload time not any metadata in the packages themselves. Good to know the filtering is working.
Good to see some OSS alternatives showing up!
Of course, the malware could just embed itself as an IIFE and get launched when the package is loaded, so disallowing postinstall is not really a security solution.
Yes if someone compromises a package then they can also inject malicious code that will trigger at runtime.
But the thing about the recent NPM supply chain attack - it happened really quickly. There was a chain reaction of packages that got compromised which lead to more authors getting compromised. And I think a big reason why it moved so quickly was because of post-install scripts. If the attack happened more slowly, then the community would have more time to react and block the compromised packages. So just slowing down an attack is valuable on its own.
maybe its better to disallow latest than use age as a metric.
If I see someone using npm as a cli tool unironically...
But the real solution to this kind of attack is to stop resolving packages by name and instead resolve them by hash, then binding a name to that hash for local use.
That would of course be a whole different, mostly unexplored, world, but there's just no getting around the fact that blindly accepting updated versions of something based on its name is always going to create juicy attack surface around the resolution of that name to some bits.
you can only unpublish.
content hash integrity is verified in lockfiles.
the problem is with dependencies using semver ranges, especially wide ones like "debug": "*"
initiatives like provenance statements [0] / code signing are also good complement to delayed dependency updates.
also not running as default / whitelisting postinstall scripts is good default in pnpm.
modifying (especially adding) keys in npmjs.org should be behind dedicated 2fa (as well as changing 2fa)
The only immutability that counts is immutability that you can verify, which brings us back to cryptographic hashes.
As for lock files, they prevent skulduggery after the maintainer has said "yeah, I trust this thing and my users should too" but the attacks we're seeing is upstream of that point because maintainers are auto-trusting things based on their name+version pair, not based on their contents.
A better (not perfect) solution: Every package should by AI analysed on an update before it is public available, to detect dangerous code and set a rating.
In package.json should be a rating defined, when remote package is below that value it could be updated, if it is higher a warning should appear.
But this will cost, but i hope, that companies like github, etc. will allow package-Repositories to use their services for free. Or we should find a way, to distribute this services to us (the users and devs) like a BOINC-Client.
