Tell HN: Camelgate NPM Outage (Cloudflare)

122 pointsbavarianbob11mo ago36 comments

EDIT: Back online?!

NPM discussion: https://github.com/npm/cli/issues/8203

NPM incident: https://status.npmjs.org/incidents/hdtkrsqp134s

Cloudflare messaging: https://www.cloudflarestatus.com/incidents/gshczn1wxh74

GitHub issue: https://github.com/sindresorhus/camelcase/issues/114

Anyone experiencing npm outage that's more than just the referenced camelcase package?

36 comments

tom_usher11mo ago

Seems to be a change in Cloudflare's managed WAF ruleset - any site using that will have URLs containing 'camel' blocked due to the 'Apache Camel - Remote Code Execution - CVE:CVE-2025-29891' (a9ec9cf625ff42769298671d1bbcd247) rule.

That rule can be overridden if you're having this issue on your own site.

internetter11mo ago

> any site using that will have URLs containing 'camel' blocked

What engineer at cloudflare thought this was a good resolution?

Raed66711mo ago

I doubt the system is that simple. No one wrote a rule saying `if url.contains("camel") then block()` it's probably an unintended side-effect

3 more replies

cbovis11mo ago

Confirmed here: https://www.cloudflarestatus.com/incidents/gshczn1wxh74

oncallthrow11mo ago

WAFs are so shit

ronsor11mo ago

WAFs are literally "a pile of regexes can secure my insecure software"

2 more replies

UltraSane11mo ago

But are they less shit than the shitty software they filter traffic for?

Recursing11mo ago

Any path with the word "camel" seem to trigger this: https://www.npmjs.com/search?q=camel | https://registry.npmjs.org/camel123 | https://registry.yarnpkg.com/camel456

Some discussion here https://github.com/npm/cli/issues/8203

Edit: this is resolved now https://status.npmjs.org/incidents/hdtkrsqp134s

pvg11mo ago

This is not CF WAF's first rodeo https://news.ycombinator.com/item?id=20421538

Cementing its track record as a product that mostly doesn't do anything except for occasionally break the internet here and there to keep things fun and interesting.

lynnesbian11mo ago

> a product that mostly doesn't do anything except for occasionally break the internet

I wouldn't say that. The postmortem you referred to links to another CloudFlare blog post - one about a pretty serious RCE vuln in Microsoft SharePoint that was blocked by their WAF: https://blog.cloudflare.com/stopping-cve-2019-0604/

pvg11mo ago

I mean, it's hardly surprising CloudFlare will tell you this is a useful product. But it is to securing a web application what regex is to parsing HTML.

1 more reply

AdamJacobMuller11mo ago

I'm not sure why "WAF has false positives" makes it useless, nor would I say this is anywhere near the scale of "breaking the internet" and I'm not even fan of the concept of WAFs in general.

pvg11mo ago

The last one took out a lot more stuff than this one but the argument is the same - this product is a checkmark thing and when it's not fulfilling its checkmark purpose, it causes outages. Still an amusing bi-modality! I suppose it shares it with DNSSEC.

1 more reply

calvinmorrison11mo ago

we've used it to rescue some vintage appliances that are basically unsecurable.

nwalters51211mo ago

The npm folks have officially acknowledged an incident now: https://status.npmjs.org/incidents/hdtkrsqp134s

miyuru11mo ago

Outsourcing WAF is a double-edged sword.

I would have thought a large company like GitHub or Microsoft can have their own WAF team for their apps.

(NPM is owned by GitHub, and GitHub is owned by Microsoft)

klysm11mo ago

This is what you get when you buy security as an add-on product

troyvit11mo ago

Some orgs can't afford not to.

mplanchard11mo ago

Glad you posted something, thought I was going nuts

drusepth11mo ago

Is this also why unpkg has been up and down all morning?

ycombinatrix11mo ago

unpkg barely works even when there's no incident

time4tea11mo ago

Scunthorpe problem

j / k navigate · click thread line to collapse

36 comments

tom_usher11mo ago

That rule can be overridden if you're having this issue on your own site.

internetter11mo ago

> any site using that will have URLs containing 'camel' blocked

What engineer at cloudflare thought this was a good resolution?

Raed66711mo ago

I doubt the system is that simple. No one wrote a rule saying `if url.contains("camel") then block()` it's probably an unintended side-effect

3 more replies

cbovis11mo ago

Confirmed here: https://www.cloudflarestatus.com/incidents/gshczn1wxh74

oncallthrow11mo ago

WAFs are so shit

ronsor11mo ago

WAFs are literally "a pile of regexes can secure my insecure software"

2 more replies

UltraSane11mo ago

But are they less shit than the shitty software they filter traffic for?

Recursing11mo ago

Any path with the word "camel" seem to trigger this: https://www.npmjs.com/search?q=camel | https://registry.npmjs.org/camel123 | https://registry.yarnpkg.com/camel456

Some discussion here https://github.com/npm/cli/issues/8203

Edit: this is resolved now https://status.npmjs.org/incidents/hdtkrsqp134s

pvg11mo ago

This is not CF WAF's first rodeo https://news.ycombinator.com/item?id=20421538

Cementing its track record as a product that mostly doesn't do anything except for occasionally break the internet here and there to keep things fun and interesting.

lynnesbian11mo ago

> a product that mostly doesn't do anything except for occasionally break the internet

pvg11mo ago

I mean, it's hardly surprising CloudFlare will tell you this is a useful product. But it is to securing a web application what regex is to parsing HTML.

1 more reply

AdamJacobMuller11mo ago

I'm not sure why "WAF has false positives" makes it useless, nor would I say this is anywhere near the scale of "breaking the internet" and I'm not even fan of the concept of WAFs in general.

pvg11mo ago

1 more reply

calvinmorrison11mo ago

we've used it to rescue some vintage appliances that are basically unsecurable.

nwalters51211mo ago

The npm folks have officially acknowledged an incident now: https://status.npmjs.org/incidents/hdtkrsqp134s

miyuru11mo ago

Outsourcing WAF is a double-edged sword.

I would have thought a large company like GitHub or Microsoft can have their own WAF team for their apps.

(NPM is owned by GitHub, and GitHub is owned by Microsoft)

klysm11mo ago

This is what you get when you buy security as an add-on product