We went from hundreds of thousands of requests per day to 5 million per day. Traffic was web scraping bots based on the obscure URLs. The URLs were valid (mediawiki history links, etc) and not attempts to hack the site. Banning IPs did not help, the traffic would move to new subnets. Mostly IPv4, some IPv6. The user agent was popular Chrome agent strings so I'm guessing it was masked puppeteer.
It was a DDoS in practice but I get the feeling it's an immature web crawler.
I think people are likely building a new generation of crawlers to feed LLMs as fast as possible.
The caching aspect of Cloudflare helped a lot. Putting specific url patterns behind Cloudflares dynamic JavaScript challenge also helped. It was surprisingly easy to setup.
And I know what some of you will say, Cloudflare is bad. I've personally been annoyed with them for making specific sites more difficult to use while on VPN. But it's not a hard choice when it's either taking your site offline or using their free tier offering.
One thing that got me was seeing some of the malicious traffic originate from the same /24 as I use at home. Whatever botnet was being used certainly has good penetration of residential ISPs in the US!
At any rate, taking down the Blender website wouldn't help them I don't think.
Worth refraining for using it.
[1]: https://www.theverge.com/2023/3/3/23623473/blender-stable-di...
They litterally did that :)
And that can be controlled by your site's security level. Highest settings will show the JS page to all vistors, medium only shows it to likely bots.
"Under Pressure" by Queen.
