undefined | Better HN

0 pointsburkaman0y ago0 comments

Just opened a private window to try this, I did one search and clicked on four results, then a second search and got a 429 error. That is wild. I guess it's an anti-scraper measure?

0 comments

fencepost0y ago

Given the occasional articles that crop up showing the sheer volume of badly-behaved (presumably) AI scraper bots this makes all kinds of sense.

I can't find it now, but sometime in the past week or so I saw something that (IIRC) related to the BBC (?) blocking a ton of badly-behaved obvious scraper traffic that was using Meta (?) user-agents but wasn't coming from the ASNs that Meta uses. The graphs looked like this ended up reducing their sustained traffic load by about 80%.

Items where I'm doubting my recall (since I didn't find anything relevant doing some quick searches) are marked with (?)

j / k navigate · click thread line to collapse

0 comments

fencepost0y ago

Given the occasional articles that crop up showing the sheer volume of badly-behaved (presumably) AI scraper bots this makes all kinds of sense.

Items where I'm doubting my recall (since I didn't find anything relevant doing some quick searches) are marked with (?)

j / k navigate · click thread line to collapse