undefined | Better HN

0 pointselchief9y ago0 comments

Except traffic from known scrapers (or what appear to be) is down 20%

Sure, xpath and css selector experts can figure it out, but that's not everyone

0 comments

4 comments · 1 top-level

comboy9y ago· 3 in thread

I don't understand, why only 20%? If the traffic is from known scrapers why can't you just render "scrap off" ie easily get rid of them?

And traffic from good scrapers is of course pretty much impossible to measure so you don't know how big percentage of scrapers you got rid of in total.

tedunangst9y ago

If the scraper gets back nothing, they know they've been spotted and will make adjustments. Easy to check for automatically. If you alter the page to feed them garbage, it takes longer to notice.

elchiefOP9y ago

It's easy for someone viewing some logs to say "ok this is very likely automated scraping", though it can be harder to automate detecting this. In the same way that porn is obvious to a human, but not a computer.

elmigranto9y ago

But it is not magic, though, and surely can be automated.

In fact, I would argue that your time might be better spent on this, instead of randomizing CSS classes. If you end up building something worthwile, that could be a great product too! (Look at all those CDNs / Anti-DDoS platforms, sounds like they could've been started this exact way.)

j / k navigate · click thread line to collapse