undefined | Better HN

0 pointssuperkuh4y ago0 comments

Yep. I 403 turnitin and similar companies via nginx configuration, if ($http_referer ~* (TurnitinBot|PaperLiBot|idmarch|FairShare|Lightspeedsystems|ZmEu|BPImageWalker|semrushBot|ias_crawler|360spider|copyrightinfringementportal|PetalBot|Adsbot|SlySearch|NPBot)) { return 403; }

But my favorite robots.txt is,

    User-agent: Zombies
    Disallow: /brains

0 comments

4 comments · 2 top-level

easrng4y ago· 2 in thread

Why are you blocking PetalBot? It's an actual search engine.

superkuhOP4y ago

Legit Huawei IP ranges identifying as Huawei PetalBot were being abusive, definitely not obeying robots.txt, and searching for subsets of content that indicated they were looking to identify political dissidents with no worries about actually indexing the full site. I don't consider it a real search engine.

But yeah, maybe not a good fit for this list of educational and copyright parasites.

easrng4y ago

Oh yikes, that sounds bad. It is a real search engine though, https://petalsearch.com/

rcarmo4y ago

Shouldn’t that be…

    User-agent: Zombies
    Disallow: /braaains

?

1 more reply

j / k navigate · click thread line to collapse