undefined | Better HN

0 pointsthreeseed1y ago0 comments

You just have a rule that says block everything except crawlers: A, B, C.

Also the AppleBot was known about before it appeared in Siri.

0 comments

So you expect all websites to block FoobarSearch so it never gets off the ground and becomes a big search engine that people know to unblock.

Then FoobarSearch learns to ignore robots.txt wildcards, and we're back at square one.

IIRC this happened to DDG or Bing.

Websites have always had the ability to precisely control who has access to their content.

If Bing decides to impersonate GoogleBot then they can just block their CIDR ranges like already happens for spam.

j / k navigate · click thread line to collapse