Show HN: Robots.txt generator for blocking AI crawlers (opens in new tab)

(aisearchwatch.com)

9 pointseummm1y ago1 comments

1 comments

It's incredible that the author of this is not aware that robots.txt is widely ignored.

How more useful tool would be something that generates a piece of web server config to actually block bots.

j / k navigate · click thread line to collapse

It's incredible that the author of this is not aware that robots.txt is widely ignored.

How more useful tool would be something that generates a piece of web server config to actually block bots.

j / k navigate · click thread line to collapse