undefined | Better HN

0 pointsJimDabell7mo ago0 comments

> Many people don't want their data used for free/any training.

That is true. But robots.txt is not designed to give them the ability to prevent this.

0 comments

It is in the name, rules for the robots. Any scraping ai or not, and even mass recrsive or single page, should abide by the rules.

j / k navigate · click thread line to collapse

0 pointsJimDabell7mo ago0 comments

> Many people don't want their data used for free/any training.

That is true. But robots.txt is not designed to give them the ability to prevent this.

gunalx7mo ago

It is in the name, rules for the robots. Any scraping ai or not, and even mass recrsive or single page, should abide by the rules.

j / k navigate · click thread line to collapse