undefined | Better HN

0 pointsandrew_zhong1mo ago0 comments

We do respect robots.txt production - also scraping browser providers like BrightData enforces that.

I will add a PR to enforce robots.txt before the actual scraping.

0 comments

How can people believe that you are respecting bot detection in production when your software's README says it can "Avoid detection with built-in anti-bot patches"?

andrew_zhongOP1mo ago

I hear you loud and clear - will replace the stealth browser with plain playwright and remove anti-bot as a feature.

j / k navigate · click thread line to collapse

0 comments

plastic0411mo ago

How can people believe that you are respecting bot detection in production when your software's README says it can "Avoid detection with built-in anti-bot patches"?

andrew_zhongOP1mo ago

I hear you loud and clear - will replace the stealth browser with plain playwright and remove anti-bot as a feature.

j / k navigate · click thread line to collapse