PSA – ByteDance and Bytespider Bots – Recommend Blocking (opens in new tab)

(wordpress.org)

4 pointsexar08152y ago2 comments

2 comments

2 comments · 2 top-level

Also, interesting Video - albeit low content density - pointing out that those bots cause humongous amounts of traffic, maxing out most vps bandwidth plans and are not currently blocked by Cloudflare.

For me, it looks like ByteDance is aggressively scraping the web for ML purposes.

bediger40002y ago

The article is correct: Bytespider does not observe robots.txt directives.

I set Apache to give Bytespider 404 on anything. Not even Ahrefsbot is this poorly behaved.

j / k navigate · click thread line to collapse