Crawlers impact the operations of the Wikimedia projects (opens in new tab)

(diff.wikimedia.org)

13 pointsmanuelfcreis11mo ago3 comments

3 comments

Seems like a serious issue many sites have nowadays. I really don't want this solved by Captchas everywhere.

If it comes to it, just block all known IP blocks of cloud providers.

There are likely a number of beneficial bots[0] running from cloud providers. And residential proxies are also widely available for very determined scrapers.

[0]: <https://stuartgeiger.com/talks/2013-08-03-wikisym-levee-brea...>

j / k navigate · click thread line to collapse

3 comments

anotherpaul11mo ago

Seems like a serious issue many sites have nowadays. I really don't want this solved by Captchas everywhere.

Handy-Man11mo ago

If it comes to it, just block all known IP blocks of cloud providers.

thcipriani11mo ago

There are likely a number of beneficial bots[0] running from cloud providers. And residential proxies are also widely available for very determined scrapers.

[0]: <https://stuartgeiger.com/talks/2013-08-03-wikisym-levee-brea...>

j / k navigate · click thread line to collapse