undefined | Better HN

0 pointsdogma11384mo ago0 comments

Do they actually do anything to circumvent paywalls or do websites just whitelist their crawlers?

0 comments

Websites don’t whitelist their crawlers, they maintain custom bypasses for a wide variety of websites.

If the websites were inclined to whitelist these crawlers, they’d also whitelist archive.org which is actually easy to whitelist. Archive.is is not

j / k navigate · click thread line to collapse

0 pointsdogma11384mo ago0 comments

Do they actually do anything to circumvent paywalls or do websites just whitelist their crawlers?

Websites don’t whitelist their crawlers, they maintain custom bypasses for a wide variety of websites.

If the websites were inclined to whitelist these crawlers, they’d also whitelist archive.org which is actually easy to whitelist. Archive.is is not

j / k navigate · click thread line to collapse