undefined | Better HN

0 pointsRebelgecko5y ago0 comments

Does that matter if they just add the whole website to their robots.txt in December of 2024?

0 comments

3 comments · 1 top-level

pabs35y ago· 2 in thread

IIRC archive.org pay attention to the robots.txt now rather than in 2024.

They do not. They have in the past removed public access to pages due to robots.txt changes.

pabs35y ago

Hmm, this old blog post sounds like they were planning on ending that practice:

j / k navigate · click thread line to collapse