I figure if it's on the web and a human can read it, my computer ought to be able to read it too.
What about domains that have been sharked ? Does controlling robots.txt now give me the right to suppress all content ever originating from that domain, for as long as I control robots.txt?
Internet archive is right to spider the site, but defer showing. Collection != dissemenation.
The IA isn't synthesizing, selling, cross referencing or afaict doing anything nefarious with the data.
You are literally picking on the last org on the internet that needs to get picked on.
True but at the same time I do understand those who generally want others to abide by their Robots.txt. INAL, but ideally I would love to see the IA having the right to ignore Robots.txt (however, if some one wants to opt out of the IA they should be given the option) while others shouldn't be allowed to do so.
