I don't believe that robots.txt has ever had the backing of law.

Robots.txt doesn't restrict anything. It's just a request of what should and shouldn't be scraped by search engine spiders.