robots.txt files are not mechanisms for authorizing access; they are guidelines designed to instruct user-agents on relevant resources. They essentially say to the user-agent "the content marked as 'disallow' should not be crawled for indexing purposes for unspecified reasons"

Adhering to a robots.txt file is not mandatory, though it does make you a good netizen/curries favour with server admins.

So is selling counterfeit goods, so I'd guess they figure it's mutually-assured-destruction. And of course it's a lot easier to prove that someone is selling counterfeit goods from your brand than for them to prove that you intentionally wrote a web crawler to exceed unauthorized access. And IIRC it's considered legal to exceed unauthorized access in order to report a crime (I think that's been the case a few times with people who've hacked into computers and then reported child porn they've found to the Feds, which is kinda moral hazard IMO because it'd be very easy for a hacker to frame someone that way).