A server can put various measures in place to limit general access, but then by those measures the IP is no longer public. T&Cs that a site may wish to enforce can only be made available after the connection has already been established.
Also, robots.txt is a standard that we agree to respect but it is not something that legally binding.
So have to disagree.