Elaborate? It is the best thing web has for crawling rules. Yea ofc crawling rules can be broken and violated but all serious internet search engines respect robots.txt.