Uh oh... Something didn't work. > http://disqus.com/human.txt
Tried to curl it, exact content, no 302 towards a "<script>window.close</script>",... Got anything?
Not that you should do that. Robots.txt is a nicety though, the client doesn't have to respect it, and the server doesn't have to allow your HTTP requests.
robots.txt is basically a list of rules that lay out "This is how we'd like you to crawl us. We might stop serving you if you don't comply", rather than a hard-and-fast set of directives that specify how a webcrawler will be guaranteed to behave.
