There’s not much room to squeeze in when your competitors hold the keys to 15 million top websites.
Why would website authors _want_ to prevent crawling by other search engines?