Don't stop at robots.txt blocking. Look through your access logs, and you'll likely find a few IPs generating a huge amount of traffic. Look them up via "whois," then block the entire IP range if it seems like a bot host. There's no reason for cloud providers to browse my personal site, so if they host crawlers, they get blocked.
