You could maybe still only follow robots.txt rules for Googlebot.