Well, if they rework things so that requests all originate from the same IP address or a small set of addresses, then regular IP-based rate limits should work fine right?
The point is just to stop what is effectively a DDoS because of shitty web crawlers, not to stop the crawling entirely.