The observation is that Google makes money from eyeballs on ads. The reason they make money is that they can lead people to the content that they want.

The content that they want comes from websites, which benefit from the traffic google sends their way. If Google stops sending traffic their way, they won't block google in robots.txt: they'll simply shut down, because they're not making money off the eyeballs on their site.

So, it's a symbiotic relationship: Google sends traffic to sites. Sites send their contents to Google. If Google tries too hard to keep eyeballs on their own ads, then they become a strangler vine on their own ecosystem, depriving it of the resources that it needs to grow.

There needs to be a balance between the traffic that it sends to sites, and what it surfaces directly on the SERP.