> Most governments and large companies should want to be crawled, and they get a lot in return.

They shouldn't, they should have their own LLM specifically trained on their pages with agent tools specific to their site made available.

It's the only way to be sure that the answers given are not garbage.

Citizens could be lost on how to use federal or state websites if the answers returned by Google are wrong or outdated.