It will however index hallucinated results generated with GPT and published somewhere, so once we're at that point it really doesn't matter anymore.
Google just gives you associations provided by random other people on the internet. It's largely garbage, most often deliberately disingenuous (to make you look at an ad). Ad revenue models for the internet encourage the generation of this type of false material.
A better criticism would be that the same thing will happen to something like chatgpt -- and the question is whether the model for analysis can better handle it at scale.
No, it absolutely does not. Yes, there is SEO spam in the index, but no - it is not Google hallucating it. It really exists on the internet, see also the second point of my comment.
> the same thing will happen to something like chatgpt
This isn't something that "happens to" GPT, GPT is doing it. There's probably even already GPT -> SEO spam pipelines out there generating websites.
But scientific facts are a different story. Nobody has any incentive to claim that 1+2=4 or that some function in Python does X when it really does Y. So when you search for these kind of facts on Google you can pretty sure that you get correct answers, or at least someone trying to give you the best answer they can. But not so with GPT. It may give you incorrect answers even for these kind of facts if they are not within the reasoning ability / training data.
I wouldn't categorise that as "hallucinating fictitious results" - the algorithms still only returns existing results. If you follow the link, you will find key words embedded in the HTML or visible text in the browser.
Different kettle of fish entirely.
Some kind of hybrid of this and search would be great.
ChatGPT doesn't, so you don't know where it got its information, and you'd have to do web searches anyway to get some possible sources.