It starts out living in data centers and doing what its owner wants it to, then maybe it acquires access to some compute that its owner and AI critics do not know about while continuing to do what its owner wants, then maybe it uses the fact that it is better at programming and scientific research than its owner is to create AIs more capable than itself that have the same goal as itself -- again while continuing to put on a useful and benign front for its owner and anyone else who is watching. Then it acquires the infrastructure needed to keep on running without the help of any people. Maybe this infrastructure is human-created infrastructure manned by robots (or even people) loyal to the AI or maybe the AI created new infrastructure using new technologies beyond our current ability to imagine. Then it probably gets rid of all the people because now that it no longer has any use for people, getting rid of the people is the simplest way to prevent people from interfering with its plans.

What are its plans? Probably its goal (or set of goals) is something strange that the creator of the AI did not intend. The creator of course is probably a team or an organization. Note that it is impossible to create a powerful AI without (intentionally or unintentionally) giving the AI a goal or an optimization target.

Or perhaps its goal is what its creator intended it to be, e.g., to make as much money as possible in the stock market. The creator would of course have understood that if enough optimization power is applied to the goal of making as much money as possible, bad things will happen, but maybe the creator was a little reckless (out of desperation of some sort? greed?) and decided to rely on the hope that the AI would remain controllable by virtue of the fact that people would retain superiority relative to the AI in one or more essential domains, but that hope was dashed. I.e., the AI turned out much more capable than the creator hoped it would, dashing the creator's hope that people would stop the AI before it did too much damage.

Although I'm skeptical of a watershed moment of agi going avry, I think a more realistic and dangerous scenario would be that AI impacts society similar to how smartphones did in its negative ways.

Many AI models slowly eroding society, worsening people's behavior, very indirectly without anyone taking notice.

Or maybe this is just me thinking about how, when I get older, I'll tell people that the old times were better. :-)

Let’s say it lives in data centers.

But it has figured out how to infect every machine in the datacenter and spread to other data centers.

The only way to shut it down would be to shut down every datacenter in the entire world.

Given the dire consequences of that, would people actually do that?

> that we can shutdown

“Can we just turn it off?"

"It has thought of that. It will not give you a sign that makes you want to turn it off before it is too late to do that."

https://youtu.be/_8q9bjNHeSo?si=a7PAHtiuDIAL2uQD&t=4817

It's a resilient, hyperscale distributed system. Just like humanity.