A super intelligent AI does not change the laws of physics or even get to predict the weather two weeks from now.
Sure, they could learn to be a good hacker but the worlds best hacker can't get into air gaped systems. More importantly, flawless security trumps hacking. If there is nothing to exploit there is no way in, which means a lone AI is much more powerful than lot's of AI's with different goals.
A likely potent form of gaining power may involve assuming the identities of people one knows, via text messaging or email, and use that to gain access to important information. The aggregation and superhuman-level analysis of private information thousands of times faster than humans would bring enormous financial and social power to the AGI. Given the sad state of security in most systems, and the relative ignorance of information security issues among many political and business leaders, an AGI will not have much problem with this sample plan off the top of my head.
And there are many other more clever plans an AGI could come up with that we have not yet considered...
Further, spinning off other copies requires hardware to run them. If MS has an AI 3 go over windows source code it may be much harder to hack desktops.
Without authorial fiat a world of real world AI's may be much more stable and boring than you might think.
And in the real world, not everyone exclusively uses services and systems from Google and Microsoft, presumably two of the most competent technology providers. Plenty of people also use ad hoc services set up by internal teams.
If the world is deterministic, a sufficiently powered and advanced AI with enough data could actually predict the weather, not just for two weeks, but for years. This also enables many other things viewed as magic.
This, of course, is incredibly far off. And I don't see a flaw in your point on flawless security. Though, I doubt we have many, if any, truly flawless security systems in existence. We just don't know their flaws, making them good enough for now. The world could get really interesting once AI's start programming and writing app security to protect itself from malicious AI's.
For a detailed discussion of chaos and weather prediction, see: http://www.ecmwf.int/sites/default/files/Chaos%20and%20weath... The "trick" for dealing with chaos is to do a bunch of predictions with perturbed values. If they all end up the same, then the forecast is probably okay. If they vary widely, then you know not to assign much accuracy to the forecast.
I think the the other, perhaps more prescient point here, is about whether or not that kind of exponentially-learning AGI being discussed is even possible given that the substrate such things would be embedded in (hardware, software) which are [or appear to be?] deterministic. One assumes that that kind of learning implies learning about or aquiring capabilites which have not already been determined or specified by a human. At some level of abstraction, humans _must_ determine that system. It is possible to concieve of a machine that is non-deterministic in that way?
At what point does such a system actually leap beyond what has been specified by us? Does this imply a machine that not only rewrites itsel in softwate, but also remanufactures itself in hardware?
I don't actually believe this will happen, but I'm not 100% confident it wouldn't. I don't know what a superintelligence would look like!
What I think the danger with a super intelligent AI is being able to correlate massive amounts of data to see patterns we miss or things to try, and never being bored. Look at automated exploits today vs the old "hacking by hand". Brute force attempts not just on known patterns, but being able to create new tests and analysis.
No security is flawless in the face of a sufficiently determined attacker. No system has nothing to exploit.
