In theory, but can we control ourselves? There are a lot of financial incentives in developing better machines and algorithms. We couldn't stop nuclear weapons or global warming, and AI is a lot more attractive and powerful than either of those to business and governments. Not to mention that nuclear weapons and global warming are harmful in a very easily understood way, whereas AI might be harmful in very strange ways. It's like a pack of wolves left alone with a poisonous steak. It's a delicious steak and the poison is somewhat beyond their understanding; some wolves even think there's no poison at all. Isolating the poison from the steak is as difficult for the wolves as making safe AI is for us.
You can imagine how incredibly valuable an AI capable of doing a programmer's work would be. It's a technology far off, but not implausible.
But as soon as we reach that point, it seems unlikely that things won't spiral out of control. Imagine a thousand highly intelligent programmers who are capable of research, think fluently in statistics, and cooperate perfectly. Additionally, these programmers can examine and modify how their own brains work, and boost their performance, remove their errors. With the press of a button, they can also create more copies of themselves.
Everything might spiral out of control.
How do you make sure things don't spiral out of control?
