There's a "system trap" where building intelligent systems or augmenting humans will grant at least a short term advantage, it has natural "arms race" dynamics.

Historically people have not done well at avoiding systems traps and maintaining any sort of global consensus not to do things. Particularly things that a) don't seem to be an immediate atrocity and b) grant economic advantage.