People seem to have this idea of AGI that it is an all knowing oracle of truth that is perpetually beyond the current capabilities. This is useful for convincing VCs that you need more funding, and fear mongering the government into regulating away competition. A simple and reasonable alternative conclusion is that AGI has been here for years, and that reality just isn't quite as exciting as sci-fi.
Will AGI capabilities increase? Sure, as we build out more tools for AGI to reach for, and as the intelligent agents themselves mature. Fundamentally, it is here.
Uhm. If you knew how to build AGI, what is your logical next step? Is this step in the interest of humanity?
I doubt this would happen. But can we rule it out 100%? We've become dependent on technology + networked systems to a high degree. If that's messed with (large-scale, worldwide, many different systems simultaneously or in short order), can we still 'unplug'? (those AGI systems, or ourselves - take your pick)
For the naysayers: some possibilities:
# AGI systems co-operating. Or taking over other systems to further their goals.
# Discovering ways to erase (or corrupt / subtly modify) most data stored in datacenters, and most backups too.
# Exploiting 0-days to do similar bad stuff to PC's, smartphones, etc. Remember most such devices are always-connected these days & employ automatic updates.
# Mess with critical infrastructure like power grids, logistics chains, public transport / flight control systems, etc. Or plunge stock markets into chaos.
# Develop a deadly biological weapon, have that synthesized somewhere, and cause it to be released.
# Mess with social media & news networks, to send humans into mass hysteria (or blissfully unaware what's about to hit them).
Granted, such a "rise of the machines" scenario sounds pretty wild. But "99.999% certain this won't happen", doesn't cut it imho. A 100% safety guarantee is needed here.
- Aladdin (1992)
