I think the middle ground where several of the OpenAI board members were trying for is to "responsibly develop AGI", which means developing at a moderate pace while trying to avoid kicking off an investment gold rush through making heavily commercial use cases, and spending a substantial amount of resources on promising safety research (such as Ilya's work).

In my opinion, it was not a very strong position because the allure of money and trying to be the biggest is too strong (as we're seeing now), but I think it was at least coherent.

I don't know whether I have "like-minded views", but I think the risk is there, and it is not fantastical to me because the AGI doesn't need to "do" anything outside of humans assisting it -- no killer robots or nanobots. It just needs to persuade humans to kill for it, or gather resources for it, and so on. And because it's an AGI, it has superhuman abilities for persuasion.

The intuition for GPT-4 being on the way towards superhuman intelligence or persuasion abilities is that the only significant difference between it and GPT-3 appears to be the amount of compute power applied to training.

The previous US President was much less than superhuman, and he still managed to seriously threaten democracy in the US, and by extension cause a minor existential risk to civilization. Imagine what a more intelligent agent with enough attention to be personally persuasive instead of generically persuasive could achieve for itself, given a goal. (If you think that LLMs can't express goals or preferences, read some of the Bing Sydney transcripts.)

I feel like the “Terminator Judgement Day” scenario is a strawman more often than not. There are reasons to push for the safe and responsible development of AI that aren’t based on the belief that AI is literally going to launch the nukes.

The biggest risks I see from AI over the next 2 decades are:

1. A complete devolution in what is true (because creating convincing fakes will be trivial). From politics to the way that people interact with each other on a daily basis, this seems like it has the potential to sew division

2. the societal turmoil/upheaval arising from the mass unemployment (and subsequent unemployability) of millions of people without the social safety nets in place to handle that. When people are suddenly unable to work and support their families and those who have consolidated the power have no interest in sharing the wealth, there will inevitably be pushback.

In his recent Ted talk, Ilya talks about AI as 'brains in a computer', and explains why he believes they aren't that far off from AGI. [0] GPT4 is a substantially worse software engineer than I am, but it is 10x GPT3. And if we 10x for at most 2 or 3 iterations, it could be better than me. While it is unclear that we can get there, it doesn't seem implausible to me, and would have deep implications for human existence. That is already true if it just stayed at that level, but what most doomers are really worried about is the possibility of GPT8 being as good or a better AI researcher than most or all humans.

[0] https://www.youtube.com/watch?v=SEkGLj0bwAU

No idea if the concerns are realistic, but dismissing them because they are fantastical surely is a fallacy, no?

We currently almost have scifi-level AI we can talk to on our smartphones, which would have been fantastical only a few years ago.

> fill in the blanks

What exactly are you looking for here, proof of concept? This stuff is inherently speculative - because if it wasn't speculative, we'd probably be dead.

I have a feeling EA[1] is like anarcho capitalist communities (or just inverse for socialism) where you have a good idea like markets==good then apply it like a hammer to everything you see in life. Any hint of gov/statism is treated as the enemy, regardless of its IRL negative impact on markets or net freedom, or whatever positive end goal they envision.

OpenAI is (hypothetically) progressing to AGI, therefore bad. Whether GPT specifically is a half a foot on a 1000km journey or potentially insignificant, it doesn't matter to them as they've pre-selected any potential for progress as all bad.

[1] if you search "Helen Toner" on YouTube it's just a collection of videos of her talking at EA conferences. So she's a believer

No one has, no.