In the end my concern comes down to that those who rise to power in our society are those who are best at playing the capitalist game. That's mostly, I guess, fine if what they're doing is being most efficient making cars or phones or grocery store chains or whatever.
Making intelligent machines? Colour me disturbed.
Let me ask you this re: "the domain is technical and cognitively demanding" -- do you think Sam Altman (or a Steve Jobs, Peter Thiel, etc.) would pass a software engineer technical interview at e.g. Google? (Not saying those interviews are perfect, they suck, but we'll use that as a gatekeeper for now.). I'm betting the answer is quite strongly "no."
So the selection criterion here is not the ability to perform technically. Unless we're redefining technical. Which leaves us with "intellectually demanding" and "smart", which, well, frankly also applies to lawyers, politicians, etc.
My worry is right now that the farther you go up at any of these organizations, the more the kind of intelligence and skills trends towards the "is good at manipulating and convincing others" kind of spectrum vs the "is good at manipulating and convincing machines" kind of spectrum. And it is into the former that we're concentrating more and more power.
(All that said, it does seem like Sutskever would definitely pass said interview, and he's likely much smarter than I am. But I remain unconvined that that kind of smarts is the kind of smarts that should be making governance-of-humanity decisions)
As terrible as politicians and various "abstract model" applying folks might be, at least they are nominally subject to being voted out of power.
Democracy isn't a great system for producing excellence.
But as a citizen I'll take it over a "meritocracy" which is almost always run by bullshitters.
What we need is accountability and legitimacy and the only way we've found to produce on a mass society level is through democratic institutions.
