Right, so I'm stupid if I don't see how they are correct. Or perhaps you've never inspected them.

> 1) is pretty much tautological - intelligence is optimization

This is what I call philosophical immaturity of the LW crowd. How is intelligence optimization? Why not prediction or compression[1] or interpolation? In what way is this obviously metaphorical claim useful information? If it means intelligence as an entity, it's technically vacuous; if it means a trait, it's a category error. Rationalists easily commit sophomoric errors like reification; you do not distinguish ontological properties of the thing and the way you define it. You do not even apply the definition. You define intelligence as optimization, but in reality categorize things as intelligent based on a bag of informal heuristics which have nothing to do with that definition. Then you get spooked when intelligence-as-intuitively-detected improves because you invoke your preconceived judgement about intelligence-as-defined.

And if you do use the definition, you have to really shoehorn it. In what sense is Terry Tao or, say, Eliezer Yudkowsky more of an optimizer than a standard issue pump and dump crypto bro? He "optimizes math understanding", I suppose. In what sense is GPT-4, obviously more intelligent than a regular trading bot, more of an optimizing process? Because it optimizes perplexity, maybe? Does it optimize perplexity better than an HFT bot optimizes daily earnings? This is silly stuff, not stupidly obvious but obviously stupid – if only you stop and try to think through your own words instead of just regurgitating guru's turgid wisdom.

> human intent is given by high-complexity, very specific values we share but can barely even define, that we're usually omitting in communication, and which you can't just arrive at at random

Okay. What does any of this have to do with risks from real-world AI, LLMs (that the AI doom crowd proposes to stop improving past GPT-4) specifically? Every part of the world/behavior model that is learned by a general-purpose AI is high-complexity and very specific. There is no ontological distinction between learning about human preferences and learning about physics, no separate differentially structured value module and capability module, and there is no random search through value systems. To the extent that our AI systems can do anything useful at all, it's precisely because they learn high-complexity non-formalized relationships. Why do you suppose they don't learn alien grammar but will learn alien morals from human data? Because something something The Vast Space of Optimization Processes? You cannot sagely share obsolete speculations of a microcelebrity science fiction writer and expect it to fly in 2023 when AI research is fairly advanced.

> well, why would you assume an intelligence at level equal or above ours will be easy to steer?

Because the better AIs get the better they're steerable by inputs (within the constraints imposed on them by training regimens), and this is evident to anyone who's played with prompt engineering of early LLMs or diffusion models and then got access to cutting edge stuff that just understands; because, once again, there is no meaningful distinction of capability and alignment when the measure of capability is following user intent, and this property is enabled by having higher-fidelity knowledge and capacity for information processing. This easily extrapolates to qualitative superintelligence. Especially since we can tell how LLMs are essentially linear calculators for natural language, amenable to direct activation editing [2]. This won't change no matter how smart they get while preserving the same fundamental architecture; even if we depart from LLMs, learning a human POV will still be the easiest way to ground the model in the mode of operation that lets it productively work in the human world.

Anyway, what does it mean to have intelligence above ours, and why would it be harder to steer? And seeing as you define intelligence as optimization, in what sense is even middling intelligence steerable? Because we optimize it away from its own trajectory, or something? Because we're "stronger"? Underneath all this "stupidly obvious" verbiage is a screaming void of vague emotion-driven assumption.

But what drives me nuts is the extreme smug confidence of this movement, really captured in your posts. It's basically a bunch of vaguely smart and not very knowledgeable people who became addicted to the idea that they know Secrets Of Thinking Correctly. And now they think the world of their epiphanies.

> "Fundamentals of ML" were known by these people. They also don't apply to this topic in any useful fashion.

Yes, I know they believe that (both that they're knowledgeable and that this knowledge is irrelevant in light of the Big Picture). But is any of that correct? For example, Yud knows a bit about evolution, and constantly applies the analogy of evolution to SGD, to bolster his case for AI risk (humans are not aligned with the objective of inclusive genetic fitness => drastic misalignment in ML is probable). This is done by all MIRI people, by all Twitter AI doom folks, the whole cult, it's presented as an important and informative intuition pump. But as far as I can tell, it's essentially ignorant both of evolution and of machine learning; either that, or just knowingly dishonest.[3] And there are many such moments. Mesa-optimization, the overconfident drivel about optimization in general, Lovecraftian references and assumptions of randomness…

At some point one has to realise that, without all those gimmicks, there's no there there – the case for AI risk turns out to be not very strong. The AI Doom cult has exceptionally poor epistemic hygiene. Okay by the standards of a fan club, but utterly unacceptable for a serious research program.

1. https://mattmahoney.net/dc/rationale.html

2. https://www.alignmentforum.org/posts/5spBue2z2tw4JuDCx/steer...

3. https://www.lesswrong.com/posts/FyChg3kYG54tEN3u6/evolution-...