I agree with that. I didn't want to imply that you get intelligence by throwing enough compute at a gradient descent. It's multi-domain, cross-paradigm kind of optimization. But I find it worthwhile to deanthropomorphize intelligence, so that it's easier to appreciate how different can a powerful mind be from the ones we have.
> and I think there's a ton of stuff we don't understand about the operation of our own minds. It's yet another reason I don't think human-level or beyond AI is imminent. We barely understand how we think.
I also don't think AI is imminent - hell, we can't even get a decent webapp generator working, we have a lot of research in front of us to build self-improving systems. But I share the concern of the FAI crowd that we may eventually get there, and when we do, it's important to do it right the first time - otherwise, a runaway optimization process may not give us another chance. As you said, we barely understand how we ourselves think - so it's good to figure that one out before someone manages to build an actual AI.