> It's difficult to fully describe, so let's just give up and use a deeply flawed benchmark? Why not try to develop benchmarks that actually work and tell us something useful instead?

Two reasons. First: in this sub-thread I'm focusing on employment issues due to AI, so consider the quote from above:

> At the point that we have an AI that's capable of every task that say a 110 IQ human is, including manipulating objects in the physical world, then basically everyone is unemployed unless they're cheaper than the AI.

IQ doesn't capture what machines do, but it does seem to capture a rough approximation of what humans do, so when the question is "can this thing cause humans to become economically irrelevant?", that's still a close approximation of the target to beat.

You just have to remember that as AI don't match human thinking, so an AI which is wildly superhuman at arithmetic or chess isn't necessarily able to tie its own shoelaces. The AI has to beat humans at everything (at least, everything economically relevant) at that IQ level for this result.

Second: Lots of people are in fact trying to develop new benchmarks.

This is a major research topic all by itself (as in "I could do a PhD in this"), and also a fast-moving topic (as in "…but if I tried to do a PhD, I'd be out of date before I've finished"). I'm not going to go down that rabbit hole in the space of a comment about the exact performance thresholds an AI has to reach to be economically disruptive.

For a concrete example of quite how fast-moving the topic is, here's a graph of how fast AI is now beating new benchmarks: https://ourworldindata.org/grapher/test-scores-ai-capabiliti...

More importantly, basically all of the IQ tests are in the training sets, so it's hard to know how the models would perform on similar tests not in the training set.