Humans get things wrong too. A better question is: what error rate is acceptable for this task?

Jobs where higher error rates are acceptable, or where errors are easier to detect, will succumb to automation first. Art and poetry fit both of these criteria.

The claim is that as the model and training data sizes increase, these errors will get more and more rare.

We will see...

I am very optimistic about the far future. However, there will be a transition period where some jobs have been automated away but not others. There will be massive inequality between the remaining knowledge workers and manual laborers. If I was in a role on the early automation side of the spectrum then I would be retraining ASAP.

You are imagining that overnight we'll just use chatgpt to answer if a loan should be granted to a customer, and of course it can't do that reliably. But think turning that decision into steps that we can chip away at the problem. E.g.:

Step 1 will be to use chat gpt to get all of the loan inputs from documents, step 2 could be to identify any information that is missing that we should use to make the decision, step 3 will be making the decision. At each step well checks/balances and have human feedback. But don't kid yourself this is coming and the benefit for those that make the shift first are huge.

We are still very, very far away from having robotics overtake human dexterity. Even if AI can replace all knowledge workers, barbers, surgeons, and athletes will have a job for a long time.