That's indeed the second worst issue with current model architectures. For a model to be trained to something nearing usability for an actual task it needs an amount of data that is far beyond what can be obtained. Companies like Facebook and OpenAI downloaded pirated copies of every single book humans have written to reach the current level of text generation, and even with that, it's not like those models are perfect or that intelligent.

It is going to severely limit the possibilities of building actual agentic AIs. We do not have an endless amount of data of humans performing menial chores. And normal people will probably more hostile than the kool aid drinking software developers when it comes to being spied on, who's going to agree to wear a camera while working so as to help train their own replacement? Yet it's kinda what devs are doing gleefully adopting software filled with telemetry and interacting with copilot.