I strongly believe this gets into an information theoretical constraint akin to why perpetual motion machines don't work.

In theory, yes you could generate an unlimited amount of data for the models, but how much of it is unique or valuable information? If you were to compress all this generated training data using a really good algorithm, how much actual information remains?