Pardon me. I did conflate the overall claim that it inherently violates copyright law with a more specific claim (not made) that it "is copying." Since copyright also enumerates the "making derivatives" rights as well as the "copy rights" I acknowledge you have in your argument more than the zero legs to stand on that i implied.
> They're packaging up a derivative work and selling it. Don't have to look hard to see examples that this is just as infringing as outright copying.
This is an interesting claim. It rests on the question of whether the model itself is a derivative work, or if it's a tool (or something between a tool and a trained person).
A photocopier can be used to reproduce ASOIAF and a word processor can be used to create a blatantly derivative work, but I assume we agree that that isn't the problem of Xerox or Microsoft. The derivative works produced with those technologies are the 'illegal' items, not the programs that were used to build them.
If I wrote my own GoT fanfiction, ripping off whole characters, names, and settings, and read my own stories in the privacy of my own home, am I breaking any copyright law? I don't think I would be. I would rightly get in hot water if I tried to sell them, and would probably rightly get in hot water even if I just posted them to Github for free given that I'm distributing the derivative works.
I think using AI tools to generate derivative works could place the user (Not OpenAI, etc) in rightful legal jeopardy if they distribute or sell those works -- on the other hand, if they are simply keeping them for their own personal enjoyment I think it's not that different than if they wrote them themselves. (I also think that rightsholders are acting a little paranoid with those concerns, as though anyone would seriously choose not to buy the latest book or movie or painting only because some poor AI knock-offs exist, but I acknowledge that has little bearing on whether some action is or is not legal.)