What about pictures still containing watermarks? Regardless of the actual legality, this does not fit "certainly".
> The most analogous situation to what transformer models do is a person learning from experience and creating their own work _influenced_ by what they’ve observed
No, it is not. It is called machine "learning" so clearly that is a fly made out of butter. Maybe courts will agree, maybe they won't, but the analogy to human learning is quite strenuous at best.