In the case you linked, it wasn't infringement for Robin Thicke to be impressed by Marvin Gaye. It was infringement to create a substantial copy.

Applying this same legal doctrine to LLMs, it's totally fine for them to train on copyrighted works. A problem would only arise if they produced substantial copies of those works.

It may be difficult to quantify the harm done in that case. Robin Thicke sold millions of copies of blurred lines. Each LLM output is generally consumed by a single person. There could be millions of examples of blatant copyright infringement by LLMs that will never come to light in court. There also could be vanishingly few. We may need new regulations to police it.

But that case involves publishing of content. If you want to compare them, it seems like you'd argue that the content of Pharrell Williams' head is somehow copyright infringement. I've yet to see anyone credible say that an AI model outputting exact content and then someone publishing that should be allowed. If you manage to make an AI model output copyrighted content, you can't then claim it's not copyrighted. If you sit on one side of a table and read a book out loud and I write it down (with minor transcription errors), that content is almost certainly still copyrighted content and I could not distribute it legally.