> a lot of evidence that they're quite alien to us
It's an exact moral equivalent to learning. I wouldn't expect PyTorch to be running the citric acid cycle under the hood, and that shouldn't matter.
> Copilot reproducing `Q_rsqrt` verbatim, including the magic numbers, formatting and comments.
Humans can recite lots of things verbatim. Song lyrics, Shakespeare, digits of pi, Bible verses, you name it. We're so early, and the models are only one little engineering trick away from not doing this unless requested. We'll probably be able to tackle this at the level of the weights themselves.
> copyright law, which is designed to protect the human creative process only.
This is the point. Does AI contribute positively to the economy and the human condition? Does it outperform the old status quo? We're going to find out, and I'd expect for the law to fit to the outcome it views as optimal.
I'm betting that the law will view this similar to industrialization. That more people will be enabled through AI than inconvenienced, that more works will be created, and that more money will move and flow than before.
