Interesting observation, I read a similar paper recently which reported LLMs experiencing degradation from compression. To me, these observations indicate that the neural net model isn’t sufficient for mimicking human learning or memory.
My pov is that computer intelligence can be better than human intelligence, but it needs to be programmed on flexible hardware, i.e., new connections don’t depend on software alone for formation and maintenance. But the current set of solutions we have for programmable hardware, as seen in FPGAs or ASICs, is also an incomplete solution because the interconnect isn’t flexible. I think we need wetware for training energy efficient and memory resilient AI. If AI is trained using quantum computers, then I don’t think the semiconductor medium used for training will be an issue, but that’s just a guess.
