Insanely bad compared to books or other permanent records. The human memory system did not evolve to be an accurate record of the past. It evolved to keep us alive by remembering dangerous things.

Chimpanzees have much better short term memories than humans do. If you test them with digits 1-9 sequentially flashed on a screen, they're able to reproduce the digits with lower loss than undergraduate human students.

https://link.springer.com/article/10.1007/s10071-008-0206-8

That's a very anthropocentric view. Technology isn't a series of deliberate inventions by us, but an autonomous, self-organizing process. The development of a spear, a bow, or a computer is an evolutionary step in a chain of technological solutions that use humans as their temporary biological medium. The human brain is not the starting point or center of this process. It is itself a product of biological evolution, a temporary information-processing system. Its limitations such as imperfect memory, are simply constraints of its biological origin. The tools we develop, from writing to digital storage are not just supplements to human ability, but the next stage in a system that is moving beyond its biological origins to find more efficient non-biological forms of information storage and processing. Human pride in creation is a misinterpretation. We are not the masters of technology. We're just the vehicle of it. Part of a larger process of technological self-improvement that is now moving towards an era where it might no longer require us

That is easily fixed, ask it to summarize it's learnings, store it somewhere, and make it searchable through vector indexes. An LLM is part of a bigger system that needs not just a model, but context and long term memory. Just like human needs to write things down.

LLMs are actually pretty good at creating knowledge: if you give it a trial and error feedback loop it can figure things out, and then summarize the learnings and store it in long term memory (markdown, RAG, etc).

Over time though, presumably LLM output is going into the training data of later LLMs. So in a way that's being consolidated into the long-term memory - not necessarily with positive results, but depending on how it's curated it might be.

Is this not a tool that could be readily implemented and refined?

my knowledge graph mcp disagrees

And LLM memories are stored in an electrical charge trapped in a floating gate transistor (or as magnetization of a ferromagnetic region on an alloy platter).

Or they write CLAUDE.md files. Whatever you want to call it.

It's not that either.

I don't believe this has been really proved yet.

I mean, not getting confused / forgetful is a pretty big one!

I think one thing it does is help you get rid of the UX where you have to manage a bunch of distinct chats. I think that pattern is not long for this world - current models are perfectly capable of realizing when the subject of a conversation has changed

Yeah to some degree that's already happened. Anecdotally I hear giving your whole iMessage history to Gemini results in pretty reasonable results, in terms of the AI understanding who the people in your life are (whether doing so is an overall good idea or not).

I think there is some degree of curation that remains necessary though, even if context windows are very large I think you will get poor results if you spew a bunch of junk into context. I think this curation is basically what people are referring to when they talk about Context Engineering.

I've got no evidence but vibes, but in the long run I think it's still going to be worth implementing curation / more deliberate recall. Partially because I think we'll ultimately land on on-device LLM's being the norm - I think that's going to have a major speed / privacy advantage. If I can make an application work smoothly with a smaller, on device model, that's going to be pretty compelling vs a large context window frontier model.

Of course, even in that scenario, maybe we get an on device model that has a big enough context window for none of this to matter!