A big problem with current AI is a simple one - we don't document our thought processes that go into building solutions, at least not in enough detail.

Without the thousands of micro decisions that go into building even the simplest of solutions, it doesn't matter how large your context window is. It's not about holding a system in your mind - it's about what you do with it, what decisions you make to move towards your goal.

At least that's my take on current LLMs and their limitations.