There
is an internal memory, but it's very short lived, like a register in a traditional CPU architecture. It's like a CPU disconnected even from L1 cache, let alone RAM.
I like to think of consciousness as not the substrate, but the information flowing through it. Not the H2O molecules in a river, but the current. We're like eddies and swirls in mountain streams, our thoughts are the patterns of flow, not the unmoving rocks that set them up.
In this sense, when an LLM like GPT produces output, it's like a loop that has been cut, a single pass through what would be a circular process in a human brain. It can take a fixed number of steps from a standing start, but no more. This is because there is information flowing through it, transformed step-by-step. It just doesn't recirculate, it always exits after a fixed number of steps.
Think of an unrolled for loop that always exits. It takes a few steps, which look like a loop, but can never actually iterate.