procha

Three years after ChatGPT’s release, LLMs are in everything—demos, strategies, and visions of AGI. But from my observer’s perspective, the assumptions we’re making about what LLMs can do seem to be drifting from architectural reality.

LLMs are amazing at unstructured information—synthesizing, summarizing, reasoning loosely across large corpora. But they are not built for deterministic workflows or structured multi-step logic. And many of today’s most hyped AI use cases are sold exactly like that.

Architecture Matters

We often conflate different AI paradigms:

    LLMs (Transformers): Predict token sequences based on context. Great with language, poor with state, goal-tracking, or structured tool execution.

    Symbolic AI / State Machines: Rigid logic, excellent for workflows—bad at fuzziness or ambiguity.

    Reinforcement Learning (RL): Optimizes behavior over time via feedback, good for planning and adaptation, harder to scale and train.

Each of these has a domain. The confusion arises when we treat one as universally applicable. Right now, we’re pushing LLMs into business-critical automation roles where deterministic control matters—and they often struggle.

Agentic Frameworks: A Workaround, Not a Solution

Agentic frameworks have become popular: LLMs coordinating with other LLMs in roles like planner, executor, supervisor. But in many cases, this is just masking a core limitation: tool calling and orchestration are brittle. When a single agent struggles to choose correctly from 5 tools, giving 10 tools to 2 agents doesn’t solve the problem it just moves the bottleneck.

Supervising a growing number of agents becomes exponentially harder, especially without persistent memory or shared state. At some point, these setups feel less like robust systems and more like committee members hallucinating their way through vague job descriptions.

The Demo Trap

A lot of what gets shown in product demos—“AI agents booking travel, updating CRMs, diagnosing errors”—doesn’t hold up in production. Tools get misused, calls fail, edge cases break flows. The issue isn’t that LLMs are bad it’s that language prediction is not a process engine.

If even humans struggle to execute complex logic reliably, expecting LLMs to replace structured automation is not vision it’s optimism bias.

On the Silence of Those Who Know Better

What’s most puzzling is the silence of those who could say this clearly: the lab founders, the highly respected researchers, the already-rich executives. These are people who know that LLMs aren’t general agents. They have nothing to lose by telling the truth and everything to gain by being remembered as honest stewards.

Instead, they mostly play along. The AGI narrative rolls forward. Caution is reframed as doubt. Realistic planning becomes an obstacle to growth.

I get it, markets, momentum, investor expectations. But still: it’s hard not to feel that something more ethical and lasting is being passed over in favor of short-term shine.

A Final Thought

I might be wrong—but it’s hard to ignore the widening gap between what LLMs are and what C-level execs and investors want them to be. Engineering teams are under pressure to deliver the Hollywood dream, but that dream often doesn’t materialize. Meanwhile, sunk costs pile up, and the clock keeps ticking. This isn’t pessimism it’s recognizing that hype has gravity, and reality has limits. I’d love to be proven wrong and happily jump on the beautiful AI hype train if it ever truly arrives.