Current LLMs are powerful because they are trained on the patterns of existing data.
Without training on real token patterns, what outputs would you even expect to see? It's an LLM. Literally, a model of language. A language is necessary, by definition, even if you have to conjure it within your context window.
You're asking something closer to: "if LLMs can handle ANY language, why not assembly or something low level, close to the metal as possible?" That's a valid question.
My initial response is two-fold:
1) because someone still needs to verify and debug the source code - so human readability still matters.
2) because semantics are king. LLMs work because they can suss out relationships between tokens. At the lowest level, hardware doesn't really operate on tokens - moving 8 bytes from one address to another carries no information about what those 8 bytes represent to the larger program. Higher-level languages allowed humans to form richer semantics, just as they allow LLMs to express richer semantics with fewer tokens.
