You'd still have to describe to an LLM what to do, which strikes me as about equivalent complexity to simply writing the code in the first place (which is, after all, a formal description of how a program should behave, even if we perceive it as of a complexity which should not be strictly necessary). The big wins have been simply leveraging LLMs to apply common patterns to multiple codebases (albeit in a buggy and haphazard fashion), but it's still up to humans to compose these patterns into meaningful programs and validate that it's actually functioning as expected or desired. Humans still have far superior understanding of what software is, how it functions, what our intentions are using software, and how to derive good software from bad—we know what a bug is intuitively in a way that LLMs have not been able to demonstrate at all.

However, being able to rewrite a program with formally well-defined behavior (i.e. code) should be in an LLM's capability, but LLMs are a long ways away from demonstrating semantically coherent coding skills, just the ability to regurgitate common patterns (often filled with bugs and/or incoherent semantics).