Like you, I am a sucker for abstraction (often to my detriment). I just don’t see any resemblance between an LLM layer and abstraction, aside from that they both introduce complexity. I guess that abstraction superficially ”hides” its inner workings but this is a misnomer, it’s never actually hidden, it’s only claimed that you don’t need to look behind the curtain (but you can, and then you can see exactly what’s going on).

> I'm optimistic this problem will be resolved in the mid-term

Which one? Stability? Assume you can get open source models running locally and with no floating point non-determinism. Now you have some narrow sense local stability, yes. For exactly the same version of the LLM compiler and same input (all prompt history), you can reproduce the result. But this is not enough. If you roll a new version of the compiler (meaning new training data or algorithm), all previous responses are invalidated. So how would you maintain backwards compatibility?

With abstractions OTOH (like say C++), you don’t check in the generated assembly or binary. It’s enough to check in the C++ code, the rest is derived directly. When C++ releases their new foot-machinegun feature, you simply update your compiler and your old code works.

My point is that LLMs are not engineering products. They are closer to organic in the sense they act in the moment, and by necessity they are non-stable with respect to small variations in input and training. I heard this good quote from this researcher at anthropic who claimed we don't program the (neural) networks, we ”grow” them.

I remember reading that C compilers were buggy/generated lots of instructions. He may have a point back then