I see less "painting as a luddite" in response to statements like this, and more... surprise. Mild skepticism, perhaps!

Your experience diverges from that of other experienced devs who have used the same tools, on probably similar projects, and reached different conclusions.

That includes me, for what it's worth. I'm a graybeard whose current work is primarily cloud data pipelines that end in fullstack web. Like most devs who have fully embraced LLMs, I don't think they are a magical panacea. But I've found many cases where they're unquestionably an accelerant -- more than enough to justify the cost.

I don't mean to say your conclusions are wrong. There seems to be a bimodal distribution amongst devs. I suspect there's something about _how_ these tools are used by each dev, and in the specific circumstances/codebases/social contexts, that leads to quite different outcomes. I would love to read a better investigation of this.

Yesterday i wanted to understand what a team was doing in a go project. I have never really touched go before. I do understand software, because I develop for plus 20 years. But chatgpt was perfectly able to give me a summary on how the implementation worked. Gave me examples and suggestions. And within a day fulltime pasting code and asking question i had a good understanding of the codebase. It would have be a lot more difficult with only google.

Totally respect your position, given that you actually tried the tool and found it didn't work for you. That said, one valid explanation is that the tool isn't good for what you're trying to achieve. But an alternative explanation is that you haven't learned how to use the tool effectively.

You seem open to this possibility, since you ask:

> I've never actually seen any LLM-driven developers work in real time. Are there any live coding channels that could convince the skeptics what we're missing out on something revolutionary?

I don't know many yet, but Steve Yegge, a fairly famous developer in his own right, has been talking about this for the last few months, and has walked a few people through his "Chat Oriented Programming" (CHOP) ideas. I believe if you search for that phrase, you'll find a few videos, some from him and some from others. Can't guarantee they're all quality videos, though anything Steve himself does is interesting, IMO.

I have very similar experience. For me LLM are good at explaining someone else's complex code, but for some reason they don't help me write new code well. I would also like to see any LLM-driven developers work in real time.

You're the middle ground I was talking about. You tried it. You know where it works and where it doesn't.

I've used LLM to generate code samples and my IDE (IntelliJ) uses an LLM for auto-suggestions. That's mostly about it for me.

I have, in person. Their code requires a lot of cleanup and then there's the AI pit of death they often are not able to crawl out of due to... Mostly language differences. They don't know enough English to look stuff up and figure out how to fix things. Programming resources in other languages are pretty much non-existent

My experience thus far is that LLMs can be quite good at:

* Information lookup

-- when search engines are enshittified and bogged down by SEO spam and when it's difficult to transform a natural language request into a genuinely unique set of search keywords

-- Search-enabled LLMs have the most up to date reach in these circumstances but even static LLMs can work in a pinch when you're searching for info that's probably well represented in their training set before their knowledge cutoff

* Creatively exploring a vaguely defined problem space

-- Especially when one's own head feels like it's too full of lead to think of anything novel

-- Watch out to make sure the wording of your request doesn't bend the LLM too far into a stale direction. For example naming an example can make them tunnel vision onto that example vs considering alternatives to it.

* Pretending to be Stack Exchange

-- EG, the types of questions one might pose on SE one can pose to an LLM and get instant answers, with less criticism for having asked the question in the first place (though Claude is apparently not above gently checking in if one is encountering an X Y problem) and often the LLM's hallucination rate is no worse than that of other SE users

* Shortcut into documentation for tools with either thin or difficult to navigate docs

-- While one must always fact-check the LLM, doing so is usually quicker in this instance than fishing online for which facts to even check

-- This is most effective for tools where tons of people do seem to already know how the tool works (vs tools nobody has ever heard of) but it's just not clear how they learned that.

* Working examples to ice-break a start of project

* Simple automation scripts with few moving parts, especially when one is particular about the goal and the constraints

-- Online one might find example scripts that almost meet your needs but always fail to meet them in some fashion that's irritating to figure out how to coral back into your problem domain

-- LLMs have deep experience with tools and with short snippets of coherent code, so their success rate on utility scripts are much higher than on "portions of complex larger projects".

Fallacy fallacy

The middle ground between hyping the new tech and being completely skeptical about it is usually right. New tech is usually not everything it's hyped up to be, but also usually not completely useless or bad for society. It's likely we're not about to usher in the singularity or doom society, but LLMs are useful enough to stick around in various tools. Also it's probably the case that a percentage of they hype is driven by wanting funding.

Yup! I don't have a lot to say about LLMs for coding. There are places where I'm certain they're useful and that's where I use them. I don't think "generate a react app from scratch" helps me, but things like "take a CPU profile and write it to /tmp/pprof.out" have worked well. I know how to do the latter, but would need to look at the docs for the exact function name to call, and the LLM just knows and checks the error on opening the file and all that tedium. It's helpful.

At my last job I spent a lot of time on cleanups and refactoring and never got the LLM to help me in any way. This is the thing that I try every few months and see what's changed, because one day it will be able to do the tedious things I need to get done and spare me the tedium.

Something I should try again is having the LLM follow a spec and see how it does. A long time ago I wrote some code to handle HTTP conditional requests. I pasted the standard into my code, and wrote each chunk of code in the same order as the spec. I bet the LLM could just do that for me; not a lot of knowledge of code outside that file was required, so you don't need many tokens of context to get a good result. But alas the code is already written and works. Maybe if I tried doing that today the LLM would just paste in the code I already wrote and it was trained on ;)