Im my experience, ChatGPT breaks down when it needs to consider more than 350 lines and its performance is sloppy before that.
To get solid performance out of it, I essentially need to specify the important areas and changes as well as the desired approach.
That being said, I’ve found it has cut my development time for most features by at least half, even in large codebase so I would be curious to know how you are approaching it currently.
- writing complex sql queries for throw-away code, i.e. need to create a few charts from big query - writing internal front end tooling - polishing any internal messages - helping design docs with clarity - helping unblock highly technical areas where i haven't had any experience - helping learn the sdk from new vendors - helping read api documentation, i.e. does the instagram api allow for ABC
areas where it's been mildly helpful:
- helping to write verbose methods that i'm too lazy to write. e.g. maybe a custom date parser that takes a datetime and outputs some custom text according to a spec
areas where it hasn't been that helpful
- building features e2e. as you mentioned, the context window issue is really the crux here. - building simple code like a crud api. our system, like all others, has its own idioms and spinning up a new crud api is easy enough. getting an llm to write it is do-able but by the time i've edited things to make it consistent with the rest of our system, i've lost the benefit
If you HAD to deal with 1,000 individual functions - let’s say due to hardware and organizational limitations - how would you manage the obvious risks? Would that be net-net cheaper than a fleet of microservices and development teams?
Or just ask the LLM's to write it for you.
This famous essay
https://worrydream.com/refs/Brooks_1986_-_No_Silver_Bullet.p...
points out why 10x is a pipe dream with any technology. The core of the argument is that software development involves a number of steps, let's say
* requirements gathering
* systems support (build/dependencies/version control)
* software architecture
* user interface design
* database design
* coding
* testing
* documentation
* user training
* deployment
[1] the argument wouldn't be too different if one of these was 30% of the work, and one certainly isn't
The obvious answer is that generating code is not the hard part of building products.
Because willingness to question one’s assumptions is good faith’s hallmark.
Which also means the OP should not be dogmatic that AI cannot speed features up 10x.
In any case changes to process don’t come for free. Faster features of the same quality may mean faster testing, faster communication with customers, faster analysis, faster marketing updates, etc.
bc hn is a community of smart ppl. I'm sure these discussions are happening at most software companies so I was just curious how others are dealing with it
> Have you actually tried using the tech? Then you should have the answers. There are lots of different types of software development and in some areas gen AI will be extremely useful, but a net negative in others. Only you can answer the questions about your environment.
yes, daily. :) here's my rough answer to my own question:
- as another commenter mentioned, context windows aren't large enough to capture every detail in our system - coding is actually the easy part of the job. Generally speaking, once we know the details of the task (i.e. building table X with N fields, build a service that does XYZ, etc), the coding portion doesn't take a lot of time. So the cost saving of outsourcing that to Claude/GPT/etc can be helpful, but it's not a game changer - team alignment is hard and figuring out what to build is hard - we lose eng time on the unknowns that gen ai doesn't really help with: debugging an issue with a vendor's sdk, identifying the cause of a race condition, figuring out how to mitigate a bot issue, etc.
1. AI is business jargon for Large Language Models.
2. LLMs are predictive models using plain text.
3. LLMs are not more accurate or creative than what they are fed.
4. LLMs can provide excellent documentation for prior encountered problems but cannot provide original solutions to new problems.
5. A more effective means at cost reduction is to reduce/eliminate regression. This provides the same benefit as LLMs but without sacrificing creativity.
6. This is an opportunity for executives to reduce expenses and dramatically and simultaneously radically increase product quality. Lead the developers to increase test automation coverage and increase execution speed.
7. The alternative for the developers is their replacement by LLMs. LLMs cannot replace people but cost so much less they make up the difference if product quality remains marginal or maintenance cost remains high.
That is how you do it. Small sentences, use numbers to explain the finances, and play devils advocate.
You still need to do code reviews, especially on the AI. It's quite bug free, more than human code, but you still need to explain the product. v0 doesn't quite replace a good product designer.
It's not yet at the level it can do architectural work and it doesn't understand the scope or goal of products. It doesn't understand roadmaps. It can't plan the code around where it needs to be in 1 year. You need a proper architect to do this better.
The average code it's trained on is from 2019 or so. New models has people write new data for it, but most of this data is not production data. So you're likely using an old design too and it tends to recommend these until encouraged otherwise.
Also if you're not using the newer tools like Cursor, Aider, Windsurf, a lot of the contribution of AI is better test coverage. The value that comes with the "agent" tools is they will write and edit code across multiple files, and they save you the trouble of explaining context when you can just share the source code.
Another commenter had a great note, that you should take this opportunity to advance your career, given you are in an excellent position to do so. It made me giggle, but then think that you really need to make sure your PoC was executed thoughtfully and seriously, that you understand what the SoTA is wrt using assistants and their various modalities (chat, agent, copilot etc) and despite your expertise it was a no go. Because if you don’t, somebody else who is taking it dead seriously is going to take that commenters advice and demonstrate the value that you didn’t, and this may reflect poorly on you. Execs are getting hype from their back channels or vendors like I have never seen in my career, and you are going to go against that. ($bigco perspective there)
Then I come to check out Ask HN and see this as top post.
If somebody else seems to be getting a 10x-speed up they got lucky for a simple problem, are lying (want to make it big as an AI influencer) or are delusional.
Could some product come out next year that's better? Maybe. Right now it's not a productive conversation to have to look for some "nuance" which will get you to 10x.
How did they come to this belief in the 10x AI developer? Get them to question their base assumptions, ask them to justify their expectations
Most AI coding tools can generate a Todo app from a small prompt. This is because that problem is well understood.
When you try to use AI coding tools on your own projects you need to start writing a prompt that teaches the AI about your current architecture and decisions.
So the initial prompt is large.
Often the task needs knowledge of other files in your project. You can add them by hand or some AI tools will search the code base.
The prompt is now huge.
When you run that prompt you may or may not get what you expected.
So now the issue is how much time do you spend getting the prompt correct vs just writing the code yourself.
This area is brand new and there are very few resources on how to use AI coding tools effectively.
I have yet to see one demonstration of effective AI coding tool use on a project of reasonable complexity.