undefined | Better HN

0 pointsdolebirchwood2mo ago0 comments

I accept there are productivity gains, but it's hard to take "10x" seriously. It's such a tired trope. Is no one humble enough to be a meager 2.5x engineer?

0 comments

signatoremo2mo ago

10x probably means “substantial gain”. There is no universal unit of gain.

However if the difference is between doing a project vs not doing is, then the gain is much more than 10x.

bpodgursky2mo ago

I don't know what to tell you, it's just true. I have done what was previously days of BI/SQL dredging and visualizing in 20 minutes. You can be shocked and skeptical but it doesn't make it not true.

isodev2mo ago

There is no x is because LLM performance is non deterministic. You get slop out at varying degrees of quality and so your job shifts from writing to debugging.

llmslave22mo ago

Even 2.5x is absurd. If they said 1.5x I might believe them.

OsrsNeedsf2P2mo ago

I'm building an AI agent for Godot, and in paid user testing we found the median speed up time to complete a variety of tasks[0] was 2x. This number was closer to 10x for less experienced engineers

[0] tasks included making games from scratch and resolving bugs we put into template projects. There's no perfect tasks to test on, but this seemed sufficient

nicoburns2mo ago

Have you evaluated the maintainability of the generated code? Becuause that could of course start to count in the negative direction over time.

Some of the AI generated I've seen has been decent quality, but almost all of it is much more verbose or just greater in quantity than hand written code is/would be. And that's almost always what you don't want for maintenance...

teaearlgraycold2mo ago

One concern is those less experienced engineers might never become experienced if they’re using AI from the start. Not that everyone needs to be good at coding. But I wonder what new grads are like these days. I suspect few people can fight the temptation to make their lives a little easier and skip learning some lessons.

llmslave22mo ago

That sounds reasonable to me. AI is best at generating super basic and common code, it will have plenty of training on game templates and simple games.

Obviously you cannot generalize that to all software development though.

2 more replies

thornewolf2mo ago

I estimated that i was 1.2x when we only had tab completion models. 1.5x would be too modest. I've done plenty of ~6-8 hour tasks in ~1-2 hours using llms.

enraged_camel2mo ago

Indeed. I just did a 4-6 month refactor + migration project in less than 3 weeks.

kmoser2mo ago

I recently used AI to help build the majority of a small project (database-driven website with search and admin capabilities) and I'd confidently say I was able to build it 3 to 5 times faster with AI. For context, I'm an experienced developer and know how to tweak the AI code when it's wonky and the AI can't be coerced into fixing its mistakes.

kmoser2mo ago

Downvoted for...confidently saying how successful I was using an AI? I don't get it.

llmslave22mo ago

What's the link?

1 more reply

j / k navigate · click thread line to collapse

0 comments

signatoremo2mo ago

10x probably means “substantial gain”. There is no universal unit of gain.

However if the difference is between doing a project vs not doing is, then the gain is much more than 10x.

bpodgursky2mo ago

I don't know what to tell you, it's just true. I have done what was previously days of BI/SQL dredging and visualizing in 20 minutes. You can be shocked and skeptical but it doesn't make it not true.

isodev2mo ago

There is no x is because LLM performance is non deterministic. You get slop out at varying degrees of quality and so your job shifts from writing to debugging.

llmslave22mo ago

Even 2.5x is absurd. If they said 1.5x I might believe them.

OsrsNeedsf2P2mo ago

I'm building an AI agent for Godot, and in paid user testing we found the median speed up time to complete a variety of tasks[0] was 2x. This number was closer to 10x for less experienced engineers

[0] tasks included making games from scratch and resolving bugs we put into template projects. There's no perfect tasks to test on, but this seemed sufficient

nicoburns2mo ago

Have you evaluated the maintainability of the generated code? Becuause that could of course start to count in the negative direction over time.

teaearlgraycold2mo ago

llmslave22mo ago

That sounds reasonable to me. AI is best at generating super basic and common code, it will have plenty of training on game templates and simple games.

Obviously you cannot generalize that to all software development though.

2 more replies

thornewolf2mo ago

I estimated that i was 1.2x when we only had tab completion models. 1.5x would be too modest. I've done plenty of ~6-8 hour tasks in ~1-2 hours using llms.

enraged_camel2mo ago

Indeed. I just did a 4-6 month refactor + migration project in less than 3 weeks.

kmoser2mo ago

Downvoted for...confidently saying how successful I was using an AI? I don't get it.

llmslave22mo ago

What's the link?

1 more reply

j / k navigate · click thread line to collapse