undefined | Better HN

0 pointsbrendoelfrendo5mo ago0 comments

> For instance, I had to rename a collection of files almost following a pattern. I know that there are apps that do this and normally I’d reach for the Perl-based rename script. But I do it so irregularly that I have to install it every time, figure out how I can do a dry run first, etc. Meanwhile, with the Raycast AI integration that also supports Finder, I did it in the 10-15 seconds that it took to type the prompt.

> On the other hand LLMs constantly mess up some algorithms and data structures, so I simply do not let LLMs touch certain code.

See, these two things seem at odds to me. I suppose it is, to a degree, knowledge that you can learn over time: that an LLM is suitable for renaming files but not for certain other tasks. But for me, I'd be really cautious about letting an AI rename a collection of files, to the point that the same restrictions apply as would apply to a script: I'd need to create the prompt, verify the output via a dry run or test run, modify as necessary, and ultimately let the AI loose and hope for the best.

Meanwhile, I probably have a script kicking around somewhere that will rename a batch of files, and I can modify it pretty quickly to match a new pattern, test it out, and be confident that it will do exactly what I expect it to do.

Is one of these paths faster than the other? I'm not sure; it's probably a wash. The AI would definitely be faster if I was confident I could trust it. But I'm not sure how I can cross that threshold in my mind and be confident that I can trust it.

0 comments

saxenaabhi5mo ago

> See, these two things seem at odds to me. I suppose it is, to a degree, knowledge that you can learn over time: that an LLM is suitable for renaming files but not for certain other tasks. But for me, I'd be really cautious about letting an AI rename a collection of files, to the point that the same restrictions apply as would apply to a script: I'd need to create the prompt, verify the output via a dry run or test run, modify as necessary, and ultimately let the AI loose and hope for the best.

Why? I never understand this level of caution since don't we all use VC? Just feed it the prompt and if it messes up undo the changes.

acuozzo5mo ago

> don't we all use VC?

This assumes you're working with text files.

What if you're working with ~100MiB (each!) frames from a scan of a 35mm movie?

(Note: This isn't fictional. I've worked with file-sets like this in film restoration many times.)

1 more reply

brendoelfrendoOP5mo ago

As another commenter suggested, this only works for some workflows. I'd also argue it kind of undermines the idea that an LLM can do this work better than a script.

gmadsen5mo ago

as part of the prompt, have a test suite with test files. Its still fully automated by the LLM but adds confidence

recursive5mo ago

If it's under the umbrella of LLM automation, then I'd also need to verify that the test suite behavior actually matches the "production" behavior.

gmadsen5mo ago

sure, but that is less work. you can also have separate LLM QA prompts that assess test suite behavior to production behavior.

ultimately you are right, the buck needs to stop somewhere, but at least in my experience, the more you add quality/test checks as LLM workflows, the higher the rate of success.

j / k navigate · click thread line to collapse

0 pointsbrendoelfrendo5mo ago0 comments

> On the other hand LLMs constantly mess up some algorithms and data structures, so I simply do not let LLMs touch certain code.

0 comments

saxenaabhi5mo ago

Why? I never understand this level of caution since don't we all use VC? Just feed it the prompt and if it messes up undo the changes.

acuozzo5mo ago

> don't we all use VC?

This assumes you're working with text files.

What if you're working with ~100MiB (each!) frames from a scan of a 35mm movie?

(Note: This isn't fictional. I've worked with file-sets like this in film restoration many times.)

1 more reply

brendoelfrendoOP5mo ago

As another commenter suggested, this only works for some workflows. I'd also argue it kind of undermines the idea that an LLM can do this work better than a script.

gmadsen5mo ago

as part of the prompt, have a test suite with test files. Its still fully automated by the LLM but adds confidence

recursive5mo ago

If it's under the umbrella of LLM automation, then I'd also need to verify that the test suite behavior actually matches the "production" behavior.

gmadsen5mo ago

sure, but that is less work. you can also have separate LLM QA prompts that assess test suite behavior to production behavior.

ultimately you are right, the buck needs to stop somewhere, but at least in my experience, the more you add quality/test checks as LLM workflows, the higher the rate of success.

j / k navigate · click thread line to collapse