undefined | Better HN

0 pointsgspencley11mo ago0 comments

> "generate unit tests" is interesting? I have literally never written a unit test before and the code it produces is lengthy. Unit tests are a waste of time; but others disagree and so AI can produce it.

This statement worries me for a number of reasons.

First, I work on a very massive codebase, with a large engineering organization. And I have seen a number of AI generated unit tests. I have not seen a single example of an LLM-generated unit test that didn't contain a number of test anti-patterns. To the extent where I would argue that they are creating a self-fulfilling prophecy. You said you think that unit tests are a waste of time. I would argue that they CAN be even worse than that.

The benefit of unit tests is that, at their best, they give you a safety net for refactoring existing code. If you change the implementation of a system under test, and the tests pass, you know you didn't introduce a breaking change.

But a lot of tests couple themselves tightly to the implementation details. Every single LLM-generated unit test I have observed in the wild introduces this anti-pattern. If you have a system under test, and changing the implementation of that system without breaking its behaviour causes a test to fail... that's called the "Fragile Test" problem. Now your unit test is not only failing to encourage you to refactor code... it's actively DISCOURAGING you from refactoring code. In this case, the unit test is providing DISVALUE rather than value.

So the fact that a) you think unit tests are a waste of time and b) you look at AI as a way to save you from a chore ... tells me that you have no business ever writing uint tests with or without AI. Please stop. You are making the world worse by leveraging an LLM to do these things for you.

I have NEVER looked at writing a unit test by hand as a "chore" or as a "waste of time." I often write my tests before even writing my implementation code, because doing so helps me think through both the design and requirements of my code... and gives me a little mini-sandbox context where I can make sure that the brand new code that I am writing is doing what I want it to. It's a problem solving tool. Not something to be done after the fact as a chore.

"Write Documentation" is not writing code. And if you don't read the documentation you're generating, no one else will. So what's the point of having it in the first place if no one reads it? Food for thought.

"Find problems" I see as being akin to a spell check, or the red squigglies when you have a syntax error. I do see the value of having ML tech within the internal IDE implementation. We've been using ML for email SPAM filters for decades. There are interesting problems that can be solved there. But this is an area where I want my IDE to flag something for me. It's not me actively prompting an LLM to generate code for me.

0 comments

incomingpain11mo ago

>First, I work on a very massive codebase, with a large engineering organization.

I sure dont. I'm a cyber security 0day writer or linux bash scripter or yikes coder, who has made some bad turns and ended up with some open source projects in python that absolutely nobody would call large or massive.

>The benefit of unit tests is that, at their best, they give you a safety net for refactoring existing code. If you change the implementation of a system under test, and the tests pass, you know you didn't introduce a breaking change.

Never refactor. This is literally why everyone loves objected orientated and paint themselves in a corner that blows up in their face.

>Now your unit test is not only failing to encourage you to refactor code... it's actively DISCOURAGING you from refactoring code. In this case, the unit test is providing DISVALUE rather than value.

That sounds perfect to me. Never refactor, you end of life or version that old code out, and replace it with the new perfect code. We have WPA #3, TLS 1.3, SSH 2, IKE v2, HTTP 3.

It's fine to replace until you get it right.

>So the fact that a) you think unit tests are a waste of time and b) you look at AI as a way to save you from a chore ... tells me that you have no business ever writing uint tests with or without AI. Please stop. You are making the world worse by leveraging an LLM to do these things for you.

We live in very different worlds. I do like the idea that you feel like im making the world worse by having AI do the chores.

>I have NEVER looked at writing a unit test by hand as a "chore" or as a "waste of time."

Me neither. I just dont do them. with how awesome AI has been... im probably going to have the time to read for the first time: https://docs.djangoproject.com/en/5.2/topics/testing/

>I often write my tests before even writing my implementation code, because doing so helps me think through both the design and requirements of my code... and gives me a little mini-sandbox context where I can make sure that the brand new code that I am writing is doing what I want it to. It's a problem solving tool. Not something to be done after the fact as a chore.

I dont code anything so complicated. I come from linux, but they call it the 'unix philosophy'. Do one thing and do it well. Kind of akin to KISS principle.

Write clean simple code that self-documents itself that legit cant really go wrong. Especially if some jerk comes along like me who throws a fuzzer at it or worse user input.

>"Write Documentation" is not writing code. And if you don't read the documentation you're generating, no one else will. So what's the point of having it in the first place if no one reads it? Food for thought.

Good point.

>"Find problems" I see as being akin to a spell check, or the red squigglies when you have a syntax error.

Community pycharm has all that for free. The "find problems" is entirely next level. I'm so very sold.

I just wrote a stupid def test(): which returns a string. The linter has not a single thing to say.

1. *Try Block Contains No Risk Operations*: - The try-except block is wrapping a simple string operation that won't raise exceptions - Makes the try-except block unnecessary

>I do see the value of having ML tech within the internal IDE implementation. We've been using ML for email SPAM filters for decades. There are interesting problems that can be solved there. But this is an area where I want my IDE to flag something for me. It's not me actively prompting an LLM to generate code for me.

Since im public open source project on github I got copilot for free like 3 years ago and my biggest complaint was that it was generating garbage code for me and forcing it. It got so annoying i disabled it and stopped using it. Im sure it has improved since then.

Pycharm's AI on occasion has a tab completion thats really good, but isnt forcing.

gspencleyOP11mo ago

> Never refactor. This is literally why everyone loves objected orientated and paint themselves in a corner that blows up in their face.

I try to get people to think about "write once" code as much as possible. Keep things small, single-purpose and composable. You maximize reuse potential, make testing a lot easier and greatly ease debugging.

But you're right about one thing: we do live in very different worlds. In the product world, the business never knows what they are making until something is in front of them and they decide they want something else. Therefore the majority of developer time is spent "maintaining" existing code (read: making code changes) vs building new code.

In that world, the "never refactor" mentality is suicide. Refactoring is improving the design of existing code in order to make it simpler and easier to change as requirements change.

And that is the one universal truth of software: it changes.

If you don't need the code to change, ever, then you might as well stick with a fixed circuit.

> I dont code anything so complicated. I come from linux, but they call it the 'unix philosophy'. Do one thing and do it well. Kind of akin to KISS principle.

So do I. I use Linux on all of my devices and came up as a *nix developer. Unit tests favour the unix philosophy. I often think of a "unit" of code as a single logical operation. But logic is a function of the human mind and humans make errors. Even in simple, one-liner, single-responsibility functions logic errors creep up. Then consider how your programming language helps you or aids you. I see type casting induced errors all the time that a simple unit test would have caught and avoided.

> Write clean simple code that self-documents itself that legit cant really go wrong.

Then why did you even bring up writing documentation lol. We're aligned here. I never document code because I consider that to be a failure of the code. Documentation and code need to be kept in sync... which never happens, not to mention people don't read it. So favour code that is self-explanatory.

> I do like the idea that you feel like im making the world worse by having AI do the chores.

You're deliberately straw-manning me there. If writing a unit test is a chore, your approach to testing is flawed. That was my point and you know it. My comment was also specific to using AI to write unit tests. Not to do anything else.

j / k navigate · click thread line to collapse