undefined | Better HN

0 pointsbcrl4y ago0 comments

It's a class of error that would be caught by even the most basic testing. A better title for the article is that 5% of 666 Python repos have typos that demonstrate the code in them that is completely untested. It doesn't matter which language it is: untested code is untested code in any language.

0 comments

7 comments · 3 top-level

geofft4y ago· 2 in thread

The errors were usually in tests themselves. Are you arguing that tests need their own tests to test that they are testing the right thing? Usually I think people believe that tests do not need to be tested and should not be tested, i.e., that you measure "100% coverage" against non-test code alone.

samhw4y ago

I don't think anyone could disagree: you could never exceed 0% code coverage if your definition was recursive (i.e. included tests, tests-of-tests, tests-of-tests-of-tests, ...).

Cpoll4y ago

Only if you generate infinite tests, then your coverage approaches 0%. But 100% covered code + 0% covered tests = ~50% total coverage.

Also, the obvious solution is self-testing code. (Jokes aside, structures like code contracts attempt something like this).

rikatee4y ago· 2 in thread

unfortunately like 10% of the bugs were in the tests themselves. e.g., the sentry one https://codereviewdoctor.medium.com/5-of-666-python-repos-ha...

the tests are only as good as the code they're written with, and as good as the code review process they were merged under.

bcrlOP4y ago

One of the habits I have when writing kernel code is to intentionally break code in the kernel to verify that my test is checking what I think it's checking. That's because of a lesson I learned a long, long time ago after someone reviewed my code and caught a problem: when your code has security implications, you need to make sure the boundary conditions that your tests are supposed to cover actually get tested. Having implemented a number of syscalls exposted to untrusted userland over the years, this habit has saved my bacon several times and avoided CVEs.

wott4y ago

I believe that, whenever possible, tests should be written in a different language that the one used for the code under test (even better, in a dedicated, mostly declarative, testing language).

It avoids replicating the same category of errors in both the test and the code under test, especially when some calculation or some sub-tests generation is made in the test.

jollybean4y ago

"It's a class of error that would be caught by even the most basic testing. "

You could say that about anything and everything in software.

It's not acceptable that testing needs to be run for something the language should 100% accommodate.

The whole point of the language is to provide algorithmic clarity and avoid these things.

This isn't really an issue of 'trade offs' is just a bad feature of the language that should have been remedied more than a decade ago.

The lack of proper declaration of variables is even more absurd, there's only downside to that.

j / k navigate · click thread line to collapse