undefined | Better HN

0 pointsdsp123410y ago0 comments

Imagine a function that is something like:

  function X(Y, Z){
    if(Y){
      foo();
    }
    if(Z){
      bar();
    }
  }

If you have tests like:

  X(true, false);
  X(false, true);

Then your code coverage from the "line" point of view is 100%. Those two tests execute every line of code. However, the two states:

  X(true, true);
  X(false, false);

have not been tested, and if foo() and bar() both manipulate some global state, or otherwise have side effects, then the system could still break.

This is an issue with many code coverage tools. The lines themselves have been executed, but not necessarily all program states.

0 comments

deckar0110y ago

Ideally foo and bar would be spies/mocks in that unit test, which means that X is 100% covered.

The advantage of unit testing modular code bases is that you don't have to test every program state, you just have to test every module state.

There is a huge difference in the feasibility of 100% code coverage if you need O(2^n) vs O(n) tests.

Even if your modules are highly coupled you should be able to mock away external dependencies and side-effects.

nimnio10y ago

"The advantage of unit testing modular code bases is that you don't have to test every program state, you just have to test every module state."

I don't think that's true with a capital T. Modules interact with each other. You can modularize, microize, or nanoize your application but the minimum subset of test-worthy states stays the same.

enqk10y ago

That is why I'd rather focus on a tool that would show lack of coverage and certainly would not even say percents of coverage, as it tricks people into getting that number to 100% and stop at that

dsp1234OP10y ago

a tool that would show lack of coverage

Except, as pointed out, there is no lack of coverage in my example. All lines are 'covered', and no line is left unexecuted. So by definition, there is no 'lack of coverage' from the perspective of lines. The key is to not think about 'coverage' as a metric on lines, and instead think of 'coverage' as a metric on program states. However, that's generally not what people think of when 'code coverage' is mentioned unless something specific like 'program space coverage' or 'code state coverage'.

Keep in mind that this can be really hard though since even very simple code can have a large state space.

Also, if you have a favorite tool that does program space coverage for node.js projects, it might be helpful to mention it here so that others can benefit. Most of the tools for node that I know are only line coverage.

enqk10y ago

I wasn't disagreeing!

My reasoning is as follows. Since evaluating coverage in terms of the state space of the program is hard, and no tools that I am aware of is actually calculating that coverage, tools that display coverage in terms of number of lines should not mislead by showing a percentage, and only focus on showing places with no coverage at all.

While in technical terms it seems like the same metric, I think it would encourage its users not to celebrate an arbitrary cap of 100% coverage, and be aware of what the tools can and cannot do.

j / k navigate · click thread line to collapse

0 pointsdsp123410y ago0 comments

Imagine a function that is something like:

  function X(Y, Z){
    if(Y){
      foo();
    }
    if(Z){
      bar();
    }
  }

If you have tests like:

  X(true, false);
  X(false, true);

Then your code coverage from the "line" point of view is 100%. Those two tests execute every line of code. However, the two states:

  X(true, true);
  X(false, false);

have not been tested, and if foo() and bar() both manipulate some global state, or otherwise have side effects, then the system could still break.

This is an issue with many code coverage tools. The lines themselves have been executed, but not necessarily all program states.

0 comments

deckar0110y ago

Ideally foo and bar would be spies/mocks in that unit test, which means that X is 100% covered.

The advantage of unit testing modular code bases is that you don't have to test every program state, you just have to test every module state.

There is a huge difference in the feasibility of 100% code coverage if you need O(2^n) vs O(n) tests.

Even if your modules are highly coupled you should be able to mock away external dependencies and side-effects.

nimnio10y ago

"The advantage of unit testing modular code bases is that you don't have to test every program state, you just have to test every module state."

I don't think that's true with a capital T. Modules interact with each other. You can modularize, microize, or nanoize your application but the minimum subset of test-worthy states stays the same.

enqk10y ago

That is why I'd rather focus on a tool that would show lack of coverage and certainly would not even say percents of coverage, as it tricks people into getting that number to 100% and stop at that

dsp1234OP10y ago

a tool that would show lack of coverage

Keep in mind that this can be really hard though since even very simple code can have a large state space.

enqk10y ago

I wasn't disagreeing!

While in technical terms it seems like the same metric, I think it would encourage its users not to celebrate an arbitrary cap of 100% coverage, and be aware of what the tools can and cannot do.

j / k navigate · click thread line to collapse