undefined | Better HN

0 pointszmgsabst10mo ago0 comments

But reality doesn’t care:

If half your requests are 2x as long and half are 2x as fast, you don’t take the same wall time to run — you take longer.

Let’s say you have 20 requests, 10 of type A and 10 of type B. They originally both take 10 seconds, for 200 seconds total. You halve A and double B. Now it takes 50 + 200 = 250 seconds, or 12.5 on average.

This is a case where geometric mean deceives you - because the two really are asymmetric and “twice as fast” is worth less than “twice as slow”.

0 comments

ayhanfuat10mo ago

There is definitely no single magical number that can perfectly represent an entire set of numbers. There will always be some cases they are not representative enough. In the request example you are mostly interested in the total processing times so it does make sense you use a metric based on addition. But you could also frame a similar scenario where halving the processing time lets you handle twice as many items in the same duration. In that case a ratio-based or multiplicative view might be more appropriate.

zmgsabstOP10mo ago

Sure — but the arithmetic mean also captures that case: if you only halve the time, it also will report that change accurately.

What we’re handling is the case where you have split outcomes — and there the arithmetic and geometric mean disagree, so we can ask which better reflects reality.

I’m not saying the geometric mean is always wrong — but it is in this case.

A case where it makes sense is what happens when your stock halves in value then doubles in value?

In general, geometric mean is appropriate where effects are compounding (eg, two price changes to the same stock) but not when we’re combining (requests are handled differently). Two benchmarks is more combining (do task A then task B), rather than compounding.

j / k navigate · click thread line to collapse