undefined | Better HN

story

0 pointsFrost1x1y ago0 comments

There’s a current trend of obsession with “data driven” or “evidence based” assessments. While measured data from reality is useful, it’s not without its own sets of flaws. Much data may not be representative or usefully representative of reality due to complexity of the situation (what we measure isn’t isolated or cant be easily linked, or our measurement process itself is flawed). The sort of pinnacle of relying on data assessments is the assumption of removing bias, which is often simply not true.

Not only is bias introduced from accidental collection flaws, it’s also often tampered with intentionally cherry picking data, choosing interesting data or in some cases flat out falsifying data. In addition, evidence based reasoning often suffers from there being a lack of evidence to make a decision from. Or in some cases some critical aspect surrounding the decision is very niche to the case so the data may not take that into account unless it’s highly tailored data (evidence based reasoning tends to focus on breadth of applicability because gathering evidence is a long and often expensive process).

There’s still a lot of place for using theory and reasoning in conjunction with or in absence of data. Things like experience, professional opinion, etc. Medicine should be no different in that regard to any profession. The key is of course to always strive for sound empirical evidence/data where possible, but to use sound documented reasoning and theory in its absence if you want the best objective results.

0 comments

exe341y ago

> Much data may not be representative or usefully representative of reality due to complexity of the situation

I've personally been on the receiving end of "the data we collected shows...x" (in a non-medical setting), but when I asked to have a look at it, it turned out that while this was true for a large part of the population sampled, there was a material difference between that population and a smaller population that can be clearly identified and for the latter, the data showed the exact opposite conclusion.

(think 100 men and 30 women, kind of scenario, except the difference wasn't gender, but job role).

mekoka1y ago

The data-driven human is a particularly annoying fallacy of our times. There seems to be a bias at play here. The ethos of the person that identifies as "rational" and whose trust in the data makes them seem smart. The result is this increased prevalence of people who think that ignoring the feedback of their subjective experience for the most mundane phenomenon, just because "where's the data", is somehow rational.

It's even more amazing to see this at play in domains that are directly tied with your well-being, where there's abundant anecdata, but people are holding out until very long and very expensive research have made a pronouncement. Like, thousands of people claim this particular herb is {$positive_claim}. I could try it for a week and see for myself, but I think I'll hold on to my 10$ and wait until research confirms this by the time I turn 90.

Another aspect of this fallacy is nicely highlighted by the aphorism Not everything that counts can be measured. Not everything measured counts.

j / k navigate · click thread line to collapse