undefined | Better HN

0 pointsMrJohz2y ago0 comments

I work with code, which is about as reproducible as it is possible to get - the artifacts I produce are literally just instructions on how to reproduce the work I've done again, and again, and again. And still people come to me with some bug that they've experienced on their machine, that I cannot reproduce on my machine, despite the two environments being as identical as I can possibly make them.

I agree that reproduction in scientific work is important, but it is also apparently impossible in the best possible circumstances. When dealing with physical materials, inexact measurements, margins of error, etc, I think we have to accept that there is no set of instructions that, if followed to a T, will ever ensure perfect replication.

0 comments

3 comments · 1 top-level

johnnyworker2y ago· 2 in thread

> And still people come to me with some bug that they've experienced on their machine, that I cannot reproduce on my machine

But this is the other way around. Have you ever written a program that doesn't run anywhere except a single machine of yours? Would you release it and advertise it and encourage other people to use it as dependency in their software?

If it only runs on one machine of yours, you don't even know if your code is doing something, or something else in the machine/OS. Or in terms of science, whether the research says something about the world, or just about the research setup.

MrJohzOP2y ago

I think you misunderstand the point of scientific publication here (at least in theory, perhaps less so in practice). The purpose of a paper is typically to say "I have achieved these results in this environment (as far as I can tell)", and encourages reproduction. But the original result is useful in its own right - it tells us that there may be something worth exploring. Yes, it may just be a measurement error (I remember the magic faster than light neutrinos), but if it is exciting enough, and lots of eyes end up looking, then flaws are typically found fairly quickly.

And yes, there are often overly excited press releases that accompany it - the "advertise it and encourage others to us it as a dependency" part of it analogy - but this is typically just noise in the context of scientific research. If that is your main problem with scientific publishing, you may want to be more critical of science journalism instead.

Fwiw, yes of course I've written code that only runs on my machine. I imagine everyone has, typically accidentally. You do it, you realise your mistake, you learn something from it. Which is exactly what we expect from scientific papers that can't be reproduced.

johnnyworker2y ago

> But the original result is useful in its own right - it tells us that there may be something worth exploring.

I disagree. It shows that when someone writes something in a text editor and publishes it, others can read the words they wrote. That's all it shows, by itself. Just like someone writing something on the web only tells us that a textarea accepts just about any input.

And even if it did show more than that, when someone "explores" it, is the result is more of that, something that might be true, might not be, but "is worth exploring"? Then at what point does falsifiability enter into it? Why not right away? To me it's just another variation of making it someone else's problem, kicking the can down the road.

> if it is exciting enough, and lots of eyes end up looking, then flaws are typically found fairly quickly.

If that was true, there wouldn't even be a replication issue, much less a replication crisis. It's like saying open source means a lot of people look at the code, if it's important enough. Time and time again that's proven wrong, e.g. https://www.zdnet.com/article/open-source-software-security-...

> yes of course I've written code that only runs on my machine. I imagine everyone has

I wouldn't even know how to go about doing that. Can you post something that only runs on one of your machines, and you don't know why? Note I didn't say your machine, I said one machine of yours. Would you publish something that runs on one machine of yours but not a single other one, other than to ask "can anyone tell me why this only runs on this machine"? I doubt it.

2 more replies

j / k navigate · click thread line to collapse