There is plenty of science out there which financially, practically, or ethically simply by definition cannot be replicated. That doesn't mean their results should not be published. If peer review shows that their methods and analysis are sound, there is no reason to doubt the results.
> Where are you going to get the budget to build a second LHC solely for replication?
In cases like this you could simply have a second, independent team time-sharing the LHC and using it to replicate experiments run by the first team. (And vice- versa). It’s not a perfect replication but it’s probably still an improvement over the “just trust me bro” status quo.
> How are you going to replicate a long-term medical cohort study which has been running for thirty years?
Run two independent studies in parallel from the beginning.
> What about a paper describing a one-off astronomical event, like the "Wow!" signal?
There was a ton of effort invested into trying to replicate that observation! Since nobody else ever managed to do so, we can’t draw any conclusions from it.
> What if you research the long-term impact of high-dose radiation exposure during Chernobyl?
That doesn’t preclude replication unless, for some reason, you’re the only researcher researching the long-term impact of high-dose radiation exposure during Chernobyl.
That's not a true replication, and it isn't going to avoid issues like the OPERA experiment measuring neutrinos going faster than the speed of light due to a loose connector. It would not be any different from having the second team just run their own analysis on the data from the first team - at which point the second team can just as well simply validate the first team's analysis like peer review is currently doing.
> Run two independent studies in parallel from the beginning.
So all currently-running long-running research has to be thrown out? What if the two studies find very small differences, are you allowed to publish either of them? Are the two teams allowed to collaborate at all?
> There was a ton of effort invested into trying to replicate that observation! Since nobody else ever managed to do so, we can’t draw any conclusions from it.
You can't "replicate" an observation of a freak astronomical event because you can't trigger a freak astronomical event. At best you can do observations and hope it happens again. We indeed cannot draw any conclusions from it, but that doesn't mean you can't publish papers about it. If replication is mandatory, you would not be allowed to do anything with it at all.
> That doesn’t preclude replication unless, for some reason, you’re the only researcher researching the long-term impact of high-dose radiation exposure during Chernobyl.
It cannot be reproduced because it would be unethical to expose people to near-fatal levels of radiation simply for reproduction. Simply reusing data from the original test subjects isn't a reproduction, after all.
As you already quoted me as saying, it's not a perfect replication. Which is fine! I'm advocating a position of "replicate findings as much as reasonably possible", not a position of "we need to build redundant copies of every multi-billion-dollar research megaproject". My whole point is that this doesn't need to be an absolutist true-or-false sort of thing.
> and it isn't going to avoid issues like the OPERA experiment measuring neutrinos going faster than the speed of light due to a loose connector
Maybe not. I never claimed this would solve every problem in all of science forever.
> So all currently-running long-running research has to be thrown out?
No. I think it's reasonable to propose more rigorous standards for future studies without throwing out every in-progress study that didn't follow those same standards. After all, there are literally centuries of published science that didn't even follow the contemporary standards of peer review, and we haven't thrown any of that out.
> What if the two studies find very small differences, are you allowed to publish either of them?
That's an extremely broad question. You might as well ask, "what does it mean for a finding to be replicated?".
If you and I each independently go out to measure the length of the Golden Gate Bridge in millimeters, there are likely to be very small differences in the result you get and the result I get. There's an expected margin of error here where we can agree that our results are consistent with each other. Sometimes the differences are reasonable and can be explained, and sometimes they can't be explained.
Regardless, I also think it might even be valuable for the studies to be published in some form even if they don't replicate at all; just not necessarily with the imprimatur of some credible or prestigious journal.
> Are the two teams allowed to collaborate at all?
I would suggest keeping the two teams independent at least until they both have results. Afterwards, it might be valuable for the team to collaborate in terms of trying to reconcile their results.
> You can't "replicate" an observation of a freak astronomical event because you can't trigger a freak astronomical event. At best you can do observations and hope it happens again. We indeed cannot draw any conclusions from it, but that doesn't mean you can't publish papers about it. If replication is mandatory, you would not be allowed to do anything with it at all.
Yeah, I guess I'm fine publishing a paper that just says "here's this anomalous observation we had" if it's an especially interesting anomalous observation like that.
> It cannot be reproduced because it would be unethical to expose people to near-fatal levels of radiation simply for reproduction.
Obviously. I think you know that's not what I'm suggesting at all here.
> Simply reusing data from the original test subjects isn't a reproduction, after all.
At this point, I agree we're mostly stuck with whatever data we managed to get 37 years ago. But if a similar incident happened in the future, you could have independent teams collecting redundant sets of data.
