http://www.marginalrevolution.com/marginalrevolution/2005/09...
They run regressions on a data set, adding and subtracting independent variables until the t values and standard errors start looking good.
Then they construct the linear model, assume the Gauss-Markov assumptions and sometimes (though not always) try to explain the causal relationship between the variables.
This is obviously very wrong and nobody has any clue what the distribution of the least squares estimators to these models are. But I've seen plenty of examples of this, which is enough to void the results of the paper (even if the model they come up with is somewhat plausible).
Widespread use of data-mining software does make it much easier to do dodgy things on a wide scale.
This is another good reason to ignore the media hype for every new paper that comes out. (Besides the fact that journalists perform lossy compression on data.)
But it seems like it's how science is supposed to work: publish your results, see if others confirm your findings, because you might be wrong even if you seem to have done everything correctly and honestly to the best of your ability.
Regarding "if you seem to have done everything correctly", I don't think that any honest scientist can claim that his/her study had no limitation or flaw. I regularly review papers for big conferences and there is no such thing as a perfect paper/research project/study. It's more like a threshold: despite the issues, were the findings novel, relevant, and found through a rigorous process? Would the community learn anything valuable by reading this?
Articles like the one cited by OP are useful if they make scientists and normal "folks" realize the limitations of alpha values, they are useful if they make scientists reconsider some of their methods (and way of presenting findings), but they can be harmful if the readers conclude that most scientific findings are "false" and thus, that science is bogus because it cannot find the "truth". Science is rarely, if ever, about true and false, religion is.
P.S. I realize this answer was more about the article, and less about your reply, it's just that your reply prompted me to write something :-) Again, I agree with you!
Right, that's what I was getting at. The scientist might believe they've done everything right, after checking and re-checking their work, but be missing some flaw or limitation in their work, their model, whatever.
I recall hearing once of an experiment that couldn't be reproduced, and it turned out to be due to some chemical property of the entirely normal laboratory glassware that one lab had used. Switching to another manufacturer removed the problem. (I'm probably messing up the details, like the consequences of the chemical properties of the glass. But the gist is correct. Different manufacturer of glassware cleared up a problem that was unexpected.)
Hear. Hear. This, of course, was the thrust of Richard Feynman's famous Caltech commencement speech on "Cargo Cult Science."
http://www.lhup.edu/~DSIMANEK/cargocul.htm (adapted HTML text)
http://calteches.library.caltech.edu/51/2/CargoCult.pdf (PDF version as published by Caltech)
As Feynman said, "The first principle is that you must not fool yourself--and you are the easiest person to fool. So you have to be very careful about that."
"He zoomed in on 49 of the most highly regarded research findings in medicine over the previous 13 years, as judged by the science community’s two standard measures: the papers had appeared in the journals most widely cited in research articles, and the 49 articles themselves were the most widely cited articles in these journals. These were articles that helped lead to the widespread popularity of treatments such as the use of hormone-replacement therapy for menopausal women, vitamin E to reduce the risk of heart disease, coronary stents to ward off heart attacks, and daily low-dose aspirin to control blood pressure and prevent heart attacks and strokes. Ioannidis was putting his contentions to the test not against run-of-the-mill research, or even merely well-accepted research, but against the absolute tip of the research pyramid. Of the 49 articles, 45 claimed to have uncovered effective interventions. Thirty-four of these claims had been retested, and 14 of these, or 41 percent, had been convincingly shown to be wrong or significantly exaggerated. If between a third and a half of the most acclaimed research in medicine was proving untrustworthy, the scope and impact of the problem were undeniable. That article was published in the Journal of the American Medical Association."
Clearly these findings are false... or maybe not? Dammit. http://en.wikipedia.org/wiki/Liar_paradox
http://en.wikipedia.org/wiki/Wikipedia:MEDRS
for more on distinctions among differing kinds of publications on research.
But research is very weird indeed. The more conference/journal articles you read, the less you trust them. I mean, say a field accepts results alpha < 0.05. This means that 5% of everything shown is wrong.
Feel free to correct me if you have a better grasp of statistics find what I say to be wrong.
I think most scientists don't understand the meaning of the p-value. There was an interesting discussion last year in the statistical blog community on that question, with leading statisticians involved in it: http://radfordneal.wordpress.com/2009/03/07/does-coverage-ma...
Suppose your initial guessing is 50:50 and over some years you run 200 tests. 100 times the crop really does yield better and most of those show up fine. 100 times the crop doesn't actually yield better and 5% of those result in false positives. You end up with around 100 true positives and 5 false positives. A positive result really means something.
Fast forward 80 years and research has changed. You have high throughput screening machines and can test 100,000 different molecules in your hunt for a new antibiotic. Suppose you have got lucky and there really is a new antibiotic in your combinatorial explosion of side chains. A p-value of 5% gives you 5000 false positives. With any luck you don't get a false negative and your new antibiotic also makes it through the initial screen. Now you have 5001 +/- 70 positives. The probability that a positive result is true is only 0.0002 or 0.02%. A positive results still means something important. You are searching for a needle in a haystack and you have discarded 95% of the hay, but there is still plenty of hay left and the 99.98% of the results are wrong.
And then there are fields like climate science, where "very high confidence" means 10% probability of being wrong:
For those who need clarification, if this published research and its title are true, than it is saying that research like itself are usually false. This contradicts the original assumption that it is true.
If this published research and its title are false, than research like itself is usually true since what it's saying must be wrong. This contradicts the original assumption that it is false.
Of course he's talking about studies that use significance tests, which his own paper isn't directly using to prove his point.
