SOME REASONS WHY THE REPRODUCIBILITY PROJECT WAS BOUND TO FIND SOME STUDY RESULTS THAT DON’T HOLD UP. Kids, there are good reasons to have expected that only a portion of research results would hold up when the study is replicated. In the article I linked to yesterday, Megan McArdle gave a couple of reasons why there would be some studies which would not stand up under replication.
She writes: “For starters, because in many cases, the sample sizes were larger. In general, the larger your sample, the weaker the effects you will find, because it’s harder for a few outliers to swamp the results.” Her second reason is “blind luck”—that is, the first study got its result because of randomness. If there are thousands of studies being done, and your test for publication is whether the result would occur less than 5% of the time, there will be hundreds of results that pass that test.
I would advance another reason for studies not being replicated successfully—the presence of other variables. I posted here about Richard Feynman’s description of how difficult it is to eliminate the influence of other variables.