5% TO 20%—THE GOOD NEWS ABOUT “OBSERVATIONAL STUDIES”. I am encouraged by the estimates of the percentage of findings from statistical studies that can be replicated. The estimates of 5% to 20% for multiple regression studies are higher than I would have expected, and the figure of 80% for randomized controlled experiments is heartening as well. Observational studies are much cheaper than controlled trials (tens of thousands of dollars for a study versus millions of dollars for a randomized controlled trial.) Observational studies generally serve a different purpose—formulating hypotheses and checking leads rather than testing whether a proposition is true.
The “bad news” about both kinds of statistical studies comes about because the scientific process is slower and less accurate than people think. Problems develop because results of trials are almost always reported without a caveat. If people were aware that in general only 80% of controlled studies and 5% to 20% of observational studies can be replicated, finding these facts out would not be considered bad news.