IN PRAISE OF NON-RANDOMIZED DATA GATHERING. I have posted (for example, here) about the findings by Professor John Ioannidis that a surprising percentage of articles in scientific journals turn out to be wrong. Now David Freedman has an article in the Atlantic about Professor Ioannidis and his findings. (David Freedman is also the author of WHY EXPERTS KEEP FAILING US—AND HOW TO KNOW WHEN NOT TO TRUST THEM.) The article gives figures for the refutation by later studies of different kinds of research. For non-randomized studies, Professor Ioannidis estimates that 80% are later refuted; about 25% of “gold-standard” randomized trials are later refuted; and about 10% of “platinum-standard” large-scale randomized trials are later refuted.
I think the lesson to be drawn is that we should take these studies with a grain of salt. People tend to greatly overrate what is considered to be settled science. Yet medicine does advance and life spans grow longer. Medical and scientific knowledge proceeds by trial and error. Looking at the glass as half-full, the figures for studies which stand up (at least, so far) are 20% for non-randomized studies; 75% for gold-standard studies; and 90% for platinum-standard studies. It’s important for everybody to bear this kind of estimate in mind. Yet, doctors and patients are confronted with many decisions, Since gold-standard tests are expensive, there is no gold-standard answer for many questions, in which case non-randomized data would be the best available. I’d like to know the track record for taking different numbers of antibiotic pills if there is no gold-standard test to give a better answer to the question.