SMALL SAMPLE PROBLEMS—AND MONEYBALL. The article by Jerry Adler reports that Joseph P. Simmons—who calls for much stricter standards for psychology experiments—had “at one point recommended requiring a minimum sample size of 20, but has since walked back from that notion”. Adler says that “because larger sample sizes increase the predictive power of results, Simmons now tries for at least 50 subjects in his own research.”
A sample of 20—or even 50—seems very small to draw conclusions from. In statistical articles in baseball, I often see the phrase “small sample”—and the samples are much larger than 50. Even reporters who reject or downplay statistics will note that you can’t draw conclusions based on a batter’s first 100 at bats. Today, on May 8th, fans take it for granted that there are a lot of hitters off to hot starts in their first 100 or so at bats who will “come down to earth” by the end of September when they will have 500 at bats.