SMALL SAMPLES. The issue that Professor Kahneman missed—that small samples can lead to results that can’t be replicated—is one that is discussed almost routinely in Sabermetric studies (“Moneyball”). Many is the time I have seen Nick or his colleagues hedge their hypotheses with an acknowledgment that the season is still young and of course there is a small sample problem.
Although replication studies are shining a light on sample sizes, there seems to have been a period when little attention was paid to the size of a sample. Journalists in particular often ignore the problem. Mary Jane and Annalisa on occasion do an imitation of me fuming: “Studies show….”
In addition to sample size, the size of the effect being studied affects the power of the study. Here is a discussion of the issue on a blog devoted to the importance of effect sizes.