THE PLAYOFFS AS AN EXAMPLE OF SMALL SAMPLES. It takes 162 games to determine the teams that make the major league baseball playoffs. The World Series is a maximum of 7 games.
Nick had a good analysis of how the playoffs and World Series illustrate the problems of small samples in this post on the Catbird Seat blog. Here is an excerpt from Nick’s post:
“What does it mean to say, ‘Small sample size?’
“I’ve only ever taken one formal statistics class, and it was over 10 years ago, but I still remember the first lesson very vividly. Our teacher divided the class into two groups. One half of the class was supposed to flip a coin 30 times and document the results. The other half was to write down a sequence of 30 heads or tails and try to make it look like it was the actual, random results of a coin flip.
“Our teacher guessed which ones were real and which ones were fake with, if memory serves, 100% accuracy. The reason is that the people who actually flipped coins wound up with really long streaks of heads and tails – 7 in a row or more. Meanwhile, the ones who were guessing what it would look like had a lot more “H-T-H-T-H-T” back and forth sequences. The ones trying to make up the results wound up vastly underestimating variance.”