NATE SILVER: POLITICAL POLLS THAT ARE NOT RANDOM. I posted here about Nick’s statistics class in which one half of the class was supposed to flip a coin 30 times and document the results. The other half was to write down a sequence of 30 heads or tails and try to make it look like it was the actual, random results of a coin flip. The teacher was able to identify the which was randomly generated because they had long streaks of heads or tails. Nick concluded: “The ones trying to make up the results wound up vastly underestimating variance.â€
Nate Silver, whose success at the Five Thirty Eight blog is partly due to his ability to interpret the predictions of political polls, had an article arguing that many of the polls for the 2014 elections were not based on random samples, but were the result of “herding” by pollsters. (The headline characterizes this as “putting a thumb on the scale”.) The telltale indicator of human influence is that as election day approached, the poll results converged too much. There should have been more results of sampling that seemed to be “outliers”, like the streaks of seven heads or tails in a row.
Silver says: “…sampling error is unavoidable — an intrinsic part of polling. If you’ve collected enough polls and don’t find that at least 32 percent of them deviate from the polling average by 3.5 percentage points, it means something funny — like herding — is going on.” The final poll results vastly underestimated variance.