HOW THE PREDICTION CONTEST WAS RUN. Angela Chen in an article on the Chronicle of Higher Education website describes the rules of the prediction contest which provided the evidence for Tetlock’s contention that there are some people who can predict successfully and his further contention that people can learn to make more successful predictions.
The competition, which was sponsored by a government agency called IARPA (the Intelligence Advanced Research Projects Activity), ran from 2011 to 2015. There were five teams competing plus a control team. Tetlock’s team was made up of members crowdsourced from around the world.
All the teams made predictions in response to question about future events. The questions were all precise enough to answer and relevant to national security. Chen gives two examples: What will be the highest reported monthly average of Mexican oil exports to the United States between February 5, 2014, and April 1, 2014? Will Angela Merkel win the next election for chancellor of Germany?
IARPA set a goal that the teams should beat the combined “wisdom of the crowd” forecast of the control group by 20 percent in the first year and 50 percent in the fourth year. Tetlock’s team was the clear winner, beating IARPA’s 50-percent goal in the first and all subsequent years.