29 WAYS OF DESIGNING AN EXPERIMENT. The Economist reported here on an experiment by professors Raphael Silberzahn of IESE and Eric Uhlmann to address the question: Do dark-skinned soccer players get given red cards more often than light-skinned ones?
This is the kind of question that I often see answered by a study in which a number of equations are fitted to the data, and if enough equations are fitted, there will be a publication reporting that at least one of the results is statistically significant.
Instead, the researchers here set up 29 research teams and gave them a set of data covering 2000 top soccer players in a given year.
As the Economist describes in some detail, the results were all over the lot. The 29 teams used a variety of statistical models and made different decisions about which variables to include ( should a player’s playing position be taken into account?)
The Economist argues that this experiment may explain why efforts to reproduce results published in journals often fail. There are lots of ways for scientists to “attack a data set”.