THE FILE DRAWER EFFECT.

THE FILE DRAWER EFFECT. Simmons, Nelson, and Simonsohn describe in their article (here) some of the degrees of freedom that researchers use in collecting and analyzing data. They list some of the choices researchers have to decide: “Should more data be collected? Should some observations be excluded? Which conditions should be combined and which ones compared? Which control variables should be considered?” They had hundreds of possible comparisons for the three songs and the questionnaire results, but the article as submitted described only one adjustment that had been made to the data. All the other comparisons that were negative results were not reported. They say: “it is common (and accepted practice) for researchers to explore various analytic alternatives, to search for a combination that yields “statistical significance,” and to then report only what “worked.”

Adler says: “Simmons and his colleagues duly reported their adjustment in their mock paper—but they left out any mention of all the other factors that they had tried and discarded. This was all, Simmons emphasizes, within the bounds of what is considered fair play in most psychology departments and journals.”

Adler gives the colloquial name for filing away and not reporting all the negative results that are part of an experiment: “the file drawer effect”.