WHAT THE “WHEN I’M 64” EXPERIMENT WAS DESIGNED TO SHOW. Of course, the experiment was not designed to show anything about the power of “When I’m 64″. The result is funny, but the other two songs were, I assume, also chosen to make a funny result possible. The other two songs were an instrumental called “Kalimba”, an instrumental song by Mr. Scruff, and “Hot Potato,†a children’s song by The Wiggles. Any of the songs would have done. In the event, the comparison “that worked” turned out to be the group that heard “When I’m Sixty-Four†versus the “Kalimba†group, “adjusted†for the ages of the subjects’ fathers. The experiment was designed to illustrate what Simmons, Nelson, and Simonsohn call “researcher degrees of freedom”.
The title of the paper describes what the experiment showed:
“False-Positive Psychology
Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant”