WHAT THE “WHEN I’M 64” EXPERIMENT WAS DESIGNED TO SHOW. Of course, the experiment was not designed to show anything about the power of “When I’m 64″. The result is funny, but the other two songs were, I assume, also chosen to make a funny result possible. The other two songs were an instrumental called â€œKalimba”, an instrumental song by Mr. Scruff, and â€œHot Potato,â€ a childrenâ€™s song by The Wiggles. Any of the songs would have done. In the event, the comparison “that worked” turned out to be the group that heard â€œWhen Iâ€™m Sixty-Fourâ€ versus the â€œKalimbaâ€ group, â€œadjustedâ€ for the ages of the subjectsâ€™ fathers. The experiment was designed to illustrate what Simmons, Nelson, and Simonsohn call “researcher degrees of freedom”.
The title of the paper describes what the experiment showed:
Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant”