Preregistration of secondary data analysis: A template and tutorial
Olmo R van den Akker, Sara Weston, Lorne Campbell, Bill Chopik, Rodica Damian, Pamela Davis-Kean, Andrew Hall, Jessica Kosie, Elliott Kruse, Jerome Olsen
*Meta-Psychology*
Why we wrote it
Most preregistration advice assumes you are about to collect new data. But a lot of psychology runs on data that already exist, such as large panel studies, and that is where there is the most room to look first and hypothesize later. If you already know that two variables are correlated in a dataset, a test of that correlation is no longer a real test. We wanted to give people who analyze existing data something practical to fill out.
What we did
The template began at a session at the 2018 meeting of the Society for the Improvement of Psychological Science. Participants started from the standard OSF preregistration form and edited it for secondary data. After the meeting, our group kept refining it by preregistering an actual analysis. The result has 25 questions in six parts: study information, data description, variables, knowledge of data, analyses, and a statement of integrity.
Two things set it apart from the shorter secondary-data template that was developed at about the same time. It covers more of the choices a researcher can make after seeing the data, such as how to define and handle outliers and which robustness checks to run. It also comes with comments on every question and a worked example. Our example asks whether religious people are more prosocial, using the Wisconsin Longitudinal Study.
What the template asks for
The part we consider most important is the section on knowledge of the data. Authors list every paper, preprint, and talk they have based on the dataset, say which variables they used, and describe what they learned that could shape the new hypothesis. In our example, earlier work on the same data had shown that prosocial items were related to agreeableness, so the preregistration adds agreeableness as a control. In other words, disclosing prior knowledge can improve the analysis as well as the honesty of the report.
What it means
Twenty-five questions is a lot, and not every question applies to every study. We say so in the paper, and we suggest that authors explain why a question does not apply. We also recommend answering as many questions in as much detail as possible, since each answer removes a choice that could otherwise be made after seeing the results. Preregistering secondary data analysis takes effort, and it is worth it.