Recommendations for Increasing the Transparency of Analysis of Preexisting Data Sets
Sara J Weston, Stuart J Ritchie, Julia M Rohrer, Andrew K Przybylski
*Advances in Methods and Practices in Psychological Science*
Abstract
Secondary data analysis, or the analysis of preexisting data, provides a powerful tool for the resourceful psychological scientist. Never has this been more true than now, when technological advances enable both sharing data across labs and continents and mining large sources of preexisting data. However, secondary data analysis is easily overlooked as a key domain for developing new open-science practices or improving analytic methods for robust data analysis. In this article, we provide researchers with the knowledge necessary to incorporate secondary data analysis into their methodological toolbox. We explain that secondary data analysis can be used for either exploratory or confirmatory work, and can be either correlational or experimental, and we highlight the advantages and disadvantages of this type of research. We describe how transparency-enhancing practices can improve and alter interpretations of results from secondary data analysis and discuss approaches that can be used to improve the robustness of reported results. We close by suggesting ways in which scientific subfields and institutions could address and improve the use of secondary data analysis.
Why we wrote it
Psychologists now have more existing data than ever, from national panel studies to social media traces to lab studies that can answer questions their original authors did not ask. Yet most of the open-science conversation assumes a researcher who collects new data for each question. Badges for open data, open materials, and preregistration are hard to earn with a dataset someone else collected, and it is easy to conclude that secondary data analysis is second-class research. We disagreed. Preexisting data are cheap, large, and often the only way to study something like a lifespan. They also invite trouble, and we wanted to say how to handle it.
What we recommend
The trouble is prior knowledge. A researcher who has read ten papers on the Health and Retirement Study already knows how its variables relate, so a “prediction” may not be a prediction at all. Transparency lets readers see which decisions came from theory and which came from the data.
We make four suggestions. First, link to codebooks and explain how the data were accessed. Second, disclose how the data were used before, including your own unpublished analyses. Third, share the code that turns raw data into results. Fourth, preregister the analysis, which is possible with preexisting data if you register before you access the data or see the results. We also built a preregistration template for secondary data.
To improve the inferences themselves, we describe strategies such as cross-validation, holdout samples, multicohort analyses, and multiverse analyses.
What we call for
We close with three requests. Labs should share their datasets as research outputs in their own right. Subfields should ask whether their findings come from many independent samples or from the same few, because several papers on one dataset are not several replications. Developers should build tools for tracking prior knowledge of a dataset.
We present the article as a manifesto for improving secondary data analysis.