Data overuse in aging research: Emerging issues and potential solutions
Daniel K Mroczek, Sara J Weston, Eileen K Graham, Emily C Willroth
*Psychology and Aging*
Abstract
Aging and lifespan development researchers have been fortunate to have public access to many longitudinal datasets. These data are valuable and see high utilization, yet this has a considerable downside. Many of these are heavily overused. Overuse of publicly available datasets creates dependency among published research papers giving the false impression of independent contributions to knowledge by reporting the same associations over multiple papers.
Why we wrote it
Aging researchers are lucky. Studies like the Health and Retirement Study and Midlife in the U.S. follow thousands of people for decades, and the data are public because they are expensive to collect and because funders asked for sharing. The trouble is that a handful of datasets end up answering a large share of the field’s questions. One study is one study, but when the same data appear in five papers it can look like five studies. We borrowed the idea of a “tragedy of the commons” to describe what happens when a shared resource is used until its value wears thin.
Three problems
We describe three problems that overuse creates or makes worse. The first is dependency. Papers built on the same dataset are not independent tests, yet we read a literature as if they were, and meta-analyses sometimes enter the same sample more than once. The second is sample peculiarity and overfitting. Every sample has quirks, and if one sample supplies many of the answers, its quirks carry more weight than they should. In other words, a finding can look replicated when it has only been reproduced in the same people. The third is contradiction. Papers using the same data often make different choices about covariates, waves, or models, so a small effect can appear in one paper and vanish in the next, and the authors may never notice each other’s work.
Solutions
We prefer solutions that researchers impose on themselves. These include sharing more data, so that fewer questions depend on a few datasets, preregistering analyses of existing data, using holdout samples and cross-validation, keeping registries of what has been done with a given dataset, and running coordinated analyses that fit the same model in many datasets. We also lay out a heavier option, in which funders or data-use agreements require a literature search and a preregistration before access. We describe it as a last resort, and we hope it is not needed.
What it means
We are hopeful, because the credibility movement in psychology changed norms quickly without any regulator. This paper offers a starting point, not a finished set of rules, and each solution addresses only some of the three problems. We recommend that anyone using a heavily used dataset review what has already been published on it and state the limits on how far their results generalize.