Coordinated data analysis: Knowledge accumulation in lifespan developmental psychology

Why we wrote it

Replication is a core scientific value, and it is hard to do in aging research. A study that follows people for decades, or until an event such as death, cannot be rerun next semester. We wanted to offer one way to check findings in that setting. Coordinated analysis takes the raw data from several existing studies and fits the same statistical model in each one, using shared code. Each study keeps its own result, and the results can then be combined with a random-effects meta-analysis.

What we cover

The paper has three aims. First, we describe the phases of a coordinated analysis so other labs can try it. These are a literature search and research question, an analysis plan, final selection of studies, the analyses themselves, and the manuscript. The process resembles a registered report, and we recommend preregistering before anyone sees the data. Second, we place the method within the credibility revolution in psychology. Third, we encourage researchers to use existing data networks and repositories, so that the approach is open to people without large grants.

We illustrate each phase with a project underway in our group, which asks how change in personality relates to mortality risk across several longitudinal studies.

What we recommend

Choose simple, foundational questions. Complex models often fail to converge in some datasets, and a simple model is easier to harmonize and to compare. Include datasets that have already been analyzed for the same question, since rerunning them speaks to reproducibility and the data may have grown since the original paper.

Because the datasets stay separate, a coordinated analysis also shows how much results differ from study to study. When findings hold across different countries, cohorts, and measures, we can say more than that they replicate. We can say they generalize. It also avoids some of the file drawer problem, because it does not depend on what got published.

What it means

Coordinated analysis is slow and asks a lot of early-career researchers, and some questions cannot be harmonized at all. We think the trade is worth it, and we encourage senior colleagues to help early-career researchers take part.