Coordinated data analysis: A new method for the study of personality and health

Why we wrote it

Personality psychologists who study health lean heavily on public panel studies, for good reason. They are large, they often include biological data, and they are cheap to use. The trouble is that researchers usually analyze one dataset at a time, and published results from different datasets are hard to compare. Three studies of neuroticism and mortality, for example, used three different measures of neuroticism, different covariates, and follow-up periods of 21, 11, and 14 years, and they reached different conclusions. Large datasets also make it easy to overfit a model or to test a hypothesis you have already seen the answer to.

What we recommend

A coordinated analysis runs identical, or nearly identical, models across several independent datasets and combines the results with meta-analysis. We make five recommendations. First, study simple but foundational relationships, because complicated models are difficult to fit in every dataset. Second, set minimum inclusion criteria before looking for datasets. Third, use scripts, so that every analyst runs exactly the same steps and the results can be pooled in minutes. Fourth, preregister the analyses before the data are cleaned. Fifth, treat generalizability as a goal, and estimate how much results differ across studies, cultures, and measures.

An example from our own work

We illustrate the recommendations with our test of healthy neuroticism, the idea that neuroticism is protective when conscientiousness is high. We drew on the IALSA network of aging studies, found 15 datasets that met our criteria, and split the work into three parallel projects on mortality, health behaviors, and chronic conditions. Over a dozen analysts ran one shared script, and we preregistered the plan on OSF. We also say where we did not follow our own advice. Seven outcomes is more than a simple analysis should have, for instance, though we judged the extra thoroughness worth it.

What it means

Coordinated analysis takes planning and a team. In return, it gives an estimate of an effect and a look at where it does and does not hold. It works best for simple, foundational questions that several datasets can answer.