Scaling a common assessment of associative ability: Development and validation of a multiple-choice compound remote associates task
Kendall A Mather, Sara J Weston, David M Condon
*Behavior Research Methods*
Abstract
The assessment of creativity as an individual difference has historically focused on divergent thinking, which is increasingly viewed as involving the associative processes that are also understood to be a key component of creative potential. Research on associative processes has proliferated in many sub-fields, often using Compound Remote Associates (CRA) tasks with an open response format and relatively small participant samples. In the present work, we introduce a new format that is more amenable to large-scale data collection in survey designs, and present evidence for the reliability and validity of CRA measures in general using multiple large samples. Study 1 uses a large, representative dataset (N = 1,323,480) to demonstrate strong unidimensionality and internal consistency (α = .97; ωt = .87), as well as links to individual differences in temperament, cognitive ability, occupation, and job characteristics. Study 2 uses an undergraduate sample (N = 685) to validate the use of a multiple-choice format relative to the traditional approach. Study 3 uses a crowdsourced sample (N = 357) to demonstrate high test-retest reliability of the items (r =.74). Finally, Study 4 uses a sample that overlaps with Study 1 (N = 1,502,922) to provide item response theory (IRT) parameters for a large set of high-quality CRA items that use a multiple-choice response mode, thus facilitating their use in future research on creativity, insight, and related topics.
Why we asked
Creative thinking often begins with putting together ideas that do not obviously belong together. One classic way to measure that skill is the compound remote associates (CRA) task. A person sees three words, say “cottage,” “swiss,” and “cake,” and has to find the fourth word that links them all (cheese). Researchers have used these items for decades, but nearly always with an open response box and small lab samples. That format is hard to carry into a large online survey, where scoring free text at scale is a headache.
We wanted a version that could travel. In other words, we wanted CRA items that work when tens of thousands of people answer them in a browser, and we wanted to know whether the multiple-choice format still measures the same thing.
What we did
We ran four studies. First, we gave a new set of multiple-choice CRA items to more than 1.3 million people in a very large online personality and ability project, and looked at how the items hang together and how they relate to temperament, cognitive ability, and job characteristics. Second, we gave undergraduates (N = 685) both open and multiple-choice formats to see how the format changes difficulty. Third, we asked a crowdsourced sample (N = 357) to take the items twice, about two weeks apart. Fourth, we used a second very large sample (N = 1,502,922) to estimate item response theory parameters, which tell other researchers how difficult each item is and how well it separates people.
What we found
The items behaved like a single dimension and were internally consistent (α = .97). People who took the test twice scored similarly both times (r = .74). Multiple-choice items were somewhat easier than open ones, but that difference was small. Associative ability was related to a familiar set of traits. Openness showed one of the stronger positive correlations, and neuroticism and anxiety showed some of the strongest negative ones. We did not expect that agreeableness and honesty would be related to associative ability as strongly as they were, and we can only speculate about why. These correlations were smaller than we had expected, and many changed little after we accounted for general cognitive ability.
We also note the limits. Multiple-choice items invite guessing, and we do not yet know whether solving them draws on the same insight process as the open version. We did not include a performance-based creativity measure such as divergent thinking.
What it means
The multiple-choice CRA items are reliable, easy to use in large surveys, and come with the item parameters researchers need to pick the right ones for their sample. The item bank is ready for other researchers to use.