Memorability of Word‐Pairs: Developing a Method for Generation of Calibrated Stimulus Sets for Repeat, Remote, Automated Testing

Abstract

AbstractBackgroundVerbal‐Paired Associates (VPA) is widely used measure of memory. However, the memorability characteristics of the word‐pairs are not well‐understood, and the task currently suffers from ceiling effects (Uttl et al. 2002). Here we describe a data driven process for word‐pair selection, yielding precise estimates of memorability at the item and list level. This allows tuning of task difficulty to participant characteristics, and the development of many well‐calibrated parallel forms for longitudinal automated testing.MethodFor model development we recruited a total of 185 participants aged 18–40 years without psychiatric diagnoses, chronic medication, or historical head injury using the Prolific online testing platform. We used Cambridge Cognition’s NeuroVocalix software to automatically deliver and score Learning and Recall phases of VPA.In a series of experiments, we explored the contributions of item‐level, pair‐level, list‐level, and study‐level properties to the probability of subsequent recall. At the item‐level, we considered word‐concreteness and word‐frequency. To capture the semantic relatedness of word‐pairs, we trained the GloVe neural network model (Pennington et al, 2014) to predict the co‐occurrence of words from a corpus of transcribed English. This yields high‐dimensional vectors, from which we could calculate the semantic cosine distance between items in a word‐pair. At the list‐level we considered order effects (primacy and recency), and at the study level we considered practice effects.Data from these experiments was combined in a linear mixed effects logistic regression model to yield predicted memorability estimates for new sets of word‐pairs. These test sets were then deployed with a cohort of 43 older adults (aged 55 years and upwards) to evaluate this prediction.ResultModelling showed significant contributions of item‐level, pair‐level, list‐level, and study‐level factors to probability of recall. When applied to the new set of words and tested with our older cohort, we found that this model‐derived measure was a robust predictor of recall (r = .931, p < 10−6).ConclusionWe have a robust method of predicting, for a given word‐pair, what the likelihood of recall will be. This enables the generation of sets of word‐pairs with well‐understood memorability characteristics, for use in automated, repeat, remote assessment.