Comparison of remote and in‐person digital speech‐based measures of cognition

Abstract

AbstractBackgroundNeuropsychological testing requires an in‐person visit with a trained administrator using standard/fixed materials. Speech‐based cognitive testing on mobile devices enables more frequent and timely test administration, but head‐to‐head comparisons of in‐person and remote versions of tests are rare. We compare responses to a well‐validated task conducted under supervised (in‐person) and unsupervised (remote) conditions.MethodWe used two data sets containing Cookie Theft picture descriptions (BDAE), one collected in‐person, under supervision (Wisconsin Registry for Alzheimer’s Prevention (WRAP); N = 912; age = 62.2 (SD = 6.7), 70% F) and one collected remotely, without supervision (participants online; N = 93, age = 36.5 (SD = 11.5), 65% F). WRAP participants were judged to be cognitively normal, and online participants self‐reported as healthy. Each participant provided one picture description, yielding 93 remote and 912 in‐person transcribed descriptions. Language metrics previously used in dementia research were extracted: word count (number of words spoken), MATTR (ratio of unique words to total number of words), pronoun‐to‐noun ratio, and semantic relevance. Comparing MATTR, pronoun‐to‐noun ratio, and semantic relevance values elicited in‐person and remotely is important because these characteristics have been found to be impacted by declines in cognition. Differences in word counts may reflect different levels of motivation in the two settings.ResultUsing Cohen’s d effect sizes, differences in mean word count between in‐person and remote participant transcripts was negligible (112 words in‐person and 118 words remote; d = .10, p = .21), the difference for semantic relevance was negligible (.19 in‐person and .18 remote, scale = 0 to 1; d = .14, p = .19), the difference for MATTR was small (.75 in‐person and .74 remote, scale = 0 to 1; d = .37, p<.05), and the difference for pronoun‐to‐noun ratio was moderate (.48 in‐person and .35 remote, scale = 0 to 1; d = .56, p <.05).ConclusionResults show that response length and semantic relevance of responses to the Cookie Theft picture description task are comparable under supervised, in‐person and unsupervised, remote conditions. Small to moderate differences in vocabulary (MATTR) and pronoun‐to‐noun ratio may be due to differences in age and merit further investigation.