2 Validity and Reliability of Mobile Toolbox Cognitive Assessments

Abstract

Objective:To present validation evidence for the first eight cognitive measures available through Mobile Toolbox (MTB). These measures use a remote self-administered platform to assess language, working memory, episodic memory, executive function, and processing speed.Participants and Methods:We used two separate samples, recruited as part of a larger study, to validate MTB measures. Sample I, comprised of 92 English-speaking adults ages 18-85, was used to assess internal consistency and construct validity. Participants were first administered “gold standard” cognitive measures (Wechsler Memory Scale-IV Verbal Paired Associates I and II; Wechsler Adult Intelligence Scale-IV Symbol Search, Digit Span, Coding, and Letter-Number Sequencing; Delis-Kaplan Executive Function System Color-Word Interference Test, Peabody Picture Vocabulary Test, Wechsler Individual Achievement Test-4 Spelling, and the Wisconsin Card Sorting Test), after which they completed MTB (pre-loaded on a study-provided smartphone) on their own. Internal consistency was evaluated using measure-appropriate indices (split-half reliability, Cronbach’s alpha or IRT-based indices). Pearson correlation coefficients between MTB tests and measures of similar constructs were used to evaluate concurrent validity. For two tests with timing-dependent scores, Arrow Matching and Shape-Color Sorting, separate analyses were performed for iOS and Android devices. Sample II, with 1,120 English-speaking participants ages 18-90, was used to evaluate age-related change. Participants completed MTB measures remotely on their own smartphones, in a preset order, within a 14-day period. Spearman correlation coefficients, corrected for education, were calculated to evaluate relationships between age and test scores.Results:Sample I participants were 67% female, 52% white, 99% non-Hispanic; average age=48 (SD= 17). Education was: < high school (1%); high school (55%); some college (21%); college (15%); graduate degree (8%). Internal consistency estimates ranged from 0.81 to 0.99. Pearson correlations between MTB and external measures ranged from 0.41 to 0.86 (all p < .01). Of the timed tests, only Shape-Color sorting showed significant score differences between Android and iOS devices. Sample II was 57% female, 13% Hispanic, 72% white, mean age = 45 (SD = 21). Education distribution was: < high school (2%); high school (34%); some college (34%), college (20%); graduate degree (11%). Measures of executive function (r = -0.50; r=-0.57) and processing speed (r= -0.61) showed the expected negative correlation with age (all p <0.001). Negative correlations, although weaker, were also seen on measures of working memory (r=-0.2) and episodic memory (r=-0.2, r=-0.37; p.<.001). Vocabulary performance improved with age (r=0.4; p<.001), while spelling scores remained stable (r=0.09).Conclusions:Initial studies support the validity and reliability of the first eight MTB cognitive measures in two diverse samples. MTB tests showed satisfactory construct validity, as demonstrated by the associations between MTB and well-established tests. Furthermore, most MTB measures correlated with age in the expected directions. Executive function, processing speed and memory typically decrease with age and this decrease was reflected in MTB test performance. In contrast, spelling and vocabulary, typically preserved as we age, did not decrease in our sample. Our results support the use of MTB in cognitive aging research.