Mobile Toolbox cognitive tests: validity and reliability in three samples

Abstract

AbstractBackgroundThe Mobile Toolbox (MTB), an app‐based assessment platform, enables completely remote, self‐administered assessment using participants’ own smartphones. We present psychometric evidence for eight MTB cognitive measures assessing language, working memory, episodic memory, executive function, and processing speed.MethodThree samples were used to evaluate convergent validity, age‐related change, internal consistency and test‐retest reliability. Sample I consisted of 94 adults ages 18‐85 who were administered “gold standard” cognitive measures followed by self‐administered MTB. Sample II, recruited for norming, included 1,020 participants ages 18‐90 who completed MTB measures remotely. Sample III was a convenience sample of 800 participants who completed MTB remotely with retest after 7, 14, or 21 days. Pearson correlation coefficients between MTB and external measures were used to evaluate convergent validity. Internal consistency was evaluated using measure‐appropriate tests: split‐half reliability, Cronbach’s alpha or IRT‐based indices. For measures with timing‐dependent scores, separate analyses were performed for iOS and Android devices. Spearman correlation coefficients were calculated to evaluate relationships between age and test scores.ResultConvergent validity data from Sample I (64% female, 52% white, 99% non‐Hispanic, age = 48+/‐18) revealed moderate to strong correlations between MTB test scores and performance on external measures of similar constructs (Table 1). Table 2 shows results derived from Sample II (44% female, 13% Hispanic, 74% white, age = 45+/‐22). Internal consistency estimates ranged from 0.69 to 0.98. On timed tests, Shape‐Color Sorting and Arrow Matching showed significant score differences between Android and iOS devices when controlling for age. However, internal consistency was equivalent between the two, as indicated by overlapping confidence thresholds (Table 3). Performance on executive function, processing speed, and some memory tasks negatively correlated with age. Word Meaning scores improved with age while Spelling remained stable. Test‐retest reliability, using preliminary Sample III data, ranged from 0.47 (memory tests) to 0.86.ConclusionInitial studies, using three diverse samples, support the validity and reliability of the first eight MTB cognitive measures. MTB tests showed satisfactory internal consistency and convergent validity and most measures correlated with age in the expected directions. Our results support the use of MTB in cognitive research.