Adaptation, calibration, and validation of a cognitive assessment battery for telephone and video administration.

Abstract

Events such as global pandemics can force rapid adoption of new modes of assessment. We describe the evaluation of a modified neuropsychological assessment for web and telephone administration. Telephone and video conferencing-based neuropsychological assessment procedures were developed and implemented within an ongoing observational study, the Successful Aging following Elective Surgery II (SAGES) study (N = 420 persons). Repeated cognitive assessments were used (N = 2008 observations). Analyses using latent variable psychometric methods were used to compare the measurement modes, and a nested validation sub-study (N = 100 persons) was used to test for measurement equivalence. We used item response theory methods to calibrate data collected by different assessment modes. Measurement equivalence was assessed with Bland-Altman plots and regression analysis. Only small differences were detected between in-person and video modes of assessment. The largest difference among factor loadings was shared for the Boston Naming Test and Visual Search and attention test, but the effects were very small (Cohen's q = 0.06) and not statistically significant (95% confidence interval on q, -0.06, +0.18). In terms of item difficulty differences between in-person and video, the Digit Span Backwards test was less difficult by video with a small-to-moderate effect size (Cohen's d of -0.28, 95% CI, -0.54, -0.01). The contrast of in-person and telephone assessment was larger, with telephone assessment being less difficult than in-person (largest shift in item difficulty for digit span backwards, d = -1.12 95% CI -1.35, -0.90). Calibrated scores from telephone and videoconference demonstrated good agreement (r = 0.72, 95% CI 0.61, 0.80), and the differences could be corrected with latent variable measurement models. Videoconference based neuropsychological assessment can be as precise as in-person. Calibration of ability estimates using latent variable measurement models can address measurement differences and generate scores without evidence of systematic bias.