Extending the usefulness of the verbal memory test: The promise of machine learning

Chelsea Chandler,Terje B Holmlund,Peter W Foltz,Alex S Cohen,Brita Elvevåg

doi:10.1016/j.psychres.2021.113743

Abstract

The evaluation of verbal memory is a core component of neuropsychological assessment in a wide range of clinical and research settings. Leveraging story recall to assay neurocognitive function could be made more useful if it were possible to administer frequently (i.e., would allow for the collection of more patient data over time) and automatically assess the recalls with machine learning methods. In the present study, we evaluated a novel story recall test with 24 parallel forms that was deployed using smart devices in 94 psychiatric inpatients and 80 nonpatient adults. Machine learning and vector-based natural language processing methods were employed to automate test scoring, and performance using these methods was evaluated in their incremental validity, criterion validity (i.e., convergence with trained human raters), and parallel forms reliability. Our results suggest moderate to high consistency across the parallel forms, high convergence with human raters (r values ~ 0.89), and high incremental validity for discriminating between groups. While much work remains, the present findings are critical for implementing an automated, neuropsychological test deployable using remote technologies across multiple and frequent administrations.

Highlights

Neuropsychological functioning is typically assessed in a profes sional setting during a dyadic exchange between a patient and a psy chometrician
We evaluate three aspects of this test: 1) incremental validity (Section 3.1) - the degree to which multiple administrations provide improved explanatory power for differentiating between groups, 2) criterion validity (Section 3.2) the degree to which our machine learning model predictions converge with expert human judgement of amount recalled across both patients and nonpatients, and 3) parallel forms reliability (Section 3.3) - the degree to which reliable predictions of performance quartile are consistent over testing time of nonpatient individuals
The results show that measurements from psycho metric testing are more likely to reflect the underlying cognition that we are interested in when performance over multiple parallel forms of a task over time are considered

Summary

Introduction

Neuropsychological functioning is typically assessed in a profes sional setting during a dyadic exchange between a patient and a psy chometrician. For this reason, neuropsychological assessment requires considerable resources on the part of the patient and the professional, and is not optimized for repeated administration within an individual over time (see McCaffrey and Westervelt, 1995; Ruff, 2003). The present project examined the use of a novel verbal memory test (story recall) that enables automated, remote, and frequent administration

Objectives

Methods

Results

Conclusion