Further Validation of a Self‐administered Smartphone‐based Verbal Learning and Memory Assessment

Abstract

AbstractBackgroundThe International Shopping List test (ISLT) is a verbal word list learning test set within a realistic shopping list context, supporting ease of cultural adaptation, and addressing content validity and patient relevance. Improvements to and the wide adoption of natural language processing (NLP) software has enabled the possibility for self‐administered tests of verbal learning. LilaTM a self‐administered, smartphone‐application (app) was developed, using the virtual assistant and NLP. We have previously reported acceptability and usability data from qualitative analysis of concept elicitation interviews. We now seek to provide preliminary validation data around its use.MethodA convenience sample of healthy young adults (n = 40), M(SD)age = 35 (16), were invited to install and complete one assessment of the LilaTM app. A usability survey was also administered to explore participants’ experiences. The sample was predominantly female (70%) and of Asian ethnicity (62.5%). Data were collected in Australia, using an English word pool developed for the USA.ResultAcceptability was generally high with N = 38 completing immediate recall rounds and N = 32 completing the delayed recall round. Eight‐two percent found it extremely or somewhat easy to hear and understand what items were on the shopping list and 95% indicated that it was extremely or somewhat easy to understand what they had to do to complete the assessment. Scoring accuracy was 88% at the word item level, with reprocessing to address accent, context setting, and other NLP approaches to improve accuracy underway e.g., homophones biased to the grocery item.ConclusionWhilst acceptability was high, the importance of localization of test stimuli was confirmed as well as the need for additional data processing options to account for accent. Continued optimization of data processing to score participant responses and word item pools to reduce ambiguity is expected to further improve accuracy e.g., accounting for regional accents and consideration of homophones and compound words. However, specific characteristics of speech including in those for whom English may be a second/foreign language will be an area of focus for future development.