How Usable Are Usability Tests? Examining the Suitability of Standard Usability Testing Methods for the Assessment of Apps for People Living with Dementia

Abstract

The potential for technology to support memory or enhance cognition for people living with dementia is currently an area of significant interest. The use of digital systems to facilitate reminiscing has been shown to be beneficial for people living with dementia. This paper reports on a study that examines the value of several common usability testing protocols, methods and metrics when used to evaluate the usability of a new personalised reminiscence ‘app’ called ‘InspireD’. InspireD is a bespoke app designed to support personalised reminiscence for people living with dementia. The app was co-created and designed with input from people living with dementia and their caregivers. The research question was ‘are standard usability protocols appropriate to assess the usability of apps for individuals living with dementia?’ The study aimed to determine on determining the appropriateness of methods commonly used to evaluate usability: concurrent think-aloud protocol; video recording and audio recording; metrics such as task completion rates; task completion times; single ease questions before and after task completion; and the systematic usability scale scores. Seven couples comprising a person with dementia with their primary caregiver evaluated the perceived usefulness and the level of user acceptance of the InspireD app. The study indicated that observation and recording of task completion rates and times produced the most reliable results, while the think-aloud methodology was difficult for the people living with dementia and did not produce any reliable data. Thinking-aloud whilst doing a task was perhaps a distraction since it requires a higher cognitive load. The systematic usability scale score which is derived from a post-test instrument is not reliable, as it had no association with the task completion times.