Development of a usability scale based on the three ISO 9241-11 categories “effectiveness,” “efficacy” and “satisfaction”: a technical note

Abstract

An emerging intervention in the area of dementia therapy is serious gaming. Serious gaming is understood as any meaningful use of computerized game whose chief mission is not entertainment. Given the abundance of available serious games for people with dementia, it seems desirable to appraise and compare their respective user-friendliness in a standardized way. To close the research gap of missing standardized instruments, this article outlines an approach to establish a valid and reliable scale for observer-rated usability assessments, rooted in the International Organization for Standardization (ISO) standard 9241-11 with its categories: effectiveness, efficiency and satisfaction. Three steps were undertaken to develop the scale: (1) search for usability-related terms in the literature, (2) ranking of expressions according to ISO 9241-11 and (3) statistical analysis including across-rater means, selectivity scores, principal component analysis (PCA), discriminant function analysis (DFA) and inter-rater reliability (Cronbach’s alpha). In total, 105 expressions were found. PCA yielded a single factor. DFA yielded Wilk’s Lambda of 0.243, which was statistically significant. For each category, five expressions were determined which represent that category best, being the ones with highest mean in that category and, in case of ties, expressions with highest selectivity. Cronbach’s alpha of those items was 0.435. However, even those items do not discriminate well between categories, since inter-rater reliability is inacceptable. Therefore, it is recommended to administer this scale only to assess usability as a global concept and not to discriminate between the three ISO categories. It might be worthwhile for ISO to review standard 9241-11.