Effects of Heuristic Type, User Interaction Level, and Evaluator’s Characteristics on Usability Metrics of Augmented Reality (AR) User Interfaces

Abstract

Heuristic evaluation is a famous usability inspection method often used in the assessment of Augmented Reality (AR) systems. Usability heuristics can be characterized as general or specific to a domain. During comparative studies between a new and a traditional set of heuristics, the focus is often around performance metrics (number of problems, time, errors), and little attention is given to the experience itself (i.e., evaluator’s overall satisfaction toward utilization of the set of heuristics). In addition, limited research has been done on the application of heuristics to evaluate interfaces with different levels of interaction, along with how a longer set of low-level heuristics compares against a traditionally-sized set of high-level heuristics. In this study, recruited participants (mostly university students) performed a heuristic evaluation using either a set of 110 specific heuristics (SH) or a set of 23 general heuristics (GH) to evaluate two versions of an AR application, designed with a low and a high level of interactivity. Big Five personality traits, prior experience with AR or Virtual Reality systems, and various usability data (e.g., subjective ratings related to the usability of heuristics and AR application, number of unique problems, and severity ratings) were collected through questionnaires and utilized during statistical analysis. Results show that even though the employment of the set of GH produced higher overall satisfaction of evaluators, the set of SH resulted in a significantly larger number of unique usability problems. Furthermore, the more interactive version of the AR application was perceived by evaluators as the harder version to inspect, regardless of the set of heuristics utilized. Neuroticism and agreeableness traits were found with a significant impact on subjective usability ratings.