Validation of an Objective Behaviorally-Based Usability Method for Real-Time Assessments

Abstract

Rapid increases in computer and data-link technology have resulted in revolutionary but complex displays that comprise integrated cockpits. Consequently, usability techniques must evolve to ensure that advances in information quantity do not compromise information quality, thereby compromising safety. The UserCHAT is a usability method designed to maximize efficient diagnosis of usability problems identified from a behaviorally-based perspective while minimizing time and resource limitations associated with typical assessments. The present study sought to validate the UserCHAT as a suitable methodology that provides accurate real-time identification of usability problems by evaluators with little usability training. Expert and Intermediate evaluators accurately recorded over 80% of first inefficient actions performed by users on benchmark tasks using a relatively complex MFD while Novices with minimal training with UserCHAT captured over 65%. These results suggest that the UserCHAT does not require extensive evaluator training to effectively record user performance and thereby identify usability problems.