A Methodology and Encoding Scheme for Evaluating the Usability of Graphical, Direct Manipulation Style Interfaces

Abstract

A model-based method for assessing the usability of graphical, direct manipulation style interfaces was developed and applied to a military airspace scheduling system. The method involves collecting and integrating verbal protocol data and mouse/keystroke files, and having an analyst familiar with the task, the data, and Norman's (1986) user activity model review the data and make determinations on what the data mean in terms of the model. A hierarchical encoding scheme based on the model is then applied to the integrated data to structure the human-computer interaction (HCI) process at a detailed interaction level. Meaningful patterns can be identified, frequency of events per task, and number of actions per intention can be calculated at various levels in the hierarchical breakdown, highlighting potential usability problems or instances of indirectness. Repetitious sequences, for example, could imply a missing high-level task domain object or an inability to group objects for application of a single action. Detailed model-based error encodings reflect user-system interface difficulties not only in the execution stage of HCI but in the psychological stages as well. The types of usability problems identified and the advantages of the method are discussed. Based on these results, we have begun developing a multi-media tool to support application of the method.