The State of Art in the Science of Usability Evaluation Methods: A Kuhnian Perspective

Abstract

In the 1980's researchers and practitioners developed usability evaluation methods (UEMs) that aimed at identifying usability problems in technological artifacts. Among the most known UEMs are Usability Test, Heuristic Evaluation and Cognitive Walkthrough. The development of UEMs was followed by research activities aiming at evaluating and further developing these UEMs. As the methods in addition have gained wide acceptance in design practice, the field of UEMs seems to have matured considerably. However, closer inspection reveals that there is still considerable lack of coherence and agreement. The publication of a controversial paper by Gray & Salzman (1998) underscored this point in that they questioned the methodological validity of five previously published experimental UEM studies. In addition, ten distinguished UEM researchers' very different opinions based on the Gray & Salzman paper left the impression that research in the field of UEM is far from being coherent. In order to throw light upon the current state of art of science in the field of UEMs, this paper analyzes the maturity of the field based on Thomas Kuhn's theory of scientific revolutions. We find that the field is currently in the first of three stages, the pre-paradigmatic stage, as it is lacking a general conceptual framework, as basic terms are ill-defined, and as researchers “facts-gather” almost randomly in the absence of a reason for seeking some particular form of more recondite information.