Formative Usability Evaluation Research Articles

Contemporaneous versus Retrospective User-Reported Critical Incidents in Usability Evaluation

The user-reported critical incident technique involves end-users directly in qualitative data collection during formative usability evaluations. An augmented retrospective variation was developed where participants reported incidents while watching a recording of their usability session, rather than reporting incidents contemporaneous to their occurrence during task execution. Retrospective reporting enables controlled comparisons of user-reported and expert-reported methods, since session recordings can be shown to multiple reviewers. It also allows for the collection of incidents without disrupting traditional usability measures, such as time to complete task. A within-subject study with 24 participants found retrospective reporting to be similarly effective to contemporaneous reporting. The study is described and guidelines are provided for the use of both the contemporaneous and augmented retrospective techniques.

A taxonomy for developing high impact formative usability evaluation methods

On Sunday, 24 April 1994, at the CHI'94 Conference in Boston, a workshop was held to begin development of a taxonomy for producing high impact formative usability evaluation methods. Numerous methods are emerging for formative usability evaluation, including heuristic evaluation, iterative refinement, and other approaches to usability engineering. But these methods have well-known limitations. Many are applicable only to a narrow range of interface types and have not been tried on innovative, non-routine interfaces. Almost all methods are time consuming and personnel intensive.

Users, user interfaces, and objects: Envision, a digital library

Project Envision aims to build a “user-centered database from the computer science literature,” initially using the publications of the Association for Computing Machinery (ACM). Accordingly, we have interviewed potential users, as well as experts in library, information, and computer science—to understand their needs, to become aware of their perception of existing information systems, and to collect their recommendations. Design and formative usability evaluation of our interface have been based on those interviews, leading to innovative query formulation and search results screens that work well according to our usability testing. Our development of the Envision database, system software, and protocol for client-server communication builds upon work to identify and represent “objects” that will facilitate reuse and high-level communication of information from author to reader (user). All these efforts are leading not only to a usable prototype digital library but also to a set of nine principles for digital libraries, which we have tried to follow, covering issues of representation, architecture, and interfacing. © 1993 John Wiley & Sons, Inc.