Contemporaneous versus Retrospective User-Reported Critical Incidents in Usability Evaluation

Abstract

The user-reported critical incident technique involves end-users directly in qualitative data collection during formative usability evaluations. An augmented retrospective variation was developed where participants reported incidents while watching a recording of their usability session, rather than reporting incidents contemporaneous to their occurrence during task execution. Retrospective reporting enables controlled comparisons of user-reported and expert-reported methods, since session recordings can be shown to multiple reviewers. It also allows for the collection of incidents without disrupting traditional usability measures, such as time to complete task. A within-subject study with 24 participants found retrospective reporting to be similarly effective to contemporaneous reporting. The study is described and guidelines are provided for the use of both the contemporaneous and augmented retrospective techniques.