Method Bias and Concurrent Verbal Protocol in Software Usability Testing

Abstract

Concurrent verbal protocols are gaining wide acceptance in software usability testing. In this study, the impact concurrent verbalization has on task performance during a software usability test was investigated. Subjects randomly assigned to two levels of verbalization were asked to complete four tasks of varying difficulty using a disk utility package. Subjects in the verbalization condition were asked to provide an explanation for each step taken to complete a task. Subjects in the control condition were allowed to complete each task silently. Dependent variables were task time, error frequency, and responses to subjective measures of mental workload and ease-of-use. Subjects in the verbalization condition committed fewer errors and consumed less task time than subjects in the silent condition. Further, the mean difference in error frequency and task time between conditions increased with task difficulty. These results were extremely important in revealing a potential method bias in usability tests.