Effect of Level of Problem Description on Problem Discovery Rates: Two Case Studies

Abstract

The primary purpose of this analysis was to investigate the effect of changing the level of description of usability problems on the estimate of the problem discovery rate ( p). A secondary purpose was to describe a method for using p to estimate the number of problems remaining available for discovery given the constraints associated with a particular participant population, application, and set of tasks. The level of problem description influenced estimates of p, and the direction of influence was predictable, with higher levels of description producing higher estimates of p. Practitioners need a level of description that flows easily into recommendations for redesigning products, but to keep usability studies as efficient as possible, practitioners also need to seek a level of description that takes advantage of common patterns in observed usability problems. Managing this tradeoff is only one of the challenges of usability evaluation, but it is an important one.