Measuring reading behavior in policy documents: a comparison of two instruments

Abstract

Techniques for observing selection and reading behavior in professional documents, such as the thinking-aloud and the click-and-read methods, may affect the reading process to be observed. Such so-called reactivity problems complicate the use of these instruments in experimental research and usability testing. If their influence is unknown, any experimental results obtained with these instruments may be caused by the testing method. One way to detect reactivity effects is to compare different instruments in a series of experimental studies. In this initial study, we compared the thinking-aloud method, the click-and-read method, a combination of these two methods and a silent reading condition. Subjects read and judged a 53-page policy document in one of these conditions. We investigated whether or not different observation instruments caused specific differences in information selection, judgment and knowledge. Thinking aloud did not cause any differences in the selection of information. Both the thinking-aloud method and the click-and-read method affected the judgment task outcome. Thinking aloud led to many positive and few negative judgments, whereas silent reading led to many negative and few positive judgments. The results for the click-and-read method showed a tendency toward the same effect. Neither method affected the knowledge test results.