Responses to comprehension questions and verbal protocols as measures of computer program comprehension processes

Abstract

To study cognitive processes, such as computer program comprehension, researchers often use verbal protocols to collect a process trace. However, the difficulty of collecting and analysing verbal protocol data can discourage even the most resolute researcher. Therefore, alternatives to verbal protocols, such as responses to comprehension questions, are undeniably attractive. Unfortunately, there is little methodological research to justify the use of most alternative methods. The current study compares the use of verbal protocol data with responses to comprehension questions as measures of comprehension process. According to results from the protocol analysis data, programmers used significantly different comprehension processes to understand computer programs in two phases of an experiment. If previous research was correct, then programmers' responses to different types of comprehension questions should reflect the differences in comprehension process. Unfortunately, comprehension process was not reflected in the responses to the questions. Hence, this research confirms that process tracing methods, such as verbal protocols, are a more appropriate method by which to investigate program comprehension processes.