A Simulation Approach to Validate Models Derived from Observational Studies

Abstract

Protocol analysis is a suitable approach for studying cognitive behaviors. It is a psychological research method that elicits verbal reports from research participants, and is used to study the thinking process in the cognitive sciences (cognitive psychology, behavior analysis). Consequently, much of what we know in most domains and particularly in software engineering about practitioners' cognitive processes is based on researchers' use of protocol analysis (Robillard et al, 1998). Trickett & Trafton (2002) provide a comprehensive summary of practical how-to primers on protocol analysis. Hughes & Parkes (2003) provide an excellent survey of trends in the use of verbal protocol analysis in software engineering research. They found that some difficulties associated with the technique are: (i) the effort of devising a valid and reliable encoding scheme; (ii) the time-consuming nature of the encoding process; and (iii) the problem of comparing the results of researchers who have applied different encoding schemes.