Verbal reports as evidence of the process operator's knowledge

Abstract

Verbal reports are usually collected with the aim of understanding mental behaviour. As it is not possible to observe mental behaviour directly we cannot test for a correlation between report and behaviour, and cannot assume one. Verbal data cannot therefore be used to test theories of mental behaviour. Verbal data may be produced by a separate report generating process which may give a distorted account. The data can be useful for practical purposes if these distortions are minimal. This paper attempts to assess the conditions in which this is the case. Several methods of obtaining verbal reports are surveyed: system state/action state diagram, questionnaire, interview, static simulation and verbal protocol. Techniques for collecting and analysing the data are described. In each case the small amount of data available on the correlation between reports and observed behaviour are reviewed. The results are not clear. Some verbal data are evidently misleading. Others, however, are sufficiently good to encourage the search for more information about factors affecting their validity.