Examining the test-taking process: a cognitive perspective on the discourse cloze test

Abstract

Testing consists of making a generalization about a population on the basis of what can be observed in a sample. This principle underlies not only the composition of the sample of individuals to be tested but also the selection of content for the testing instrument itself. Careful attention to the content of assessment tasks is needed to ensure that they are representative of the domain of tasks about which generalizations are to be drawn. However, content representativeness extends beyond mere attention to the problems posed and materials provided in the test. What is more crucially important is the manner in which test tasks are performed. Attention to test-taking process is especially crucial and problematic in the testing of cognitive processing skills. The sample will only be representative if the cogni tive processes involved in task performance are representative of those involved in performance of the domain of tasks in the real world. Thus the validation of tests involving cognitive processes will not be complete unless it includes some examination of the processes by which solutions to test tasks are reached. This line of thinking was propounded 40 years ago by Lennon but has only recently been taken up with the application of ethnographic methods to the study of the test-taking process. This article looks at the findings of a study of the processes employed by sub jects engaged in a discourse cloze test. Think-aloud protocols obtained from test- takers give indications of the reasoning employed in selecting items to complete gaps in the cloze passage and of the strategies employed in doing so. From these indications, cognitive processes are inferred. Comparing observed behaviours and inferred processes with those predicted in a model of ideal performance gives a measure of the construct validity both of the items and of the discourse cloze procedure in general. This introspective validation procedure reveals the test-tak ing behaviour of subjects engaged in cloze tasks and provides a cognitive perspec tive on the vexed question of what cloze actually measures.