Cognitive reality of multidimensional information processing in knowledge assessment by experts

Abstract

Two experiments were designed in order to examine the knowledge assessment task accomplished by French teachers assessing essays written by French native secondary school students. The focus of the study bears on the potentially multidimensional nature of the processing of evaluation cues. In the experimental conditions conventionally designed to test multidimensional evaluation models or techniques, information about each object to be evaluated (and hence, the dimensions along which objects are compared with each other) are explicitly provided to the subject from some external source. The experiments presented here were aimed at studying information processing by experts performing an evaluation task in which the experts themselves are required to: (i) define the relevant dimensions to be used in the evaluation and (ii) elaborate the necessary information about each of those dimensions for each object (in this case, native language essays). The question is to find out whether or not the information processing carried out by such subjects is multidimensional. If evaluators do in fact perform multidimensional processing of information, this implies not only that evaluation cues belong to differentiated classes (Experiment 1), but also that cues belonging to the same class are processed with respect to each other before being processed with respect to cues of another class (Experiment 2). The results obtained here support that conclusion. Both experiments used the self-paced display paradigm. Subjects were presented with segments of text on a computer screen, and the display time for each segment was recorded. Random coloration of errors belonging to three evaluator-defined classes was found to increase display time on the corresponding segments.