During the course of preliminary work on a cross-national study of student achievement in written composition (undertaken by the International Association for the Evaluation of Educational Achievement, or I.E.A.), we1 have had to wrestle with many questions, of which the most intractable had seemed to be whether a system of scoring compositions could be applied across languages. Even within a language group such as English, there appears to be a division of opinion as to what aspects of a composition are important in the curriculum and in evaluation. When one spreads the net wider to encompass non-alphabetic languages and quite different rhetorical and literary traditions, the task of finding a single rating system appears insuperable. To confound the task, it appeared that even the methodology for creating such a scoring scheme was lacking. Only three studies appeard to have shed any light on an approach, that of Diederich, French, and Carlton, that of John Carroll, and that of Purves and Rippere.2 The first two of these had used factor analysis, a statistical technique that groups together different judgments of the same object which appear to change in a closely related fashion (such as height and weight). Diederich had found that raters' judgments of freshman compositions produced five factors: quality and development of ideas, organization, style or flavor, wording, and mechanics (which could be further divided into grammar, punctuation, and spelling). Carroll asked for ratings (using over fifty pairs of adjectives) by scholars of professionally written prose passages, and found that the ratings grouped themselves along six continua: good-bad, personal-impersonal, ornamented-plain, abstract-concrete, serioushumorous, and characterizing-narrating. The Purves-Rippere study was based on a content analysis of student compositions and professional essays on literary topics and was the one study that was international in scope. In that analysis there emerged three different criteria for evaluating literary texts: effect on the reader, formal qualities, and scope and significance of the author's vision.
Read full abstract