Effects of Rater Frames of Reference on Peer Ratings

Abstract

THE INITIAL use of a reference population in in ter per son ratings is generally credited to Walter Dill Scott, who parented the adoption of the man-to man rating scale during World War 1 (6). Essen tially, the rating procedure required the rater to rank order a list of names of persons (officers) he had known and to form a five-man reference frame using his top and bottom ranked recalled persons as extremes on his reference frame, his median ranked recalled person for the middle or third ref erence position, and two midway between median and extreme ranked persons for corresponding m id way positions of two and four. Ratees were then to be compared or matched to the reference persons and assigned numeric values corresponding to the referent's position. Due largely to the problems of rater task difficulty in building his reference frame and of doubtful comparability across different rat ers' scales, the man-to-man scale fell into gradual disuse to be replaced by graphic type rating scales? In their experimental development of measures of group morale using fraternity samples, Gardner and Thompson (4) revised Scott's use of a personal reference frame and introduced a reference frame of "... all males you have ever known". The cur rent Syracuse Scales of Social Relations2, a product of this research, uses a rater reference frame ab stracted from a total population of "all persons known" to the rater. Essentially, any rating pro cedure operationally becomes one of judging each of several stimuli (others) as belonging to one of a series of offered categories. Within the confines of an a priori decision as to the desired meaning fulness and properties of the obtained ratings?*, the problem of choosing a "best" series category be comes a matter of empirical demonstration. The Syracuse Scales' authors reported a series of analyses of ratings obtained within a single col lege fraternity sample using two reference frames, "... all men ever known" and "... all fraternity men ever known" (4: Tables 1, 2, 4, 5, 6, 8-13). Their study revealed that correlations between rat ings made, using the wider "... all persons ever known" reference frame and those using the more restricted "... all fraternity men ever known" ref erence frame, were nearly identical to the test-re test reliability coefficients computed for these rat ings. The authors concluded, in support of their working hypothesis, that individuals rank their com panions "... in a highly similar way when respond ing to the same social need in two reference situa tions" (4: 154). No further study of interreference frame differences has been reported. The adaptation of the Syracuse Scales rating scales procedures for administration to young? r populations of high school and elementary school age pupils (5) warrants further inquiry as to refer ence frame effects. The possiblity that younger rat ers do not discriminately respond to different ref erent persons is both a practical and theoretic con cern, practical in view of the added instructional and respondent tasks of selection of referent frame persons, and theoretic in terms of relating to the growing body of developmental studies of perceptual and social discriminations. The study reported here focuses on the first concern and essentially ex amines two questions; the first, regarding the use of any reference frame at all, the second, regard ing a choice of alternative reference frames. The first question inquires as to the efficacy of explicit reference frame instructions and the follow-through task of identifying reference persons. Are ratings obtained with reference frame instruction different from those obtained without explicit instructions? A negative answer here would suggest the unimpor tance of the reference frame format; indeed, given the reasonable expectation of initial rater differ ences in their privately developed referents, it would grieveously question the entire argument of comparability of ratings across rater groups and even across raters within a given group. The sec ond question, that of differences between selected alternative reference frames was raised in the pre paration of a modification of the Syracuse Scales of Social relations amenable to the testing of third grade and educable retarded public school children (2).