Rater Differences Research Articles

The heritability of depressive symptoms: multiple informants and multiple measures.

Earlier research suggests large rater differences in heritability estimates of children's depressive symptoms in the context of significant age and sex-limitation effects. With data from an ongoing, population-based twin-family study, we estimated genetic and environmental influences on children's depression with models allowing for sex-specific effects. Our assessments of twins included self-reports and ratings made by their classmate peers, their parents and their teachers, allowing estimates of genetic and environmental effects with data from different informants. Model-fitting used maximum likelihood estimation of log-transformed data from a sample of 1,366 11- and 12-year-old twin pairs. Estimates of additive genetic effects were significant for both boys and girls across all four informants, and, standardized to reflect the percentage of phenotypic variance accounted for, those estimates ranged from .28 to .71. Significant effects from common environmental sources were found in ratings of teachers and parents, where, typically, one individual rated both co-twins, but neither in peer nominations nor self-ratings. The correlation of teacher and parent ratings was modest, and bivariate model-fitting found no correlation in either genetic or shared environmental effects between teacher and parental ratings. At this young age, sex-limitation effects were found only in teacher ratings, where genetic effects were greater in girls than in boys. Results underscore the utility of using multiple informants to measure children's behavior and provide a foundation for follow-up of these twins in later adolescence.

Competency models: are self‐perceptions accurate enough?

This study examines the accuracy of individual perceptions (self‐estimates) of acquired competence. A concept of relative competence is introduced to account for variation in rater elevation and differences in importance (significance) of specific competencies. The results indicate that the self‐estimates of job‐specific competencies are well executed. Because the distortion in elevation and stereotype accuracy is largely associated with general constructs, the findings suggest that we should focus on modeling competencies to the job. The results also show that even without a correction for interrater differences or a correction for the importance of different competencies, the competency model carries value‐relevant information.

MEASUREMENT ERROR IN RESEARCH ON HUMAN RESOURCES and FIRM PERFORMANCE: HOW MUCH ERROR IS THERE AND HOW DOES IT INFLUENCE EFFECT SIZE ESTIMATES?

Studies of the relationship between human resource (HR) practices and firm performance typically use a single respondent to assess firm level HR practices or HR effectiveness. However, previous research in other substantive areas suggests that rater differences are a potentially important source of measurement error. We demonstrate analytically the potential consequences of both random and systematic measurement error in research on HR and firm performance. However, our main focus is on random error and we show how generalizability theory can be applied to obtain better estimates of reliability by simultaneously recognizing multiple sources (e.g., items, raters) of random measurement error. These more inclusive reliability estimates, in turn, offer the possibility of more precisely quantifying substantive relationships in the HR and firm performance literature. In our sample, reliabilities (as estimated by generalizability coefficients) for single‐rater assessments of HR variables were generally below .50. This degree of measurement error, if present in substantive studies on HR and firm performance, could lead to considerable bias, given that an unstandardized regression coefficient is corrected for measurement error in the independent variable by dividing by its reliability coefficient (not its square root). We also found only limited convergent validity between HR and line managers ratings of a second type of HR measure, HR effectiveness. In general, our findings suggest that future researchers need to devote greater attention to measurement error and construct validity issues. Our study provides an example of how generalizability theory can be useful in this pursuit.

The inconsistency with which raters weight and combine information across targets

To assign an overall performance rating to a target, a rater must weight and combine various pieces of specific performance information about that target. Policy‐capturing research has demonstrated that individual differences in raters can influence the way raters combine specific performance information. The current study examined information processing from a different perspective by exploring the possibility that target differences may also influence the way raters weight and combine performance information. Raters (N = 146) rated each of six targets on six specific performance dimensions and on overall performance. Sequential moderation analyses indicated that targets influenced the way raters, as a group, combined information across targets. These results lend support to the inference that overall performance ratings may not be comparable across targets, that is, they may not reflect the same underlying performance across targets.

Statistical Modeling of Expert Ratings on Medical Treatment Appropriateness

Abstract This article uses latent structure analysis to model ordered category ratings by multiple experts on the appropriateness of indications for the medical procedure carotid endarterectomy. The statistical method used is a form of located latent class analysis, which combines elements of latent class and latent trait analysis. It assumes that treatment indications fall into distinct latent classes, with each latent class corresponding to a different level of appropriateness. The appropriateness rating of a treatment indication by a rater is assumed determined by the latent class membership of the indication, rating category thresholds of the rater, and random measurement error. The located latent class model has two alternative forms: a normal ogive form, which derives from the assumption of normally distributed measurement error, and a logistic approximation to the normal form. The approach has the following advantages for the analysis of ordered category ratings by multiple experts: (1) it assesses whether different raters base ratings on the same or different criteria; (2) it assesses rater bias—the tendency of some raters to make higher or lower ratings than others; (3) it characterizes rater differences in rating category definitions; (4) it provides theoretically based methods for combining the ratings of different raters; and (5) it provides a description of the distribution of the latent trait. The data examined are appropriateness ratings on 848 indications for carotid endarterectomy made by nine medical experts. The located latent class approach provides unique insights concerning the data. It identifies what appears to be a set of clear nonindications for carotid endarterectomy, but a corresponding set of clear indications is not evident. The results indicate that all raters measured a common latent trait of treatment appropriateness, but that some measured the trait better than others. Rater differences in overall bias and rating category definitions are evident. Two methods are used to combine raters' ratings. One uses ratings to calculate a continuous appropriateness score for each indication. The other uses ratings to assign indications to discrete outcome categories, each corresponding to a specific level of appropriateness. The located latent class approach for ordered category measures has possible applications besides the analysis of expert ratings, such as item analysis. Potential extensions of the model are discussed.

Singers and Stereotypes: The Image of Female Recording Artists*

This study investigates the image of popular female singers in different musical styles as conveyed through their album cover photographs. Slides were made of ninety‐one albums recorded by women between 1973 and 1981. They were shown in subgroups to panels of judges who rated them on thirty‐one traits. These traits represent either (a) aspects of the “traditional” feminine stereotype, (b) qualities of sexual attractiveness, or (c) positive, but not specifically feminine, traits. The results indicate that female singers were rated positively, regardless of musical style. Country singers were rated relatively high on the traditional traits, while rhythm and blues artists were high on sex appeal and positive traits. Rock singers were rated as aloof and high on positive traits but low on traditionalism and sex appeal. Pop singers were not rated high or low on any dimension. Some differences in male and female raters’evaluations were found. It was concluded that female recording artists generally are not presented in the same stereotyped manner as women are usually portrayed in other forms of media advertising.

ORGANIZATION AND RATER DIFFERENCES IN PERFORMANCE APPRAISALS

This study examines the effects of organizational differences and rater differences on performance appraisals. Self, peer, and supervisory ratings of performance for nurses in four hospitals and self, student, peer, and supervisory ratings for resident advisors in seven university dormitory complexes were used in this study. The analyses indicate that both organization and rater differences have significant, independent effects on performance ratings. The findings suggest that organizational differences may restrict the generality of the findings of performance appraisal studies across organizational settings. They also may have a negative impact on the usefulness of any particular performance appraisal form in different settings, and on the ability of managers to accurately interpret and compare performance ratings for individuals in different organizational subunits.

Environmental dispositions and the evaluation of architectural interiors

The Environmental Response Inventory (ERI) was developed, in part, to assess dispositions toward everyday physical environments and to aid in explaining variance in response to settings. The eight ERI scales were examined and resultant hypotheses were tested in a design involving evaluative ratings of 10 everyday public building interiors briefly presented in sequences of color slides to 118 participants. Settings were arranged in three levels of generality. Results support the ERI as a promising tool. Biographical items also correlate with ratings. Environment evaluation is discussed as a complex phenomenon, in which rater differences play a significant part.

Predictive utility, sex of rater differences, and interrater reliabilities of the NOSIE-30.

Results suggest that the NOSIE-30 may have predictive utility. Twelve Token Economy patients rated by nine staff members were followed for 1 year. Patients discharged within a year after the NOSIE-30 was administered had higher scores on Social Competence, Personal Neatness, Total Positive Factors, Total Patient Assets and a lower score on Total Negative Factors than patients who remained hospitalized. Generally, there were higher interrater reliabilities on subscales for which behaviors could be observed easily, such as social competence, neatness, and irratability, and less agreement on subscales such as manifest psychosis and social interest. Sex differences found in the raters' perception of a patient behavior indicated that male raters tend to be more tolerant of a patient's negative behaviors. The NOSIE-30 can be a useful tool in staff training and in patient evaluation with a multidisciplinary approach.

Reliability and validity of proverb intepretation to assess mental status

ness: 3 = abstract 2 = partially abstract I = not abstract Concreteness: 3 = concrete 2 = partially concrete I = not concrete Bizarreness: 3 = bizarre 2 = partially bizarre I = not bizarre Personalization: 3 = personal 2 = partially personal I = not personal One major purpose of this investigation was to evaluate interrater reliability in a clinical situation. PROVERB INTERPRETATION 467 Data analysis to assess interrater reliability was of two types, the kappa statistic and two-way analysis of variance to develop an intraclass correlation coefficient. The kappa, developed by Fleiss, measures the reliability between two raters when chance agreement is taken into account.” The intraclass correlation coefficient, developed by Ebel, generates a number “r” which reflects interrater reliability after two-way analysis of variance has been used to determine the variance due to rater differences, the variance due to subject differences, and the experimental error. I2 Both the kappa and the “r” have roughly the same numerical significance as other measures of correlation; that is, the correlation must be greater than approximately 0.6 in order to be statistically significant. It was hypothesized that interrater reliability would be relatively low when measured by these two statistics. A second major purpose of the investigation was to determine whether blind raters would be able to make ratings which would differentiate the various groups on the basis of their proverb interpretations. That is, would manics differ significantly from schizophrenics in their degree of bizarreness, abstractness, concreteness, personalization, and correctness ? It was hypothesized that manics and schizophrenics would resemble one another more than manics and depressives, and that schizophrenics and depressives would fall somewhere in the middle.

Rater and patient characteristics associated with rater differences in psychiatric scale ratings

Rater and patient characteristics associated with rater differences in psychiatric scale ratings

The effect of rater differences on symptom rating scale clusters.

The effect of rater differences on symptom rating scale clusters.

Using Ratings to Validate Personnel Instruments: A Study in Method

SUMMARYThe adoption of a particular personnel device is dependent upon how consistently its use will lead to positive results. A selection device, for example, should consistently pick workers with higher production records than those of workers rejected. When repeated tests of the expected relationship yield inconsistent results, the personnel device is frequently rejected without further trial. This study indicates, however, that inconsistencies of relationship between test and ratings used as measures of productivity may be due to inconsistency of the ratings rather than to any deficiency of the personnel device.In an attempt to evaluate a battery of aptitude tests for hiring purposes, several ratings were obtained from foremen to serve as yardsticks of efficiency.1 The test initially showed inconsistent relationships with the ratings. Relationships which were consistent and sufficiently high to warrant adoption of the test were obtained only after eliminating from some sets of ratings the influence of a non‐relevant factor—length of service.Although the foremen had been instructed to rank the workers on two different traits—Personality and Ability—results showed that the two ratings covered essentially the same factors. Apparently, in spite of the logical analysis in the personnel office, the foreman's ratings of personality and his ratings of ability measure much the same thing—job effectiveness. In fact, the tests designed to predict the ability aspect of effectiveness were more closely related to the Personality ratings than to the Ability ratings. This is probably because both ratings reflected what the personnel analyst would define as “ability”, but the Personality ratings were less contaminated by a spurious relationship to mere length of service. If this finding can be generalized, it may serve to account for the many job failures ascribed to personality difficulty.A further implication is that ratings on Personality may in reality measure an aspect of ability which is relatively independent of length of service, and thus is more predictable by ability tests than are Ability ratings.One measure of the effectiveness of the rating is the extent to which it can be predicted by the selection tests. Contrary to initial expectation, the Over‐all rating was not uniformly superior to the part ratings on this basis and was definitely inferior to the sum of the part ratings. Thus the foremen, in combining the various factors leading to their appraisal of over‐all effectiveness, did not ascribe the best weights; in fact their judgmental combination was not as good as simple clerical addition without statistical adjustment.The initial results showed marked inconsistencies from rater to rater in the relation between the initial ratings and the selection tests. This inconsistency, apparently reflecting adversely on the usefulness of tests, turned out to result in part from rater differences in the meanings which each attached to the basis for his rating. Thus, correction for the spurious length‐of‐service bias by use of a second technique for securing ability ratings transformed the apparent inconsistency into consistency. Similar consistent and substantial relations between tests and ratings were obtained by limiting the study to those cases where length of service was more nearly equivalent, by using the Personality rating (less contaminated with seniority), and by statistical correction for seniority.