Halo Error Research Articles

Performance appraisal in the changing world of work: Implications for the meaning and measurement of work performance.

Abstract We critically evaluate performance appraisal research by highlighting how research has traditionally focussed upon the measurement, rather than the meaning of work performance. We propose that maximizing performance appraisal effectiveness requires that we carefully examine the meaning of performance. Next, we consider current trends in appraisal research which have closely followed recent changes in the structure and process of work. The implications of this research for increasing our understanding of the meaning of work performance is considered.In the past decade, we have witnessed a number of profound changes in the workplace which, collectively, have tremendous implications for how individual work performance is assessed. Specifically, with the continuing trends of downsizing, mergers, and decentralization, new performance appraisal practices and research topics have emerged (e.g., 360-degree feedback; electronic performance monitoring).In this paper, we critically examine the state of the art in performance appraisal research. Our central thesis is that some of the most current research on the evaluation of individual level performance continues to emphasize ways of measuring performance without sufficiently concentrating upon the meaning of performance. We will argue that our continuing failure to grapple with the -- the problem of properly defining what is meant by performance -- creates a boundary condition on the utility of this research for the practice of performance appraisal (cf. Austin & Villanova, 1992). In short, we believe that attempting to define performance is a critical precondition for effective performance measurement. Although this might seem obvious and straightforward, we shall soon see that this precondition is often relegated to secondary status or ignored altogether.Over the past three decades, performance appraisal has been one of the most heavily researched topic areas in the fields of Industrial-Organizational (I-O) Psychology and Human Resource Management (Pearce & Porter, 1986). The interest in performance appraisal is understandable inasmuch as organizations appraise employee performance for a variety of purposes. For example, performance appraisals are used for personnel decisions (e.g., promotions, pay increases, identifying employees for remedial training, etc.), employee feedback and development, and personnel research (e.g., selection test validation studies).Historically, performance appraisal research has treated performance measurement as a problem, whereby the goal has been to improve the psychometric quality of performance ratings. Given that both important personnel decisions and employee feedback are often based upon performance rating data, this focus makes a great deal of sense. Ever since Thorndike's (1920) seminal paper on halo error, we have been preoccupied with rating errors and ways of eliminating them through the development of optimal rating formats and rarer training programs (Murphy & Cleveland, 1991).Unfortunately, however, relatively lost in our quest to achieve optimal rating systems has been a concomitant search for the meaning of performance. A number of scholars have repeatedly raised the concern that we have not paid sufficient attention to the meaning of performance (e.g., Austin & Villanova, 1992; Smith, 1976; Wallace, 1965). However, there have been some recent attempts to map out theories of work performance (e.g., Borman & Motowidlo, 1993; Campbell, McCloy, Oppler, & Sager, 1993). These theories attempt to provide performance dimensions (e.g., demonstrating effort, helping and cooperating with others) thought to generalize across jobs. Although ambitious, these theories are necessarily limited in that they cannot possibly capture the entire criterion domains for all jobs. Moreover, they do not provide details concerning levels of performance within dimensions -- the performance standards that underlie each dimension. …

Examination of the relationship between true halo and halo error in performance ratings.

Examination of the relationship between true halo and halo error in performance ratings.

Attributions as a Function of Expertise: The Case of the Police Decision to Arrest1

Using a longitudinal design, this study tested Carroll and Payne's (1977) attributional model on a cohort of 387 police recruits at recruitment% at the completion of academy training. and after I year of police experience. Contrary to expectations, the relationship between causal attributions and other judgments about offenders predicted by the model did not weaken with increasing police experience. However, there was an increase in the extent to which causal attributions also affected judgments about offenders which were not predicted by the model, and this was interpreted as indicating an increase in halo error. It was argued that halo error is indicative of rudimentary schemata which may precede the development of more complex offender typologies.

Examination of the relationship between true halo and halo error in performance ratings.

The purpose of this study was to test competing theories regarding the relationship between true halo (actual dimensional correlations) and halo rater error (effects of raters' general impressions on specific ratee qualities) at both the individual and group level of analysis. Consistent with the prevailing general impression model of halo rater error, results at both the individual and group level analyses indicated a null (vs. positive or negative) true halo-halo rater error relationship. Results support the ideas that (a) the influence of raters' general impressions is homogeneous across rating dimensions despite wide variability in levels of true halo; (b) in assigning ratings, raters rely both on recalled observations of actual ratee behaviors and on general impressions of ratees in assigning dimensional ratings; and (c) these 2 processes occur independently of one another.

A test of the context dependency of three causal models of halo rater error.

C. E. Lance, J. A. LaPointe, and S. A. Fisicaro (1994) tested 3 different causal models of halo rater error (general impression [GI], salient dimension [SD], and inadequate discrimination [D] models) and found that the GI model better accounted for observed halo rating error than did the SD or ID models. They also suggested that the type of halo rater error that occurs might vary as a function of rating context. The purpose of this study was to determine whether rating contexts could be manipulated that favored the operation of each of these 3 halo-error models. Results indicated, however, that GI halo error occurred in spite of experimental conditions designed specifically to induce other forms of halo rater error. This suggests that halo rater error is a unitary phenomenon that should be defined as the influence of a rater's general impression on ratings of specific ratee qualities.

Nature and consequences of halo error: A critical analysis.

The definition of halo error that dominated researchers' thinking for most of this century implied that (a) halo error was common; (b) it was a rater error, with true and illusory components; (c) it led to inflated correlations among rating dimensions and was due to the influence of a general evaluation on specific judgments; and (d) it had negative consequences and should be avoided or removed. We review research showing that all of the major elements of this conception of halo are either wrong or problematic. Because of unresolved confounds of true and illusory halo and the often unclear consequences of halo errors, we suggest a moratorium on the use of halo indices as dependent measures in applied research. We suggest specific directions for future research on halo that take into account the context in which judgments are formed and ratings are obtained and that more clearly distinguish between actual halo errors and the apparent halo effect. When an individual is rated on multiple performance dimensions or attributes, the rater's overall impression or evaluation is thought to strongly influence ratings of specific attributes (Cooper, 1981 b), a phenomenon that is referred to as halo error (Thorndike, 1920). Discussions of halo error are most frequently encountered in the context of evaluative judgment (e.g., in interviews and performance appraisals), but similar phenomena have been noted in research on illusory correlation (Chapman & Chapman, 1969), implicit personality theory (Lay & Jackson, 1969), and interpersonal judgments (Nisbett & Wilson, 1977).

Reliability and accuracy of resident evaluations of surgical faculty.

This study examines the reliability and accuracy of ratings by general surgery residents of surgical faculty. Twenty-three of 33 residents anonymously and voluntarily evaluated 62 surgeons in June, 1988; 24 of 28 residents evaluated 64 surgeons in June, 1989. Each resident rated each surgeon on a 5-point scale for each of 10 areas of performance: technical ability, basic science knowledge, clinical knowledge, judgment, peer relations, patient relations, reliability, industry, personal appearance, and reaction to pressure. Reliability analyses evaluated internal consistency and interrater correlation. Accuracy analyses evaluated halo error, leniency/severity, central tendency, and range restriction. Ratings had high internal consistency (coefficient alpha = 0.97). Interrater correlations were moderately high (average Pearson correlation = 0.63 among raters). Ratings were generally accurate, with halo error most prevalent and some evidence of leniency. Ratings by chief residents had the least halo. Results were generally replicable across the two academic years. We conclude that anonymous ratings of surgical faculty by groups of residents can provide a reliable and accurate evaluation method, ratings by chief residents are most accurate, and halo error may pose the greatest threat to accuracy, pointing to the need for greater definition of evaluation items and scale points.

Compensatory and noncompensatory information integration and halo error in performance rating judgments

Compensatory and noncompensatory information integration and halo error in performance rating judgments

Is halo error a property of the rater, ratees, or the specific behaviors observed?

The stability of halo errors when the ratees, the specific behavioral episodes observed, or both varied was studied. In a laboratory study, halo errors were highly unstable when either the ratees of the ratee behaviors varied (average stability coefficients were .20 and .18 when ratee behavior or both ratees and and their behavior varied, respectively), but halo errors were moderately stable when the ratees and the specific performance segments viewed were kept constant. In a field study using actuel teacher ratings in which the ratee's role, or the semester in which ratings were obtained was varied, very low stability coefficients were again found. The results suggest that halo error is not a stable characteristic of the rater or the ratees but rather is partly a characteristic of the unique rating situation. Practical and theoretical implications are discussed

Is halo error a property of the rater, ratees, or the specific behaviors observed?

Is halo error a property of the rater, ratees, or the specific behaviors observed?

Assessing Self-Appraisal of Job Performance as an Evaluation Device: Are the Poor Results a Function of Method or Methodology?

Self-appraised job performance was studied in relation to supervisory ratings under those conditions in which self-evaluation is most likely to predict performance outcomes (Mabe & West, 1982). In contrast to previous research, there was no evidence of leniency error or restriction of range in self-appraised job performance. Convergence between self- and supervisory ratings was also evident, and was interpreted in light of the presence of halo error.

Implications of Three Causal Models for the Measurement of Halo Error

The appropriateness of a traditional correla tional measure of halo error (the difference between dimensional rating intercorrelations and dimensional true score intercorrelations) is reexamined in the context of three causal models of halo error. Mathematical derivations indicate that the traditional correlational measure typically will underestimate halo error in ratings and can suggest no halo error or even "negative" halo error when positive halo error actually occurs. A corrected correlational measure is derived that avoids these problems, and the traditional and corrected measures are compared empirically. Results suggest that use of the traditional correlational measure of halo error be dis continued.

An Examination of Negative Halo Error in Ratings

Several studies which have used correlational measures to compare levels of observed halo in ratings with true halo levels have found "negative halo error," or true halo that exceeds observed halo. The authors derived a causal model of halo error based on a widely accepted definition of halo error and, in the context of this model, formulated three hypotheses to explain findings of negative halo error. Reanalyses of data reported recently by Kozlowski and Kirsch (1987) and Murphy and Reynolds (1988) supported the hypothesis that "negative" halo error can be indicated when positive halo actually occurs along with imperfect correlational rating accuracy. It is suggested that positive halo rating error may be more pervasive a phenomenon than has recently been thought.

The effects of rater training, job analysis format and congruence of training on job evaluation ratings

An experiment was conducted in order to assess the effects of three independent variables, rater training (accuracy versus error versus dimension), job analysis format (descriptions versus specifications) and congruence of training (congruent versus incongruent) on differential accuracy and halo and leniency error of job evaluation ratings. Subjects were 144 college students at a large midwestern university. The results indicated a significant main effect for job analysis format on all three dependent variables and a significant training X job analysis format effect on halo error. These results suggest at least some similarity in the processes across different types of rating tasks.

Development and Evaluation of Shortcut-Derived Behaviorally Anchored Rating Scales

The purpose of the present study was to evaluate a shortcut method for developing Behaviorally Anchored Rating Scales (BARS). The developmental method included two novel steps: (a) the performance of judgmental cluster analysis to define scales and (b) the direct placement of behavioral incidents on the scales rather than the use of the retranslation step. Two groups of six students each developed shortcut BARS through using an instructional booklet. The BARS were developed to evaluate teaching performance of college instructors. The sets of shortcut scales constructed by the two groups and a traditionally developed BARS were then distributed randomly to students in nine classes who were asked to evaluate both their class instructors and the instructors in their immediately preceding class. The shortcut scales, in comparison with the traditional BARS, were slightly more reliable. Both sets of scales had approximately the same degree of leniency error. The shortcut scales were in general somewhat less adequate in terms of halo error and discriminant validity, but were almost equivalent on these characteristics after the scales with very low reliabilities had been dropped. The results did not support the use of the judgmental cluster analysis, but did indicate that the retranslation step may not be necessary to develop BARS.

A reexamination of the relation between halo error and accuracy.

A reexamination of the relation between halo error and accuracy.

A reexamination of the relation between halo error and accuracy.

A reexamination of the relation between halo error and accuracy.

Halo Error: Is its Strength Dependent upon Attribute Saliency?

The effects of saliency of specific personal attributes on halo effect were investigated with two populations. It was hypothesized that the greater saliency of the specific attribute, the greater the halo in ratings of other attributes. Results did not support the hypothesis. Halo levels were equivalent across populations.

The systematic distortion hypothesis, halo, and accuracy: An individual-level analysis.

In several social perception studies investigators have concluded that raters' semantic conceptual similarity schemata serve to guide and constrain dimensional covariance in the rating judgment process. This effect has been hypothesized to be most likely when ratings are memory based and raters lack relevant job or ratee information. Recent research that has explored the effects of conceptual similarity schemata on performance ratings and halo error has provided some limited support for this systematic distortion hypothesis (SDH). However, these studies are limited because researchers have examined this phenomena using group-level analyses, whereas the theory references individual-level judgment processes. The present study investigated the phenomena at the individual level. The effects of varying levels of rater job knowledge (high, medium, and low) and familiarity with ratees (high and low) were examined for conceptual similarity-rating and rating-true-score covariation relations, for measures of halo, and for rating accuracy components. Results provided support for the SDH, but indicated a boundary condition for its operation and revealed some surprising findings for individual-level rater halo.

Halo Error

This study tested two competing hypotheses concerning the occurrence of halo error in self-(i.e., leader) and subordinate ratings of leader behavior in a field setting at the United States Military Academy (West Point). Unit leaders at Cadet Field Training and their followers described leader behavior in terms of effectiveness and several specific forms of behavior (e.g., communication processes, reward/punishment contingencies). The intercorrelations among the nine measures were examined separately for both self -and subordinate ratings. Based on the attribution literature, the first hypothesis stated that subordinate ratings would show significantly more halo error than self-ratings. A competing hypothesis, based on the rater-ratee acquaintance literature, posited that subordinate ratings would not show significantly more halo than self-ratings. The results supported the first hypothesis. The implications of this finding for future research are discussed.