Leniency Bias Research Articles

Development and Evaluation of Shortcut-Derived Behaviorally Anchored Rating Scales

The purpose of the present study was to evaluate a shortcut method for developing Behaviorally Anchored Rating Scales (BARS). The developmental method included two novel steps: (a) the performance of judgmental cluster analysis to define scales and (b) the direct placement of behavioral incidents on the scales rather than the use of the retranslation step. Two groups of six students each developed shortcut BARS through using an instructional booklet. The BARS were developed to evaluate teaching performance of college instructors. The sets of shortcut scales constructed by the two groups and a traditionally developed BARS were then distributed randomly to students in nine classes who were asked to evaluate both their class instructors and the instructors in their immediately preceding class. The shortcut scales, in comparison with the traditional BARS, were slightly more reliable. Both sets of scales had approximately the same degree of leniency error. The shortcut scales were in general somewhat less adequate in terms of halo error and discriminant validity, but were almost equivalent on these characteristics after the scales with very low reliabilities had been dropped. The results did not support the use of the judgmental cluster analysis, but did indicate that the retranslation step may not be necessary to develop BARS.

Asymmetric influence in mock jury deliberation: jurors' bias for leniency.

Investigators have frequently noted a leniency bias in mock jury research, in which deliberation appears to induce greater leniency in criminal mock jurors. One manifestation of this bias, the asymmetry effect, suggests that proacquittal factions are more influential than proconviction factions of comparable size. A meta-analysis indicated that these asymmetry effects are reliable across a variety of experimental contexts. Experiment 1 examined the possibility that the leniency bias is restricted to the typical college-student subject population. The decisions of college-student and community mock jurors in groups beginning deliberation with equal faction sizes (viz., 2:2) were compared. The magnitude of the asymmetry effect did not differ between the two populations. We hypothesized that the asymmetry effect was caused by an asymmetric prodefendant standard of proof--the reasonable-doubt standard. In Experiment 2, subjects received either reasonable-doubt or preponderance-of-evidence instructions. After providing initial verdict preferences, some subjects deliberated in groups composed with an initial 2:2 split, whereas other subjects privately generated arguments for each verdict option. A significant asymmetry was found for groups in the reasonable-doubt condition, but group verdicts were symmetrical under the preponderance-of-evidence instructions. Shifts toward leniency in individual verdict preferences occurred for group members, but not for subjects who performed the argument-generation task. The theoretical and applied significance of these findings is discussed.

Asymmetric influence in mock jury deliberation: Jurors' bias for leniency.

Investigators have frequently noted a leniency bias in mock jury research, in which deliberation appears to induce greater leniency in criminal mock jurors. One manifestation of this bias, the asymmetry effect, suggests that proacquittal factions are more influential than proconviction factions of comparable size. A meta-analysis indicated that these asymmetry effects are reliable across a variety of experimental contexts. Experiment 1 examined the possibility that the leniency bias is restricted to the typical college-student subject population. The decisions of college-student and community mock jurors in groups beginning deliberation with equal faction sizes (viz., 2:2) were compared. The magnitude of the asymmetry effect did not differ between the two populations. We hypothesized that the asymmetry effect was caused by an asymmetric prodefendant standard of proof--the reasonable-doubt standard. In Experiment 2, subjects received either reasonable-doubt or preponderance-of-evidence instructions. After providing initial verdict preferences, some subjects deliberated in groups composed with an initial 2:2 split, whereas other subjects privately generated arguments for each verdict option. A significant asymmetry was found for groups in the reasonable-doubt condition, but group verdicts were symmetrical under the preponderance-of-evidence instructions. Shifts toward leniency in individual verdict preferences occurred for group members, but not for subjects who performed the argument-generation task. The theoretical and applied significance of these findings is discussed.

Performance Evaluation: A Case for the Use of Consensus Ratings

Data from a sample of government research scientists were used to compare the psychometric properties of pooled and consensus ratings of job performance. It was hypothesized that consensus ratings would possess less halo and leniency error and greater validity than would pooled ratings. The findings supported the halo hypothesis and partially supported the validity hypothesis. Given the severity of the “criterion problem” in validation research, further research was recommended to extend the generalizability of these findings to other performance appraisal settings.

A Performance-based Cooperating Teacher Report

Phelps, Schmitz, and Wade evaluated the adequacy and effec tiveness of a performance-based student teacher evaluation format and process. The form used to assess student perfor mance included behavioral descriptors for each skill observed and a specific statement of performance expectations. Results indicated the presence of both a significant leniency error and a significant halo effect in the summative assessments made by cooperating teachers of student teacher performance.

Effects of purpose of the appraisal and expectation of validation on self-appraisal leniency.

This study examined the effects of purpose of the appraisal and the expectation of validation on selfappraisal leniency. It was hypothesized that self-appraisals conducted under a grading purpose are more lenient than those conducted under a research purpose; self-appraisals obtained under conditions with high expectation of validation are less lenient than those obtained under conditions with low expectation of validation. Using 62 subjects enrolled in two sections of an organizational behavior course, a 2 X 2 factorial experiment was conducted to assess the effect of the purpose of the appraisal and expectation of validation on the amount of leniency in students' self-reports of classroom participation. Both hypotheses were strongly supported in this study. The results were discussed in terms of their relevance and implications for future self-appraisal research. Performance appraisals can be conducted by using supervisor, peer, or self-ratings. Each method has a number of common and unique rating errors (Landy & Farr, 1980). Although extensive research has been conducted to minimize rating errors in supervisor ratings (Latham & Wexley, 1981), very little research has examined the factors that may diminish rating errors in peer or self-ratings. This study examines the factors that may affect the leniency bias of self-appraisals. One factor that may have a direct bearing on the motivation to be lenient is the purpose of the performance appraisal. Performance appraisals can be used for a variety of purposes, which include administrative decisions, personnel development, and personnel research. The motivation to be lenient is likely to be strongest when the self-appraisals are to be used for administrative decisions. In this context, employees tend to inflate self-ratings to attain extrinsic rewards. The motivation to be lenient is probably lowest when self-appraisals are used for personnel research purposes because in this context the obtained appraisals are typically available to researchers only and unlikely to affect future rewards. Previous research on purpose of the appraisal has examined self-appraisal leniency under research conditions only (e.g., Holzbach, 1978; Klimoski & London, 1974). Although these studies reported that leniency occurred in research conditions, they failed to compare the degree of leniency in self-appraisals obtained under different purposes of appraisal. Furthermore, conflicting results were also reported (e.g., Heneman, 1974). Another factor that may affect the motivation to be lenient is the expectation of validation, which refers to the belief that selfreported information will be verified against some other performance measures. A high expectation of validation tends to diminish leniency because the subject may fear punishment for falsifying information. In contrast, self-appraisals conducted with low expectation of validation are more susceptible to leniency error because of the absence of this fear. Research regarding the effects of validation expectation on

The Effects of Halo and Leniency on Cooperating Teacher Reports Using Likert-Type Rating Scales

AbstractThis study evaluated the adequacy of a Likert-type rating scale cooperating-teacher report. Results demonstrated, using a total of 161 elementary and secondary student teachers, the significant presence of a leniency error and the halo effect. The degree of each of these rating errors suggested highly questionable validity of the report as a whole. Cooperating teachers were heavily influenced by the instructional process and the self-confidence of the student teacher. Results further suggested intra- and inter-individual comparisons on the basis of the evaluations to be inappropriate due to the range restriction and the overabundance of superior ratings.

Reliability of raters: The effects of rating errors on the speech rating process

The purposes of the present investigation were to determine the sources and amount of error variance associated with rating speeches and other communicative performance. Using students’ everyday ratings of their classmates’ speeches, interrater reliability and interrater agreement were computed by utilizing the completed rating scale. Leniency error, halo error, and trait error scores were computed by utilizing the six trait scores comprising the rating scale. Raters were both unreliable and a source of considerable error. The leniency error, and the halo error accounted for the majority of the total error variance. The six trait errors accounted for the remaining error. Raters within class were consistent but there was no consistency among raters in different classes. Finally, untrained raters committed more of all eight types of errors and more leniency errors than trained raters.

Behaviorally Anchored Rating Scales vs. Summated Rating Scales: Psychometric Properties and Susceptibility to Rating Bias

Behaviorally Anchored Rating Scales developed according to Bernardin, LaShells, Smith, and Alvarez's (1976) optimal procedure was compared with a carefully constructed summated rating scale. Using both scales, 727 undergraduates rated 32 instructors. Psychometric comparisons indicated that BARS had less halo error, more leniency error, and lower interrater reliablity than the alternative format. The two formats did not differ in ratee discrimination and susceptibility to rating bias due to rater characteristics. Finally, the formats contained convergent and discriminant validity.

THE CONCURRENT VALIDITY OF STANDARDIZED ACHIEVEMENT TESTS BY CONTENT AREA USING TEACHERS' RATINGS AS CRITERIA

To assess the concurrent validity of standardized achievement tests using teachers' ratings (and rankings) of pupils' academic achievement as criteria, 42 teachers evaluated each of their students (n = 1,032) in each of five major curricular areas prior to the administration of a battery of standardized achievement tests. The teachers were directed to rate each student's proficiency disregarding attendance, attitude, deportment, and so on. Within‐class correlation coefficients were computed to eliminate rater leniency bias. The standardized achievement tests were found to have substantial concurrent validity in reading, math, language arts, science, and social studies. The normalized teacher ranks yielded significantly higher validity coefficients than did the ratings, although the magnitude of the difference was small. The concurrent validity coefficients for language arts, reading, and math were significantly higher than those in science and social studies.

The Effects of Purpose of the Appraisal and Expectation of Validation on the Quality of Self-Appraisals.

This paper focused on studying factors which could reduce the leniency bias found in self-appraisals. Using a field experiment, this study demonstrated that leniency bias in self-appraisals varied ...

SELF‐APPRAISAL BASED UPON SUPERVISORY FEEDBACK

Self‐appraisals of job performance have historically proven to be weakly related to appraisals conducted by supervisory personnel. Two studies sought to facilitate rater agreement by invoking supervisory performance feedback as a frame of reference for subordinate raters. In the first study, a sample of 401 branch managers from a large lending institution performed a conventional self‐appraisal and an appraisal containing instructions referencing supervisory feedback (Feedback Based Self‐Appraisal). Feedback Based Self‐Appraisals exhibited significantly smaller leniency error, greater total rating variance, and more agreement with superior ratings than conventional self‐appraisals. FBSAs produced small improvements over conventional self‐appraisals in predicting objective criteria of managerial job performance. Study 2 examined FBSA‐supervisory appraisal agreement for two samples of military organization personnel. The relative degree of feedback available in a work environment was found to moderate FBSA‐superior agreement.

The effects of positional stress and receiver apprehension on leniency errors in speech evaluation: A test of the rating error paradigm

This study tests predictions drawn from the rating error paradigm. The interaction of positional stress and receiver apprehension was utilized to illustrate the working of the theory. The interaction predicted was that raters in a pre‐positional stress/high receiver apprehension condition would exhibit more positive leniency errors when rating speeches than would raters in a post‐positional/low receiver apprehension condition. The hypothesis was confirmed. Applications for teachers are discussed and the implications for the paradigm are presented.

Evaluating Firefighters' Performance

This study compared the performance ratings of 171 firefighters over a 1½-yr. period using a newly developed performance evaluation system. The objective was to develop an evaluation system that would not discriminate against blacks and that would eliminate a strong leniency bias in effect for all firefighters. Data analysis for three separate rating periods showed that there were no significant differences between scores of whites and blacks and that the leniency bias had been eliminated.

Assessing the Job Performance of Nursing Home Staff

Nursing employees at four nursing homes participated in the development of Behavioral Anchored Rating Scales (BARS) to measure the job performance of individual nurses and aides. The reliability of the BARS performance measures was examined by having the immediate supervisors evaluate the job performance of nurses and aides in ten nursing homes. The BARS evaluations had significant convergent reliability between two independent evaluations of the same employees. There were no significant differences in the employees' performance ratings provided by two independent raters. Moreover, both raters tended to agree on the direction and magnitude of the changes in job performance ratings that were recorded over a six-month interval. The BARS evaluations, however, indicated a leniency bias because the performance ratings tended to cluster in the upper half of the scale ranges. There was also a "halo" effect in the evaluations, resulting in little discriminant reliability between the employees' performance ratings on different job dimensions.

Preliminary Assessment of Format-Specific Central Tendency and Leniency Error in Summated Rating Scales

Data for 4-, 5-, and 7-position Likert formats from 292 undergraduates showed systematic error varied among formats, i.e., central tendency errors tended to increase with increasing numbers of categories and to reduce variances expected.

Do grades contaminate student evaluations of instruction?

The correlation between grades and student satisfaction has been interpreted as providing support for belief in a grading leniency bias hypothesis. That is, easy graders are assumed to receive better evaluations than hard gradersbecause they are easy graders. Howard and Maxwell have demonstrated that the relationship between grades and satisfaction might be viewed as an expected result of important causal relationships of other variables (student motivation and progress in the course) with satisfaction and grades, rather than simply evidence of contamination due to grading leniency. Eighty-three students in a research methodology course provided data at two points in a semester. Cross-lagged panel correlation analysis was employed to ascertain the direction of casuality in the relationship between student satisfaction and grades. The findings replicate the Howard and Maxwell path analytic results in finding no evidence that a grades-influencing-satisfaction interpretation is more likely than its opposite, namely, a satisfaction-causing-grades one. The weak relationship between grades and student satisfaction, and the study's inability to find evidence to impugn a satisfaction-causing-grades interpretation of that relationship, renders the grade contamination objection to student evaluations even less poignant.

EFFECTS OF RATER TRAINING AND PARTICIPATION ON COGNITIVE COMPLEXITY: AN EXPLORATION OF SCHNEIER'S COGNITIVE REINTERPRETATION

Schneier (1977) proposed that rater cognitive complexity interacts with format to affect the psychometric soundness of ratings. He also speculated that cognitive complexity may be enhanced by having raters participate in training and/or scale construction programs. This study was designed to test these latter hypotheses, while also providing a partial replication of Schneier's original research. Ninety‐six undergraduate students were assigned to four groups. One participated in the construction of a set of BARS and was trained in their use, a second participated in BARS construction only, a third received training only, and the fourth served as a control group. All subjects were administered a measure of cognitive complexity before and after the experimental treatments. Also, all subjects evaluated five simulated ratees using the BARS. A training × participation (2 × 2) ANOCOV indicated no change in cognitive complexity scores as a result of the experimental treatments. A training × participation × (nominalized) cognitive complexity (2 × 2 × 2) MANOVA on mean ratings for the five stimuli found no effects on leniency error. A similar MANOVA on variances of ratings provided little evidence for effects on halo error. These results, taken in conjunction with those of other investigators who have failed to replicate Schneier's findings, suggest that the cognitive reinterpretation should be accepted with caution at best.

Performance Rating as a Function of Trust in Appraisal and Rater Individual Differences.

Rater cognitive complexity, initiating structure, and trust in the appraisal process were examined for their relationship to leniency error in ratings of patrol officers in two police departments w...

The Effect of Grouping or Randomizing Items on Leniency Response Bias

This study was concerned with the effects of item presentation mode on the degree of leniency bias inherent in responses to standard field research questionnaires. Two types of modes were examined : the first with items measuring the same dimensions grouped together and the second with items measuring the same dimensions distributed randomly throughout the questionnaire. Sixty respondents completed questionnaires containing items from the Leader Behavior Description Questionnaire (Stogdill, 1963) and the Michigan Four Factor Leadership Questionnaire (Taylor and Bowers, 1972); there were thirty respondents in each of the modes (grouped and random). The random relative to the grouped mode showed substantially less leniency response bias, as assessed by both correlational and factor-analytic procedures. However, the magnitude of these effects was still considerable for the random mode. There were no notable differences in leniency effects between the two questionnaires, but some differences were obtained for specific leadership dimensions. Based upon these and the total set of findings, implications for questionnaire validity and for future research are discussed.