Lenient Ratings Research Articles

Impact of self-construal on rater severity in peer assessments of oral presentations

ABSTRACT Raters vary from each other in their severity and leniency in rating performance. This study examined the factors affecting rater severity in peer assessments of oral presentations in English as a Foreign Language (EFL), focusing on peer raters’ self-construal and presentation abilities. Japanese university students enrolled in EFL classes responded to self-construal questionnaires and gave and assessed in-class presentations. The results of path analysis revealed that peer raters’ self-construal and presentation abilities might be potential sources of subtle biases in peer assessments. Whereas peer raters who placed a high value on maintaining harmonious relationships with others were more likely to give lenient ratings over time, those who were sensitive about being evaluated by others tended to become gradually more severe in their ratings. Furthermore, a peer rater’s presentation ability positively predicted rater severity, indicating that individuals with better presentation abilities were more severe in their peer assessments.

The impact of regulatory reforms on European bank behaviour: A dynamic structural estimation

This paper develops a dynamic structural model of bank behaviour. Banks can vary their financing structure, business model and decide on rating solicitation, in the presence of costly debt, corporation tax, insolvency costs and convex adjustment costs. The model is then simulated to examine the impact of regulation on banks’ behaviour. A bail-in regime leads to reduced bank lending activity, while having little impact on bank insolvency rates. Stringent capital requirements reduce bank insolvency rates in a crisis period, while mitigating the reduction in lending activity due to an increased uptake in marginal investments. More lenient credit ratings are associated with increased bank failures. These findings offer wide-ranging implications for policy makers and the banking industry.

Neural and self-reported responses to antisocial news stories: Entertaining versus traditional news introduction

Recently, as entertainment programs have become more prevalent in the news media, there is a growing need to understand how entertainment appeal influences viewers' sociomoral evaluation of news content. We investigated neural and self-reported moral evaluation of antisocial news content in college students who viewed real-life news reports featuring moral violations inside a functional magnetic resonance imaging (fMRI) brain scanner. Each news report was preceded by a traditional or entertaining style of introduction. The behavioral results showed that antisocial severity was reduced for antisocial news content that was presented following entertaining news introductions versus traditional news introductions. The fMRI results showed that entertainment news introductions tax more cognitive control resources than traditional news introductions during news processing, as indicated by greater activation in the dorsolateral prefrontal region; however, they diminish moral saliency, as suggested by the reduced activation in the medial prefrontal region. We also found that greater dorsolateral prefrontal activation during the early phase of news reports was associated with more lenient moral acceptability ratings in the entertainment condition. Furthermore, reduced functional connectivity of the mentalizing brain network was observed in the entertainment condition when compared with the traditional condition. These results suggest that entertainment appeal may increase resource depletion, diminish moral sensitivity, and reduce functional integration of relevant social information during news processing, thus hindering viewers' moral scrutiny. This novel study contributes to a better understanding of how entertainment features influence viewers’ sociomoral evaluation of news stories.

An Examination of Attributions, Performance Rating and Reward Allocation Patterns: A Comparative Study of China, India, Tanzania and the United States

In this study, using a sample of 191 individuals with significant work and supervisory experience from four countries (China and Tanzania—highly collectivistic, India—moderately collectivistic and the United States—highly individualistic), we examined whether cultural orientations of individualism and collectivism predicted performance ratings, internal attributions made to the poor performing employee and adherence to equity norm in reward allocations. Multiple regression analyses indicated that collectivist Chinese and Tanzanian raters provided more lenient performance ratings to a poor performing employee than individualistic Americans with the Indian raters providing more lenient ratings than Americans but more stringent ratings than Chinese or Tanzanians. Further, American raters made greater internal attributions than Tanzanians or Chinese. Chinese raters made relatively greater internal attributions to the employee than Tanzanians though both cultures are collectivistic cultures. Americans adhered to the equity norm the most and Indians adhered to equity norm but to a lesser extent than Americans but more than Tanzanians and Chinese. Implications are discussed.

Race to the Bottom? Competition Among Rating Agencies in the European Structured Debt Market – Evidence from the CRA Regulation

After analyzing the long-term credit ratings on European structured debt products before and after European Parliament’s introduction of Article 8d in June 2013, which aims to diversify the market by encouraging the use of small credit rating agencies, I find that the rating split between large and small rating agencies significantly drops. During the first half of the data sample (January 2011-May 2013) or prior to Article 8d’s introduction, the data show that the Big 3 assign on average ratings that are at least 2.5 notches more conservative than those of DBRS, the fourth-largest credit rating agency. In the second half of the data set (July 2013-May 2017) or post implementation of Article 8d, the rating split between the Big 3 and DBRS decreases by at least 1.5 notches. This implies that under heightened competition from smaller rating agencies like DBRS, the dominant credit rating agencies are more likely to engage in a “race to the bottom” and assign more lenient credit ratings to securities.

Investigating adjudicator bias in concert band evaluations: An application of the Many-Facets Rasch Model

Prior research indicates mixed findings regarding the consistency of adjudicators’ ratings at large ensemble festivals, yet the results of these festivals have strong impacts on the perceived success of instrumental music programs and the perceived effectiveness of their directors. In this study, Rasch modeling was used to investigate the potential influence of adjudicators on performance ratings at a live large ensemble festival. Evaluation forms from a junior high school concert band festival adjudicated by a panel of three expert judges were analyzed using the Many-Facets Rasch Model. Analyses revealed several trends. First, the use of assigning “half points” between adjacent response options on the 5-point rating scale resulted in redundancy and measurement noise. Second, adjudicators provided relatively similar ratings for conceptually distinct criteria, which could be evidence of a halo effect. Third, although all judges demonstrated relatively lenient ratings overall, one judge provided more severe ratings as compared to peers. Finally, an exploratory interaction analysis among the facets of judges and bands indicated the presence of rater-mediated bias. Implications for music researchers and ensemble adjudicators are discussed in the context of ensemble performance evaluations, and a measurement framework that can be applied to other aspects of music performance evaluations is introduced.

Determinants of biased subjective performance evaluations: evidence from a Taiwanese public sector organization

Drawing on the notion of guanxi, which is obligation-bound, in Confucian cultures, this study investigates whether rating leniency and rating compression in performance evaluation are based on supervisor–subordinate acquaintanceship, alma mater ties, and subordinates' physical proximity to their supervisor. After controlling for supervisor–subordinate gender similarity and age difference, this study first finds that public sector supervisors are willing to provide relatively lenient ratings and compressed ratings for their subordinates when their personal acquaintanceship with them is strong, and when there is close physical proximity between them. Second, an alma mater connection with a subordinate leads to rating leniency by the supervisor. Finally, it is worth noting where this study differs from Bol's research (from Western cultures). In particular, while Bol, J.C. [2011. The determinants and performance effects of managers’ performance evaluation biases. The Accounting Review, 86 (5), 1549–1575] demonstrates that age differences between supervisors and subordinates decrease rating leniency, this study documents insignificant results for similar demographic characteristics in age and gender. Bol, J.C. [2011. The determinants and performance effects of managers’ performance evaluation biases. The Accounting Review, 86 (5), 1549–1575] also demonstrates that if a private sector supervisor sharing a work location with subordinates faces relatively lower information-gathering costs, he or she will give less lenient and less compressed ratings. However, this study shows opposite findings for the guanxi norm in Confucian societies.

Multi-Faceted Rasch Measurement and Bias Patterns in EFL Writing Performance Assessment

This study applied multi-faceted Rasch measurement to examine rater bias in the assessment of essays written by college students learning English as a foreign language. Four raters who had received different academic training from four distinctive disciplines applied a six-category rating scale to analytically rate essays on an argumentative topic and on a descriptive topic. FACETS, a Rasch computer program, was utilized to pinpoint bias patterns by analyzing the rater-topic, rater-category, and topic-category interactions. Results showed: argumentative essays were rated more severely than were descriptive essays; the linguistics-major rater was the most lenient rater, while the literature-major rater was the severest one; and the category of language use received the severest ratings, whereas content was given the most lenient ratings. The severity hierarchies for raters, essay topics, and rating categories suggested that raters' academic training and their perceptions of the importance of categories were associated with their bias patterns. Implications for rater training are discussed.

Operational Rater Types in Writing Assessment: Linking Rater Cognition to Rater Behavior

This research investigated the relation between rater cognition and rater behavior, focusing on differential severity/leniency regarding criteria. Participants comprised a sample of 18 raters examined in a previous rater cognition study (Eckes, 2008b). These raters, who were known to differ widely in their perceptions of criterion importance, provided ratings of live examinee writing performance. Based on these ratings, criterion-related bias measures were estimated using many-facet Rasch measurement. A cluster analysis of bias measures yielded four operational rater types. Each type was characterized by a distinct pattern of differentially severe or lenient ratings on particular criteria. The observed bias patterns were related to differential perceptions of criterion importance: Criteria perceived as highly important were more closely associated with severe ratings, and criteria perceived as less important were more closely associated with lenient ratings. Implications of the demonstrated link between rater cognition and rater behavior for future research into the nature of rater bias are discussed.

Long-Term Incentives, Managerial Effort and Supervisor Evaluation Bias

We study the incidence of supervisors’ evaluation biases in a biannual incentive system in an Italian public administration. Using performance reports for 106 employees over three biannual evaluation periods (2001-2006), we analyze supervisors’ intertemporal evaluation biases. We find evidence for lenient and compressed performance ratings especially in the second year of each biannual evaluation period. We explain these biases, and their intertemporal variation, by supervisors’ relative emphasis on subjective and objective performance metrics. We further analyze the effect of performance categorization and find that leniency is enhanced for ratings closer to the lower boundary of each performance category. The results have important implications for understanding the trade-offs supervisors face when enhancing their subordinates’ long-term performance, and short-term performance measure accuracy.

Rating Level and Accuracy as a Function of Rater Personality

We investigated the predictability of rating level and two measures of rating accuracy from rater Agreeableness (A) and Conscientiousness (C) scores of the Five Factor Model. One hundred and twenty‐six students made peer ratings after participating in several group exercises under conditions designed to emulate the modal peer rating system in which raters had low accountability for their ratings. Scores were correlated with average rating level (r=.18, p<.05) and both measures of rating accuracy (p<.05) and C scores were correlated with rating level (−.20, p<.05) and both measures of rating accuracy (p<.05). As suggested by Bernardin, Villanova, and Cooke (Journal of Applied Psychology, 85, 232–236, 2000), raters who were both more agreeable and less conscientious made the most lenient and least accurate ratings. Contrary to Yun, Donahue, Dudley, and McFarland (International Journal of Selection and Assessment, 13, 97–107, 2005), more agreeable raters also tended to rate the least effective performers more leniently than did other raters.

Setting Standards for Performance Tests: A Pilot Study of a Three-Level Angoff Method

The Yes/No Angoff method is a standard-setting variant in which judges estimate whether a borderline student would or would not accomplish each item. However, when there is a 50% chance of success, systematically severe or lenient ratings may skew the cut score. This study piloted a three-level Angoff in which items were assigned to "Yes," "No," or "Maybe" categories. Five judges provided three-level Angoff ratings for seven checklists used in a clinical skills exam for fourth-year students. Systematic severity and leniency were simulated by converting all "Maybe" ratings to "Yes" or "No" ratings, respectively. Forty-one percent of 121 items had at least one "Maybe" rating. Ten percent of all ratings were in the "Maybe" category. One judge accounted for 83% of all "Maybe" ratings; three did not use it at all. Case failure rates varied considerably depending on the simulated severity of ratings. Overall failure rates were not substantially impacted. The three-level Angoff retains the cognitive simplicity of the Yes/No Angoff while addressing the challenge of items midrange for the borderline candidate.

Effects of Feedback Accountability and Self-Rating Information on Employee Appraisals: A Replication and Extension

The influence of feedback accountability and self-rating information on employee performance appraisals was examined. Undergraduate business student participants assumed the role of "supervisor" and evaluated a fictitious "subordinate" whose performance on a clerical task was either moderately poor or very good. Participants were either given fictitious self-rating information, or no self-rating information, and were told they were expected to provide performance feedback to their ratee, or there was no feedback expectation. As expected, in Study 1 both self-rating information and expected feedback-sharing independently resulted in lenient ratings for poor performance, and the combined effects resulted in the highest ratings. By contrast, results for good performance (Study 2) were not significant. Implications of the findings for human resource management practice and research were discussed.

Discretion and bias in performance evaluation: the impact of diversity and subjectivity

This paper examines the impact of performance measure diversity and the use of subjective performance measures on performance evaluation bias. The empirical results indicate that both performance measure diversity and subjectivity are positively related to performance evaluation bias. More specifically, I find that the use of multiple objective performance measures and the use of subjective performance measures are related to more compressed performance ratings and more lenient performance ratings, which could result in problems in personnel decisions and future incentives.

Assessment of English proficiency in international medical graduates by physician examiners and standardized patients.

Since 1986, the Ontario Ministry of Health has provided a medical licensure preparation programme for international medical graduates. Because of the diversity in candidates' oral English proficiency, this competency has been viewed as a particularly important selection criterion. To assess and compare the quality of ratings of oral English proficiency of international medical graduates provided by physician examiners and by standardized patients (SPs). PARTICIPANTS AND MATERIALS: The study samples consisted of 73 candidates for the Ontario International Medical Graduate (IMG) Program, and physician examiners and SPs in five 10-minute encounter objective structured clinical examination (OSCE) stations. Materials used were a seven-item speaking performance rating instrument prepared for the Ontario IMG Program. Rating sheets were scanned and the results analysed using SPSS 9.0 for Windows. Correlations between the physician and SP ratings on the seven items ranged from 0.52 to 0.70. The SPs provided more lenient ratings. Mean alpha reliability for the physicians' ratings on the seven items was 0.59, and for the SPs' 0.64. There was poor agreement between the two sets of raters in identifying problematic candidates. Notwithstanding the sizable correlations between the ratings provided by the two rater groups, the results demonstrated that there was little agreement between the two groups in identifying the potentially problematic candidates. The physicians were less prone than the SPs to rate candidates as problematic. SPs may be better placed than the physician examiners to directly assess IMG candidates' oral English proficiency.

Principles of disaster management lesson 11: personnel evaluation.

A disaster manager must be able to evaluate staff members in order to make effective staffing decisions during the transitions between phases of a disaster and to help improve the performance of the operation by determining what aspects of an individual's work needs improvement. Criteria for personnel appraisal and measures of performance, objective or subjective, are essential to conduct a fair and thorough evaluation. Errors introduce bias into the appraisal in the form of a "halo" effect, harsh or lenient ratings, a central tendency error, or recency effect, but recognition of these errors helps reduce their effect. A results-based appraisal program is favored because it focuses on what the worker achieves. Regular appraisal periods are suggested for optimal improvement in performance. Suggestions are given for an effective personnel appraisal, such as focusing on positive work performance, being aware of biases in judging people, and being specific when citing examples of work performance.

Evaluating teachers: A call for change

A standard model for evaluating teaching has been employed in our schools for decades (Boyce, 1915). In the vast majority of school districts, a nontenured teacher is observed twice a year, for a period of 20 to 30 minutes, by the building principal (Lower, 1987; Sweeney & Manatt, 1986). Tenured teachers are observed for approximately the same number of minutes, but less frequently. Sometimes a conference follows the observation where the principal's rote is to describe the strengths and weaknesses of the teacher's performance and offer suggestions for improvement (Haefcle, 1981). This dominant model of teacher evaluation is in trouble. Evaluation criteria lack validity. Evaluators are untrained. Evaluators consistently award lenient ratings to weak and incompetent teachers. These and other practices examined below clearly indicate that the current teacher evaluation model is seriously deficient (Haefele, 1992). Deeply entrenched in American schooling, any massive attempt to dump this model and start over appears unachievable. Inasmuch as school districts are unwilling to seek alternative procedures for evaluating teachers, the serious deficiencies should be identified and radically changed to improve the prevailing model. Areas where the most dramatic changes are essential are reviewed here.

A Field Study of Rating Conditions and Leniency in Performance Appraisal

Performance appraisal ratings of 89 secretarial and clerical employees of a large university were studied. Most recent administrative ratings, used to make personnel decisions, were obtained for all employees. In addition, research ratings were collected under one of two conditions. For approximately half of the employees, confidential ratings were obtained from the employees' supervisors whereby the supervisors were told that only the researchers would see their ratings. For the other half of the employees, ratings were obtained under sharing conditions whereby supervisors were told that copies of their ratings would be sent to their respective employees. As expected, administrative ratings showed relatively more leniency than corresponding confidential research ratings but did not show more leniency than corresponding sharing research ratings. Findings were discussed in terms of the cognitive dynamics which may cause more lenient ratings to occur.

Effect of Reciprocal Leniency on the Relation between Consideration Scores from the Leader Behavior Description Questionnaire and Performance Ratings

This study tested the hypothesis that “reciprocal leniency” moderated the relationship between Consideration scores on the Leader Behavior Description Questionnaire—Form XII and performance ratings. Reciprocal leniency was defined as a response style in which scores on the questionnaire are affected by harsh, lenient, or fair ratings made by the supervisor. Results partially supported the hypothesis.

The effects of specifying job requirements and using explicit warnings to decrease sex discrimination in employment interviews

To determine the relative effectiveness of instructions designed to reduce sex discrimination in employment interviews, 96 students rated resumes for both a male and a female applicant. In one instructional condition subjects were told the specific requirements necessary for success on the job. In a second condition subjects were warned that it was illegal to discriminate on the applicant's sex. A third condition received both instructions, and a fourth received neither. Results revealed a series of interactions suggesting that the use of legal warning alone tended to increase the ratings of male applicants in some cases and influenced how male evaluators rated all applicants in other cases. The major implications were that legal warnings may actually bias ratings in favor of male applicants and may occasionally lead male evaluators to make lenient ratings. The specification of job requirements was suggested as a viable method of reducing discrimination.