Rater Differences Research Articles

The raters' differences in Arabic writing rubrics through the Many-Facet Rasch measurement model.

Writing assessment relies closely on scoring the excellence of a subject's thoughts. This creates a faceted measurement structure regarding rubrics, tasks, and raters. Nevertheless, most studies did not consider the differences among raters systematically. This study examines the raters' differences in association with the reliability and validity of writing rubrics using the Many-Facet Rasch measurement model (MFRM) to model these differences. A set of standards for evaluating the quality of rating based on writing assessment was examined. Rating quality was tested within four writing domains from an analytic rubric using a scale of one to three. The writing domains explored were vocabulary, grammar, language, use, and organization; whereas the data were obtained from 15 Arabic essays gathered from religious secondary school students under the supervision of the Malaysia Ministry of Education. Five raters in the field of practice were selected to evaluate all the essays. As a result, (a) raters range considerably on the lenient-severity dimension, so rater variations ought to be modeled; (b) the combination of findings between raters avoids the doubt of scores, thereby reducing the measurement error which could lower the criterion validity with the external variable; and (c) MFRM adjustments effectively increased the correlations of the scores obtained from partial and full data. Predominant findings revealed that rating quality varies across analytic rubric domains. This also depicts that MFRM is an effective way to model rater differences and evaluate the validity and reliability of writing rubrics.

Dynamics of Rater Differences in Assessing the Age Appropriateness of Media Content: A Multilevel Moderated Mediation Analysis

ABSTRACT Using data obtained from the Common Sense Media website, this study aimed to examine the dynamics that cause differences in age ratings of media content. The study revealed significant differences in age and likability ratings among the raters. Multilevel moderated mediation modeling also showed that the differences in age ratings between the raters were moderated by three content valence variables (extent of negativity, positivity, and consumerism) and that the mediation effects of likability on the rater differences in age ratings were also moderated by the content valence, particularly the extent of negativity and positivity.

Higher depressive symptoms in early adolescents with Autism Spectrum Disorder by self- and parent-report compared to typically-developing peers

BackgroundDepression is more prevalent among male early adolescents with Autism Spectrum Disorder (ASD) than their typically- developing (TD) peers, but findings are limited to two male-only studies. Therefore, a broader understanding of depressive symptoms among both male and female early adolescents with ASD in larger samples is needed. Self- and parent-ratings are frequently used, yet rater differences may emerge and provide unique insights. MethodRatings of adolescent depressive symptoms were examined among 230 early adolescents (10:0−13:5 years) with and without ASD using self- (Children’s Depression Inventory, Second Edition; CDI-2) and parent- report (Child Behavior Checklist; CBCL) measures. The influence of diagnostic group (ASD vs. TD) and rater (early adolescent vs. parent) on ratings were examined with Full Scale IQ and sex as covariates. Additionally, the reliability and strength of agreement between raters were examined. ResultsHigher depressive symptoms were reported by both raters in the ASD group (Borderline range) compared to the TD group (Average range). The interaction of diagnostic group and rater was nonsignificant, but significant main effects emerged. Sex was a significant covariate, but Full Scale IQ was not. The reliability and strength of agreement between raters in the ASD group only were not significant. ConclusionsFindings suggest that depressive symptoms may be higher in both male and female early adolescents with ASD across self- and parent- reports. However, measurement of depression in ASD may be complicated by nonsignificant reliability and strength of agreement between raters. Results have implications for screening and intervention for early adolescents with ASD.

Rater Negotiation Scheme: How writing raters resolve score discrepancies

Abstract In practices of direct assessment of writing ability, the variability of human decision-making during scoring poses great challenges to the validity of assessment (Kane, 2006). The variables causing differences in individual raters’ scoring interpretations have been widely investigated (e.g. Eckes, 2012; Wolfe et al., 2016). However, the issue of how raters negotiate to resolve discrepancies has not received attention although rater negotiation is a widely used score resolution method. As it has been emphasized by scholars interested in the argumentative behavior of raters (e.g. Trace et al., 2017), a systematic analysis of score negotiations will enable us to analyze the dependability of score negotiations. The purpose of this study is twofold: to present a thorough analysis of the argumentative structure of rater discrepancy resolution discussions with a view to understanding their underlying dynamics, and to investigate whether the elements of the argumentative structure of negotiations differ from research settings to authentic score resolution practices. In line with this aim, rater negotiations following a written test at the language school of an English-medium university were analyzed within the framework of Argumentation Theory by Toulmin (1958) and Walton (2005, 2016). The negotiation data were obtained from 99 recorded rater discussions among 30EFL teachers, and transcribed, coded and categorized into argumentative discussion moves. A Rater Negotiation Scheme (RNS) was developed through a recursive data analysis and categorization process, and it was validated through field-testing in authentic settings. The findings have implications both for research on rater negotiations and arguments on the reliability of the method.

Rater experience and the predictive validity of Psychopathy Checklist: Youth Version scores

We compared the predictive validity of Psychopathy Checklist: Youth Version (PCL:YV) scores assigned by a licensed clinician to scores assigned by a graduate student across a sample of 82 juvenile offenders. Although both raters completed in-depth training and practice scoring cases, the graduate student had no prior clinical experience. The raters showed a high level of agreement in their scoring for 11 reliability check cases (intraclass correlation coefficient, ICCA,1 = .90 for PCL:YV Total score), but the scores assigned by the licensed clinician were better predictors of post-release recidivism (area under the curve, AUC = .77) than those assigned by the graduate student (AUC = .45). There was more variability in the scores assigned by the licensed clinician than those assigned by the graduate student, suggesting that more experienced clinicians’ willingness to assign both high and low scores may help explain rater differences in predictive validity.

Clinical Utility of Preoperative Bilingual Language fMRI Mapping in Patients with Brain Tumors.

Previous literature has demonstrated disparity in the postoperative recovery of first and second language function of bilingual neurosurgical patients. However, it is unclear to whether preoperative brain mapping of both languages is needed. In this study, we aimed to evaluate the clinical utility of language task functional MRI (fMRI) implemented in both languages in bilingual patients. We retrospectively examined fMRI data of 13 bilingual brain tumor patients (age: 23 to 59 years) who performed antonym generation task-based fMRIs in English and non-English language. The usefulness of bilingual language mapping was evaluated using a structured survey administered to 5 neurosurgeons. Additionally, quantitative comparison between the brain activation maps of both languages was performed. Survey responses revealed differences in raters' surgical approach, including asleep versus awake surgery and extent of resection, after viewing the language fMRI maps. Additional non-English fMRI led to changes in surgical decision-making and bettered localization of language areas. Quantitative analysis revealed an increase in laterality index (LI) in non-English fMRI compared to English fMRI. The Dice coefficient demonstrated fair overlap (.458 ± .160) between the activation maps. Bilingual fMRI mapping of bilingual patients allows to better appreciate functionally active language areas that may be neglected in single language mapping. Utility of bilingual mapping was supported by changes in both surgical approach and LI measurements, suggesting its benefit on preoperative language mapping.

Combining Dual Scaling with Semi-Structured Interviews to Interpret Rating Differences

Dual scaling, a variation of multidimensional scaling, can reveal the dimensions underlying scores, such as raters’ judgments. This study illustrates the use of a dual scaling analysis with semi-structured interviews of raters to investigate the differences among the raters as captured by the dimensions. Thirty applications to a one-year post-Bachelor’s degree teacher education program were rated by nine teacher educators. Eight of the raters were subsequently interviewed about how they rated the responses. A three-dimensional model was found to explain most of the variance in the ratings for two of the questions and a two-dimensional model was most interpretable for the third question. The interviews suggested that the dimensions reflected, in addition to differences in raters’ stringency, differences in their beliefs about their roles as raters and about the types of insights that were required of applicants.

The effect of training and rater differences on oral proficiency assessment

As a result of the fact that judgments of non-native speech are closely tied to social biases, oral proficiency ratings are susceptible to error because of rater background and social attitudes. In the present study we seek first to estimate the variance attributable to rater background and attitudinal variables on novice raters’ assessments of L2 spoken English. Second, we examine the effects of minimal training in reducing the potency of those trait-irrelevant rater factors. Accordingly, we examined the relative impact of rater differences on TOEFL iBT® speaking scores. Eighty-two untrained raters judged 112 speech samples produced by TOEFL® examinees. Findings revealed that approximately 20% of untrained raters’ score variance was, in part, a result of their background and attitudinal factors. The strongest predictor was the raters’ own native speaker status. However, minimal online training dramatically reduced the impact of rater background and attitudinal variables for a subsample of high- and low-severity raters. Implications suggest that brief and user-friendly rater-training sessions offer the promise of mitigating rater bias, at least in the short run. This procedure can be adopted in assessment and other related fields of applied linguistics.

Parent versus teacher ratings on the BRIEF-preschool version in children with and without ADHD

ABSTRACTCaregiver rating scales represent an important component of comprehensive child neuropsychological assessments for conditions such as Attention-deficit/Hyperactivity Disorder (ADHD); however, low inter-rater reliability (parent vs. teacher) often complicates interpretation. It has been challenging to identify the factors contributing to inter-rater variability, particularly when parents and teachers complete slightly different versions of the same rating scale. The present study examined the associations between parent- and teacher-reported executive functions in 84 children, ages 4–5 years, with and without symptoms of ADHD, using the Behavior Rating Inventory of Executive Function-Preschool (BRIEF-P). Use of the BRIEF-P allows for direct comparison of symptom ratings because parents and teachers complete the exact same measure. Significant associations between raters were observed on 4 of 5 BRIEF-P subscales when rating children with ADHD, but on only 1 subscale when rating typically developing (TD) children. The Shift scale in particular displayed low, non-significant inter-rater association in both groups. Significant group-by-rater interactions were observed for Working Memory and Plan/Organize scales, and driven by larger inter-rater T-score discrepancies in the TD group, such that teachers rated children as having more symptoms than parents. Conversely, examination of raw scores reflected no significant rater differences in the TD group, but significant or nearly significant differences on multiple scales in the ADHD group, such that parents rated more symptoms than teachers. Inter-rater associations for the BRIEF-P appear to vary based on who is being rated (i.e., children with or without ADHD), the specific subscales, and whether standardized or raw scores are analyzed.

Applying many-facet Rasch modeling in the assessment of creativity.

Creativity assessment with open-ended production tasks relies heavily on scoring the quality of a subject’s ideas. This creates a faceted measurement structure involving persons, tasks (and ideas within tasks), and raters. Most studies, however, do not model possible systematic differences among raters. The present study examines individual rater differences in the context of a planned-missing design and its association with reliability and validity of creativity assessments. It applies the many-facet Rasch model (MFRM) to model and correct for these differences. We reanalyzed data from 2 studies (Ns = 132 and 298) where subjects produced metaphors, alternate uses for common objects, and creative instances. Each idea was scored by several raters. We simulated several conditions of reduced load on raters where they scored subsets of responses. We then compared the reliability and validity of IRT estimated scores (original vs. IRT adjusted scores) on various conditions of missing data. Results show that (a) raters vary substantially on the lenient-severity dimension, so rater differences should be modeled; (b) when different combinations of raters assess different subsets of ideas, systematic rater differences confound subjects’ scores, increasing measurement error and lowering criterion validity with external variables; and (c) MFRM adjustments effectively correct for rater effects, thus increasing correlations of scores obtained from partial with scores obtained with full data. We conclude that MFRM is a powerful means to model rater differences and reduce rater load in creativity research.

INTERRATER AND TEST-RETEST RELIABILITY OF THE Y BALANCE TEST IN HEALTHY, EARLY ADOLESCENT FEMALE ATHLETES

Adolescence is the stage of development marked by peak rates of skeletal growth resulting in impaired dynamic postural control and increased injury risk, especially in female athletes. Reliable tests of dynamic postural control are needed to help identify athletes with balance deficits and assess changes in limb function after injury. To estimate the interrater and test-retest (intrarater) reliability of the Y-Balance Test in a group of early adolescent females over a one-month period when administered by novice raters. Twenty-five early adolescent females (mean age 12.7 ± 0.6 years) participated. Two physical therapy student raters, randomly selected from a pool of five, simultaneously assessed each subject's performance on the Y-Balance Test and were blinded to each other's results. Twenty-one subjects returned for a second session (mean 32.3 ± 9.6 days) and were assessed by the same two raters, blinded to previous measurements. Maximum and normalized reach distances and composite scores of the right and left limbs were collected. Intraclass correlation coefficients (ICC) were calculated for between rater and between session agreement. Measurement error and minimal detectable change values were calculated for clinical interpretation. Interrater reliability was excellent for all reach directions and composite scores of the right limb (ICC 0.973-0.998) and left limb (ICC 0.960-0.999) except for the day 1 left anterior reach which was good (ICC 0.811). Test-retest reliability were moderate to excellent for the right limb (ICC 0.681- 0.908) and moderate to good for left limb (ICC 0.714 - 0.811). Minimal detectable change values for the right and left limbs ranged between 2.02-3.62% and 2.77-3.63%, respectively. The Y-Balance Test is a reliable tool to assess dynamic balance in early adolescent females and may be utilized in a clinical setting to monitor function over a one-month time interval. Between rater differences were mainly attributed to disparities in subjective test requirements and not quantitative measures of reach distance. Level 2.

Quantitative Data From Rating Scales: An Epistemological and Methodological Enquiry.

Rating scales are popular methods for generating quantitative data directly by persons rather than automated technologies. But scholars increasingly challenge their foundations. This article contributes epistemological and methodological analyses of the processes involved in person-generated quantification. They are crucial for measurement because data analyses can reveal information about study phenomena only if relevant properties were encoded systematically in the data. The Transdisciplinary Philosophy-of-Science Paradigm for Research on Individuals (TPS-Paradigm) is applied to explore psychological and social-science concepts of measurement and quantification, including representational measurement theory, psychometric theories and their precursors in psychophysics. These are compared to theories from metrology specifying object-dependence of measurement processes and subject-independence of outcomes as key criteria, which allow tracing data to the instances measured and the ways they were quantified. Separate histories notwithstanding, the article’s basic premise is that general principles of scientific measurement and quantification should apply to all sciences. It elaborates principles by which these metrological criteria can be implemented also in psychology and social sciences, while considering their research objects’ peculiarities. Application of these principles is illustrated by quantifications of individual-specific behaviors (‘personality’). The demands rating methods impose on data-generating persons are deconstructed and compared with the demands involved in other quantitative methods (e.g., ethological observations). These analyses highlight problematic requirements for raters. Rating methods sufficiently specify neither the empirical study phenomena nor the symbolic systems used as data nor rules of assignment between them. Instead, pronounced individual differences in raters’ interpretation and use of items and scales indicate considerable subjectivity in data generation. Together with recoding scale categories into numbers, this introduces a twofold break in the traceability of rating data, compromising interpretability of findings. These insights question common reliability and validity concepts for ratings and provide novel explanations for replicability problems. Specifically, rating methods standardize only data formats but not the actual data generation. Measurement requires data generation processes to be adapted to the study phenomena’s properties and the measurement-executing persons’ abilities and interpretations, rather than to numerical outcome formats facilitating statistical analyses. Researchers must finally investigate how people actually generate ratings to specify the representational systems underlying rating data.

Mastery motivation, parenting, and school achievement among Hungarian adolescents

Children’s motivation to master challenging tasks is an important predictor of school success, and yet, such motivation declines during adolescence. It is therefore important to identify ways to intervene to reduce this decline. One malleable factor that is associated with motivation is parenting. The present study used structural equation modeling (SEM) to predict Hungarian seventh graders’ (n = 296) school grades from both youths’ and mothers’ perceptions of parenting (care/warmth, support of youths’ volitional functioning, and support of youths’ independence) and mothers’ and youths’ perceptions of the youths’ mastery motivation. Youth-rated parental care/warmth predicted youth ratings of their motivation, and mother-rated parental care/warmth predicted maternal ratings of their youth’s motivation (both of which predicted youth achievement). In addition, youth-rated parental independence encouragement predicted mother-rated motivation and both directly and indirectly predicted school achievement. In contrast, maternally rated volitional support predicted both youth-rated and maternally rated motivation, which, in turn, both predicted higher achievement. Results suggested that it is important to go beyond youth report in assessing parenting and motivation, in that results differed depending on rater. Nevertheless, despite some rater differences, findings highlighted the importance of parental care/warmth and volitional support in mastery motivation, and of mastery motivation in achievement.

Childhood aggression and the co-occurrence of behavioural and emotional problems: results across ages 3\u201316\xa0years from multiple raters in six cohorts in the EU-ACTION project

Childhood aggression and its resulting consequences inflict a huge burden on affected children, their relatives, teachers, peers and society as a whole. Aggression during childhood rarely occurs in isolation and is correlated with other symptoms of childhood psychopathology. In this paper, we aim to describe and improve the understanding of the co-occurrence of aggression with other forms of childhood psychopathology. We focus on the co-occurrence of aggression and other childhood behavioural and emotional problems, including other externalising problems, attention problems and anxiety–depression. The data were brought together within the EU-ACTION (Aggression in Children: unravelling gene-environment interplay to inform Treatment and InterventiON strategies) project. We analysed the co-occurrence of aggression and other childhood behavioural and emotional problems as a function of the child’s age (ages 3 through 16 years), gender, the person rating the behaviour (father, mother or self) and assessment instrument. The data came from six large population-based European cohort studies from the Netherlands (2x), the UK, Finland and Sweden (2x). Multiple assessment instruments, including the Child Behaviour Checklist (CBCL), the Strengths and Difficulties Questionnaire (SDQ) and Multidimensional Peer Nomination Inventory (MPNI), were used. There was a good representation of boys and girls in each age category, with data for 30,523 3- to 4-year-olds (49.5% boys), 20,958 5- to 6-year-olds (49.6% boys), 18,291 7- to 8-year-olds (49.0% boys), 27,218 9- to 10-year-olds (49.4% boys), 18,543 12- to 13-year-olds (48.9% boys) and 10,088 15- to 16-year-olds (46.6% boys). We replicated the well-established gender differences in average aggression scores at most ages for parental ratings. The gender differences decreased with age and were not present for self-reports. Aggression co-occurred with the majority of other behavioural and social problems, from both externalising and internalising domains. At each age, the co-occurrence was particularly prevalent for aggression and oppositional and ADHD-related problems, with correlations of around 0.5 in general. Aggression also showed substantial associations with anxiety–depression and other internalizing symptoms (correlations around 0.4). Co-occurrence for self-reported problems was somewhat higher than for parental reports, but we found neither rater differences, nor differences across assessment instruments in co-occurrence patterns. There were large similarities in co-occurrence patterns across the different European countries. Finally, co-occurrence was generally stable across age and sex, and if any change was observed, it indicated stronger correlations when children grew older. We present an online tool to visualise these associations as a function of rater, gender, instrument and cohort. In addition, we present a description of the full EU-ACTION projects, its first results and the future perspectives.

Sequential Effects in Essay Ratings: Evidence of Assimilation Effects Using Cross-Classified Models.

Writing assessments are an indispensable part of most language competency tests. In our research, we used cross-classified models to study rater effects in the real essay rating process of a large-scale, high-stakes educational examination administered in China in 2011. Generally, four cross-classified models are suggested for investigation of rater effects: (1) the existence of sequential effects, (2) the direction of the sequential effects, and (3) differences in raters by their individual characteristics. We applied these models to the data to account for possible cluster effects caused by the application of multiple rating strategies. The results of our research showed that raters demonstrated sequential effects during the rating process. In contrast to many other studies on rater effects, our study found that raters exhibited assimilation effects. The more experienced, lenient, and qualified raters were less susceptible to assimilation effects. In addition, our research demonstrated the feasibility and appropriateness of using cross-classified models in assessing rater effects for such data structures. This paper also discusses the implications for educators and practitioners who are interested in reducing sequential effects in the rating process, and suggests directions for future research.

Evaluating Differential Rater Functioning Over Time in the Context of Solo Music Performance Assessment

Abstract Rater variability studies in the context of music performance assessment treat rater effects as static characteristics of raters, where the effects occur similarly across each assessed performance. The purpose of this study was to investigate expert raters’ (N = 13) differential severity/leniency as dynamic processes, where the rater effects occur over time. In particular, we sought to examine the manifestation of group and individual variability using a class of rater effects referred to as differential rater functioning over time (DRIFT). DRIFT refers to the changes in rater performance in relation to a parameter of time. Three classes of Multifaceted Rasch (MFR) models were specified in order to explore differences in raters’ systematic changes in their interpretation of a 4-point rating scale structure across a 5-day rating session: (a) time-static model, (b) rater-by-time interaction model, and (c) partial credit model for time points. Results indicated a significant difference in severity/leniency across time for both the group of raters as a whole and some individual raters. Overall, raters demonstrated a general trend of decreasing severity over the 5-day rating session. Interaction analyses suggested that differential severity/leniency existed for both the raters as a group and for 9 out of the 13 individual raters. Of the total 65 potential pairwise interaction terms examined between raters and days, 21 (33.31%) were found to be statistically significant. Ten interactions systematically underestimated the performances and 11 interactions systematically overestimated the performances. Implications for the improved fairness of ratings in music assessment contexts are discussed.

Problem behaviour and psychosocial functioning in young children with Williams syndrome: parent and teacher perspectives.

There is sparse literature about problem behaviour in young children with Williams syndrome (WS) and little consideration of the perspectives of multiple respondents. Problem behaviour of 35 children with WS ages 2 to 6 was examined based on both parent and teacher report using the Achenbach preschool forms. The most prominent areas of difficulty based on both parent and teacher report were attention problems, pervasive developmental problems and emotion reactivity difficulties. Some rater differences were observed; most notably, teachers reported more externalising behaviour problems including more aggressive behaviour, Attention Deficit/Hyperactivity problems and Oppositional Defiant problems than did parents. Few relations to intellectual functioning, age or gender were observed. Some aspects of problem behaviour evident in older children (e.g. attention problems, social problems) are also apparent for young children with WS, while other areas are less prominent (e.g. anxiety). The implications of the findings for understanding the behavioural phenotype associated with WS are discussed.

Markers\u2019 criteria in assessing English essays: an exploratory study of the higher secondary school certificate (HSCC) in the Punjab province of Pakistan

BackgroundMarking of essays is mainly carried out by human raters who bring in their own subjective and idiosyncratic evaluation criteria, which sometimes lead to discrepancy. This discrepancy may in turn raise issues like reliability and fairness. The current research attempts to explore the evaluation criteria of markers on a national level high stakes examination conducted at 12th grade by three examination boards in the South of Pakistan.MethodsFifteen markers and 30 students participated in the study. For this research, data came from quantitative as well as qualitative sources. Qualitative data came in the form of scores on a set of three essays that all the fifteen markers in the study marked. For the purpose of this study, they weren’t provided with any rating scale as to replicate the current practices. Qualitative data came from semi-structured interviews with the selected markers and short written commentaries by the markers to rationalize their scores on the essays.ResultsMany-facet Rasch model analyses present differences in raters’ consistency of scoring and the severity they exercised. Additionally, an analysis of the interviews and the commentaries written by raters justifying the scores they gave showed that there is a great deal of variability in their assessment criteria in terms of grammar, attitude towards mistakes, handwriting, length, creativity and organization and use of cohesive devices.ConclusionsThe study shows a great deal of variability amongst markers, in their actual scores as well as in the criteria they use to assess English essays. Even they apply the same evaluation criteria, markers differ in the relative weight they give them.

Informant Discrepancies in the Assessment of Attention-Deficit/Hyperactivity Disorder

An attention-deficit/hyperactivity disorder (ADHD) diagnosis requires symptoms to be present across two or more settings, thus requiring information from multiple informants. Research consistently shows low to moderate agreement between parents and teachers; however, the mechanisms underlying these discrepancies remain unclear. This study examined (a) agreement between parents and teachers, (b) effects of using different combination rules in assigning diagnoses, and (c) the role of contextual influences and/or personal biases in informants’ reports. Fifty-five children, their parents, and teachers participated. Parent and teacher ratings on the Attention-Deficit/Hyperactivity Disorder Rating Scale–Fourth edition (ADHD-RS-IV) and clinician ratings on the Behavioral Observation of Students in Schools (BOSS) were obtained. Results indicated moderate agreement among parent and teacher ratings on the ADHD-RS. Diagnostically, the rule for combining information from multiple informants dramatically altered the ADHD classification assigned to the child. With regard to rater differences, the clinician-rated school observation gave some support for the notion that ratings are person rather than context specific.

Measuring Accountability in the Performance Appraisal Context: Rater Status and Organization Culture as Determinants of Rater Accountability

The purpose of this study is to develop a measure of rater accountability, test differences in raters and ratees’ perceived level of rater accountability, and examine the positive relationship between an organizational culture promoting accurate appraisals and rater accountability. A total of 374 surveys were collected from full-time civil servants working in four central government agencies and three local government offices in South Korea. The sample consisted of 254 men and 120 women, and the average organization tenure was 15 years. Rater accountability measures were developed by modifying felt accountability measures and showed internal reliability. The results of multiple regression analysis reveal that an organizational culture promoting accurate appraisals affects rater accountability. However, contrary to expectation, raters and ratees show no meaningful difference in their level of perceived rater accountability.