Rater Behavior Research Articles

Rater behaviors in peer evaluation: Patterns and early detection with learner model

Peer evaluation is a common practice in team-based learning (TBL) designs, which can cover the assessment of individual or group work. However, the integrity of peer evaluation can be compromised by unserious raters—individuals who do not earnestly engage in the evaluation process. These raters may exhibit behaviors like consistently assigning the same score, rushing through evaluations, or evaluating before or long after the target presentations. This study delves into the issue of unserious peer evaluation in group presentations, with a specific focus on understanding the behavior patterns in the digital system. Utilizing evaluation behavior analysis (EBA) indicators, we identify patterns linked to unserious raters during the peer evaluation process. Meanwhile, we also connect these patterns to rating consistency and actual course performance, underscoring the significance of behavior patterns. Further, we conduct a preliminary analysis to explore the application of learner model data available before the peer evaluation starts for the early detection of unserious raters. This finding can assist teachers in providing personalized prompts and interventions before the peer evaluation stage, hence enhancing the evaluation quality through targeted interventions in a timely manner.

Triangulating natural language processing (NLP)-based analysis of rater comments and many-facet Rasch measurement (MFRM): An innovative approach to investigating raters’ application of rating scales in writing assessment

Rater comments tend to be qualitatively analyzed to indicate raters’ application of rating scales. This study applied natural language processing (NLP) techniques to quantify meaningful, behavioral information from a corpus of rater comments and triangulated that information with a many-facet Rasch measurement (MFRM) analysis of rater scores. The data consisted of ratings on 987 essays by 36 raters (a total of 3948 analytic scores and 1974 rater comments) on a post-admission English Placement Test (EPT) at a large US university. We computed a set of comment-based features based on the analytic components and evaluative language the raters used to infer whether raters were aligned to the scale. For data triangulation, we performed correlation analyses between the MFRM measures of rater performance and the comment-based measures. Although the EPT raters showed overall satisfactory performance, we found meaningful associations between rater comments and performance features. In particular, raters with higher precision and fit to what the Rasch model predicts used more analytic components and used evaluative language more similar to the scale descriptors. These findings suggest that NLP techniques have the potential to help language testers analyze rater comments and understand rater behavior.

Investigating The Central Tendency Effect Among Raters in Complex Problem-Solving Skills in Physics Essay Items: An Application of Many-Facet Rasch Measurement

The study sought to investigate the central tendency effect among raters in complex problem-solving skills in physics essay items using Many-Facet Rasch Measurement. This became necessary because the central tendency effect can affect students' scores. The research design adopted for this study is a descriptive research design based on the survey method. The population of the study comprised eighty-eight (88) public schools in all the local government areas with a physics student population of 3642 students and ninety-four (94) physics teachers in all the Senior Secondary Schools in Uyo Senatorial District for the 2022/2023 academic session. 364 SS3 physics students and 37 physics teachers from the 31 selected secondary schools in Uyo Senatorial District were sampled using a multistage sampling technique for effective selection. The multi-stage sampling technique was adopted for the study. The instrument used for data collection was the Physics Achievement Test (PAT) obtained from WAEC and NECO 2020 Physics Essay items. However, the reliability coefficient of 0.91 and 0.90 respectively. The finding revealed that there was no central tendency effect error committed by individual raters when rating complex problem-solving skills in WAEC and NECO 2020 physics Essay Items. Also, reveals there is a significant difference at the rater's group level exhibiting central tendency effect when rating complex problem-solving skills between WAEC and NECO 2020 physics Essay Items which implies that there is no group-level central tendency effect present among raters when rating complex problem-solving skills between WAEC and NECO 2020 physics Essay Items. The researchers concluded that rater effects are sources of variance in performance ratings that are associated with the rater's behaviour and not the actual performance of the ratee. It was recommended that to prevent the central tendency effect, it is important for superior raters to regularly observe their subordinates throughout the evaluation period and keep notes for future reference.

The Role of Time on Performance Assessment (Self, Peer and Teacher) in Higher Education: Rater Drift

This study aimed to investigate the change in teacher candidates' oral presentation skills over time through self, peer, and teacher assessments using the rater drift method. A longitudinal descriptive research model was used as a quantitative research approach to achieve this aim. The study group consisted of 47 teacher candidates receiving formation education at a state university in the Eastern Anatolia Region and an instructor teaching the course. An analytical rubric was used as a data collection tool to evaluate the candidates' oral presentation skills. The data collection process lasted six weeks in total. Since the performance evaluation process aimed to examine the change over time, the many-facet Rasch model was used. When the findings of the study were examined, it was determined that the rater behavior of teacher candidates had statistically significant differences at the group level over time. It was found that 26 out of 48 peer raters had rater drift in their evaluations. It was also found that the majority of rater drift over time was positive, meaning that evaluators became more generous over time. Another result obtained in the study was that teacher assessment did not show rater drift over time, with similar ratings for six weeks. The study’s findings were discussed with previous studies in the literature, and recommendations were made to researchers.

Investigating Rater Perceptions in the Assessment of Speaking

In the assessment of spoken production, numerous reasons can be identified behind the decisions that raters make in evaluating samples of oral performance. Inter and intra rater factors are relatively well documented in various reliability and validity studies. Some that have been identified in literature involve the effects of examinee pairing or the familiarity with the examinees, others point in the direction of gender and gender role perceptions O’Sullivan (2008), others appear to be connected with body language and non-verbal cues that accompany oral production (cf.: Krahmer and Swerts 2004, Seiter, Weger, Jensen and Kinzer 2010). While some studies that address the assessment of speaking English in exam contexts suggest that raters may not feel as comfortable assessing pronunciation as they do other aspects of a speaker’s performance (Orr 2002, Hubbard, Gilbert and Pidcock 2006, Brown 2006, De Velle 2008), more recent investigations of rater behaviour involving electronic evidence from training, maintenance and online examination programmes tentatively show that pronunciation, in fact, is the first category examiners attend to (Hubbard 2011, Chambers and Ingham 2011, Krakowian 2011, Seed 2012, Tynan 2015, Kang and Ginther 2019). This paper looks at large collection of assessments stored in an electronic system to investigate what raters really seem to pay attention to when allegedly following rating scales.

Interrater reliability estimators tested against true interrater reliabilities

BackgroundInterrater reliability, aka intercoder reliability, is defined as true agreement between raters, aka coders, without chance agreement. It is used across many disciplines including medical and health research to measure the quality of ratings, coding, diagnoses, or other observations and judgements. While numerous indices of interrater reliability are available, experts disagree on which ones are legitimate or more appropriate.Almost all agree that percent agreement (ao), the oldest and the simplest index, is also the most flawed because it fails to estimate and remove chance agreement, which is produced by raters’ random rating. The experts, however, disagree on which chance estimators are legitimate or better. The experts also disagree on which of the three factors, rating category, distribution skew, or task difficulty, an index should rely on to estimate chance agreement, or which factors the known indices in fact rely on.The most popular chance-adjusted indices, according to a functionalist view of mathematical statistics, assume that all raters conduct intentional and maximum random rating while typical raters conduct involuntary and reluctant random rating. The mismatches between the assumed and the actual rater behaviors cause the indices to rely on mistaken factors to estimate chance agreement, leading to the numerous paradoxes, abnormalities, and other misbehaviors of the indices identified by prior studies.MethodsWe conducted a 4 × 8 × 3 between-subject controlled experiment with 4 subjects per cell. Each subject was a rating session with 100 pairs of rating by two raters, totaling 384 rating sessions as the experimental subjects. The experiment tested seven best-known indices of interrater reliability against the observed reliabilities and chance agreements. Impacts of the three factors, i.e., rating category, distribution skew, and task difficulty, on the indices were tested.ResultsThe most criticized index, percent agreement (ao), showed as the most accurate predictor of reliability, reporting directional r2 = .84. It was also the third best approximator, overestimating observed reliability by 13 percentage points on average. The three most acclaimed and most popular indices, Scott’s π, Cohen’s κ and Krippendorff’s α, underperformed all other indices, reporting directional r2 = .312 and underestimated reliability by 31.4 ~ 31.8 points. The newest index, Gwet’s AC1, emerged as the second-best predictor and the most accurate approximator. Bennett et al’s S ranked behind AC1, and Perreault and Leigh’s Ir ranked the fourth both for prediction and approximation. The reliance on category and skew and failure to rely on difficulty explain why the six chance-adjusted indices often underperformed ao, which they were created to outperform. The evidence corroborated the notion that the chance-adjusted indices assume intentional and maximum random rating while the raters instead exhibited involuntary and reluctant random rating.ConclusionThe authors call for more empirical studies and especially more controlled experiments to falsify or qualify this study. If the main findings are replicated and the underlying theories supported, new thinking and new indices may be needed. Index designers may need to refrain from assuming intentional and maximum random rating, and instead assume involuntary and reluctant random rating. Accordingly, the new indices may need to rely on task difficulty, rather than distribution skew or rating category, to estimate chance agreement.

Assessment as Learning: How Does Peer Assessment Function in Students' Learning?

Peer assessment (PA) is employed as one fundamental practice of classroom-based assessment in terms of its learning-oriented and formative nature. The exercise of peer assessment has multiple and additional benefits for student learning. However, research into the learning processes in peer assessment is scarce both in theory and in practice, making it difficult to evaluate and pinpoint its value as a tool in assessment as learning (AaL). This study focuses both on the learning process and outcome through assessment activities. We set out with three goals in mind: (1) to examine students' assessment performance in context, (2) to evaluate its impact on student progress, and (3) to illuminate teachers on organizing assessment activities. Three specific research questions are answered in this study: (1) How do student raters perform in the process of PA in an advanced English class? (2) To what extent do assessment activities influence the students' speaking ability? (3) What are students' perceptions of PA regarding its benefits and caveats? A total of 29 undergraduate students participated in two assessment activities on argumentative speaking. Many-Facet Rasch Model analysis was conducted to measure the rater effects both at the group level and the individual level. Bias/interaction analyses were performed to diagnose rater behavior in different contexts including the rating session, speaking session, and peer assessment vs. self-assessment. Questionnaire and semi-structured interview data were also collected to explore factors and strategies that could interfere with PA as AaL. Results show that students exhibited stable rating behavior and made progress in argumentative speaking in all dimensions, including delivery, organization, and language use. They are more stringent with themselves than with peers although there is one rare case with a bias against peers. Participants acknowledged the benefits of PA but also shared reasonable concerns in practice. This study validated the feasibility and the effectiveness of PA for student learning. Discussion on findings and guidelines for effective implementation of PA as AaL are provided.

Holistic versus analytic scoring of spoken-language interpreting: a multi-perspectival comparative analysis

ABSTRACT Rubric scoring has been gaining traction as an emergent method to assess spoken-language interpreting, with two of the most well-known methods being rating scale-based holistic and analytic scoring. While the former provides a single global score, the latter generates separate scores on different dimensions of interpreting performance. Despite the growing use of the two methods, there has been little research documenting their uses in interpreting assessment. We therefore conducted the present study to find out how scoring methods (i.e. holistic versus analytic) would affect the dependability of rater-generated scores, rater behaviour, assessment outcomes, and rater perceptions. Overall, our quantitative data analysis indicates that although the two methods rank-ordered performances similarly, the holistic scoring led to relatively higher score dependability, regardless of interpreting directions, and that the raters’ assessments of interpreting into their less dominant language were less dependable. Our content analysis of the qualitative data reveals raters’ concerns with the substantive meaning of holistic scores and the design of analytic descriptors. We discussed these findings in light of available literature on interpreting assessment. By doing so, we hope to provide some evidential basis for scale selection in rater-mediated assessment of spoken-language interpreting.

How do raters learn to rate? Many-facet Rasch modeling of rater performance over the course of a rater certification program

This study employed a mixed-methods approach to examine how rater performance develops during a semester-long rater certification program for an English as a Second Language (ESL) writing placement test at a large US university. From 2016 to 2018, we tracked three groups of novice raters ( n = 30) across four rounds in the certification program. Using many-facet Rasch modeling, rater performance was examined in terms of rater agreement, rater consistency, and rater severity. These measurement estimates of rating quality were subjected to multivariate analysis to examine whether and how rater performance changes across rounds. Rater comments on the essays were qualitatively analyzed to obtain a deeper understanding of how raters learn to use the scale over time. The quantitative results showed a non-linear, three-staged developmental pattern of rater performance for all three groups of raters. Findings of this study suggest that rater development resembles a learning curve similar to how one acquires a language and other skills. We argue that understanding the developmental pattern of rater behavior is crucial not only to understanding the effectiveness of rater training, but also to the investigation of rater cognition and development. We will also discuss the practical implications of this study in relation to the effort and expectations needed for rater training for writing assessments.

Generalizability of Writing Scores and Language Program Placement Decisions: Score Dependability, Task Variability, and Score Profiles on an ESL Placement Test

Second Language (L2) testing has increasingly relied on performance assessment to evaluate practical command of language acquired. However, such forms of assessment entail more complex task design and subjective human scoring judgment (Bachman, 2004), raising challenges for score dependability and score use due to variability associated with task design (Deville & Chalhoub-Deville, 2006; In’nami, & Koizumi, 2016), differences in rater behavior (Bachman, Lynch, & Mason, 1995), and rating rubric functionality, especially when consisting of multiple subscales (Grabowski & Lin, 2019; Sawaki, 2007, Xi, 2007). The current study illustrates the use of Multivariate Generalizability Theory (MG-Theory) analyses for examining score variability and dependability for written performance assessment on an ESL placement test, rated using an analytic rubric with three subscales. In particular, this study identified the presence of task-related variability that did reduce score dependability for the writing scores yielded from this test. By the same token, this variability could substantively be justified as an artifact of representing the construct of L2 writing ability in a sufficiently broad manner. Simply said, should we expect test takers to have equivalent levels of proficiency when writing a review of an experience as a customer and when writing an argumentative essay as a student?

Examining the Differential Rater Functioning in the Process of Assessing Writing Skills of Middle School 7th Grade Students

When students present writing tasks that require higher order thinking skills to work, one of the most important problems is scoring these writing tasks objectively. The fact that raters give scores below or above their performance based on several environmental factors affects the consistency of the measurements. Inconsistencies in scoring negatively affect the validity and reliability of student performance and cause the scores obtained to be questioned. In regard to the validity and reliability of these measurements, it is significant to identify the rater behavior and correct the sources of error. This study aims to analyze the differential rater functioning (DRF), which is one of the problematic rater behaviors, in evaluating compositions written by middle school 7th-grade students within the scope of the Turkish course. 86 students attending a public school were participated the study. Students' compositions were rated using an analytical rubric by 8 teachers from different institutions. In this correlational research, the many facet Rasch model was used, and five variables including students, raters’ and, students’ gender, students’ qualification, and evaluation criteria were examined. it was examined whether the raters show DRF on an individual and group basis based on the dual interaction analysis, including the gender of the student x rater and the student's competence x rater. The findings have revealed that DRF at the group level does not interfere with the measurements, while the individual level DRF is involved in the measurements. It was determined that the level of DRF mixing in the measurements of successful students was the lowest. Especially rigid and lenient raters were found to show DRF. In the present study, it was observed that the raters showing DRF was also the most lenient raters, while these raters did not show DRF in terms of the gender of the student.

Case study meta-analysis in the social sciences. Insights on data quality and reliability from a large-N case survey.

Meta-analytical methods face particular challenges in research fields such as social and political research, where studies often rest primarily on qualitative and case study research. In such contexts, where research findings are less standardized and amenable to structured synthesis, the case survey method has been proposed as a means of data generation and analysis. The method offers a meta-analytical tool to synthesize larger numbers of qualitative case studies, yielding data amenable to large-N analysis. However, resulting data is prone to specific threats to validity, including biases due to publication type, rater behaviour, and variable characteristics, which researchers need to be aware of. While these biases are well known in theory, and typically explored for primary research, their prevalence in case survey meta-analyses remains relatively unexplored. We draw on a case survey of 305 published qualitative case studies of public environmental decision-making, and systematically analyze these biases in the resultant data. Our findings indicate that case surveys can deliver high-quality and reliable results. However, we also find that these biases do indeed occur, albeit to a small degree or under specific conditions of complexity. We identify a number of design choices to mitigate biases that may threaten validity in case survey meta-analysis. Our findings are of importance to those using the case survey method - and to those who might apply insights derived by this method to inform policy and practice.

But how many push-ups can she do? The influence of sexism on peer ratings in a military setting

Perceived gender differences remain salient in occupational settings and biases that arise maintain inequalities. We examined rater behavior of male military cadets to test how hostile sexism impacted occupationally-relevant attributions, namely, perception of peers' physical fitness, social skill, and military demeanor, items from a larger rating system and previously linked to gender bias. Linear mixed models determined how ratings were influenced by raters' own beliefs or performance, ratee performance, and whether these relationships differed by ratee gender (N = 2520 raters and 4154 ratees). Men with sexist beliefs rated women lower on military demeanor and physical fitness. Sexism was further associated with rating women's physical fitness, more so than other men's, according to push-up scores, suggesting sexist men “keep track” of women's upper body capabilities. Physical fitness scores were positively related to military demeanor rating for women but unrelated to men's score, regardless of rater's sexism. Overall, ratings were associated with different performance characteristics for men vs. women, especially for physical abilities, suggesting a mechanism for expression of gender bias in a field setting.

Predictive Modeling of Rater Behavior: Implications for Quality Assurance in Essay Scoring

ABSTRACT We evaluate the feasibility of developing predictive models of rater behavior, that is, rater-specific models for predicting the scores produced by a rater under operational conditions. In the present study, the dependent variable is the score assigned to essays by a rater, and the predictors are linguistic attributes of the essays used by the e-rater® engine. Specifically, for each rater, the linear regression of rater scores on the linguistic attributes is obtained based on data from two consecutive time periods. The regression from each period was cross validated against data from the other period. Raters were characterized in terms of their level of predictability and the importance of the predictors. Results suggest that rater models capture stable individual differences among raters. To evaluate the feasibility of using rater models as a quality control mechanism, we evaluated the relationship between rater predictability and inter-rater agreement and performance on pre-scored essays. Finally, we conducted a simulation whereby raters are simulated to score exclusively as a function of essay length at different points during the scoring day. We concluded that predictive rater models merit further investigation as a means of quality controlling human scoring.

Analysis of Evaluator Reliability for the Raters' Calibration Training

This study analyzed the change in the rater reliability based on the student's practice evaluation process conducted at Yonsei University College of Dentistry. Through this, we suggest the significance of the rater calibration training in the student's practical evaluation of dental college. Nine professors from the department of Conservative Dentistry, Yonsei University College of Dentistry, analyzed the results of class II restoration cases twice in 2017 and once in 2018. Intra Class Correlation (ICC) which is a statistic used to determine the consistency of raters with three or more scores, was also calculated. ICC values increased as raters participated in rater calibration meetings and grading experiences. This shows that the rater reliability is related to the grading experience and feedback from calibration meeting. Based on the results of previous studies that grading experiences and rater calibration training can cause a meaningful change in rater behavior, we propose to conduct rater calibration training to ensure the evaluator reliability.

Rater Negotiation Scheme: How writing raters resolve score discrepancies

Abstract In practices of direct assessment of writing ability, the variability of human decision-making during scoring poses great challenges to the validity of assessment (Kane, 2006). The variables causing differences in individual raters’ scoring interpretations have been widely investigated (e.g. Eckes, 2012; Wolfe et al., 2016). However, the issue of how raters negotiate to resolve discrepancies has not received attention although rater negotiation is a widely used score resolution method. As it has been emphasized by scholars interested in the argumentative behavior of raters (e.g. Trace et al., 2017), a systematic analysis of score negotiations will enable us to analyze the dependability of score negotiations. The purpose of this study is twofold: to present a thorough analysis of the argumentative structure of rater discrepancy resolution discussions with a view to understanding their underlying dynamics, and to investigate whether the elements of the argumentative structure of negotiations differ from research settings to authentic score resolution practices. In line with this aim, rater negotiations following a written test at the language school of an English-medium university were analyzed within the framework of Argumentation Theory by Toulmin (1958) and Walton (2005, 2016). The negotiation data were obtained from 99 recorded rater discussions among 30EFL teachers, and transcribed, coded and categorized into argumentative discussion moves. A Rater Negotiation Scheme (RNS) was developed through a recursive data analysis and categorization process, and it was validated through field-testing in authentic settings. The findings have implications both for research on rater negotiations and arguments on the reliability of the method.

Investigation of Rater Tendencies and Reliability in Different Assessment Methods with Many Facet Rasch Model

One of the most commonly used methods for measuring higher-order thinking skills, such as problem-solving and written expression is open-ended items. Three main approaches are used to evaluate responses to open-ended items: general evaluation, rating scale, and the rubric. In order to measure and improve the problem-solving skills of the students, firstly, an error-free measurement process should be performed. Error caused by rater is a common problem in the evaluation of open-ended items. Errors caused by the rater, such as bias, high or low tendency to score, adversely affect the accuracy of decisions to be made. In this study, the raters' tendencies are evaluated in terms of general evaluation, rating scale, and rubric conditions used to evaluate open-ended items. The rater behaviors in each assessment method and the raters' opinions about the assessment methods were determined. The participants of the study consisted of 12 different mathematics teachers, and the analyses were based on the Many Facet Rasch Model. The scoring reliability of each method was estimated. When using the rating scale, it was concluded that the raters had a more homogeneous scoring tendency. In addition, while the majority of raters stated that they prefer to use rubric, the most difficult method to use was stated by the raters.

Investigation of the Use of Electronic Portfolios in the Determination of Student Achievement in Higher Education Using the Many-Facet Rasch Measurement Model

This study aimed to determine the rater behavior in the evaluation process of student electronic portfolios used to measure student achievement in higher education, and thus to evaluate the usability of the electronic portfolio system. Considering that rater behavior adversely affects both validity and reliability in determining the performance of individuals, it is important to identify the effect of this factor and evaluate the related results in line with this effect. The data of the study were collected from the students enrolled in an English language teaching program at Gazi University Gazi Education Faculty within the scope of the measurement and assessment course in the fall semester of 2017-2018. An analytic rubric developed by the researchers was used in the evaluation of the student electronic portfolios. The study included two participants groups consisting of three raters and 61 students (11 male, 50 female). In the analysis of the data, the many-facet Rasch measurement model was used as an analysis method since it was appropriate for the nature of the current data set. When the findings of the study were examined, it was found that one or more rater behaviors interfered with the performance of the individual in the use of non-objective measurement tools, and consequently negatively affected the validity and reliability of the measurements. In conclusion, it can be stated that the individual’s performance related to electronic portfolios in higher education is generally affected by the rater behavior in the evaluation process independent of the measurement tool. In addition, it has been confirmed that electronic portfolios can be used to determine individual performance in higher education.

A new coefficient of interrater agreement: The challenge of highly unequal category proportions.

We derive a general structure that encompasses important coefficients of interrater agreement such as the S-coefficient, Cohen's kappa, Scott's pi, Fleiss' kappa, Krippendorff's alpha, and Gwet's AC1. We show that these coefficients share the same set of assumptions about rater behavior; they only differ in how the unobserved category proportions are estimated. We incorporate Bayesian estimates of the category proportions and propose a new agreement coefficient with uniform prior beliefs. To correct for guessing in the process of item classification, the new coefficient emphasizes equal category probabilities if the observed frequencies are unstable due to a small sample, and the frequencies increasingly shape the coefficient as they become more stable. The proposed coefficient coincides with the S-coefficient for the hypothetical case of zero items; it converges to Scott's pi, Fleiss' kappa, and Krippendorff's alpha as the number of items increases. We use simulation to show that the proposed coefficient is as good as extant coefficients if the category proportions are equal and that it performs better if the category proportions are substantially unequal. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

Pedagogical Considerations for Examining Rater Variability in Rater‐Mediated Assessments: A Three‐Model Framework

AbstractRater‐mediated assessments are a common methodology for measuring persons, investigating rater behavior, and/or defining latent constructs. The purpose of this article is to provide a pedagogical framework for examining rater variability in the context of rater‐mediated assessments using three distinct models. The first model is the observation model, which includes ecological/environmental considerations for the evaluation system. The second model is the measurement model, which includes the transformation of observed, rater response data to linear measures using a measurement model with specific requirements of rater‐invariant measurement in order to examine raters’ construct‐relevant variability stemming from the evaluative system. The third model is the interaction model, which includes an interaction parameter to allow for the investigation into raters’ systematic, construct‐irrelevant variability stemming from the evaluative system. Implications for measurement outcomes and validity are discussed.