Rater Drift Research Articles

New Tests of Rater Drift in Trend Scoring

ABSTRACT Trend scoring constructed response items (i.e. rescoring Time A responses at Time B) gives rise to two-way data that follow a product multinomial distribution rather than the multinomial distribution that is usually assumed. Recent work has shown that the difference in sampling model can have profound negative effects on statistics usually used to monitor rater drift. In this paper, three statistics, termed E-statistics, that account for the product-multinomial sampling by comparing conditional distributions, are introduced. A simulation compares performance with the paired t-test and Stuart’s Q in detecting rater drift. Both the paired t-test and Q suffered extreme Type I error inflation for certain rescore study designs. The new E-statistics maintained good Type I error control and had good power to detect rater drift across occasions.

The Role of Time on Performance Assessment (Self, Peer and Teacher) in Higher Education: Rater Drift

This study aimed to investigate the change in teacher candidates' oral presentation skills over time through self, peer, and teacher assessments using the rater drift method. A longitudinal descriptive research model was used as a quantitative research approach to achieve this aim. The study group consisted of 47 teacher candidates receiving formation education at a state university in the Eastern Anatolia Region and an instructor teaching the course. An analytical rubric was used as a data collection tool to evaluate the candidates' oral presentation skills. The data collection process lasted six weeks in total. Since the performance evaluation process aimed to examine the change over time, the many-facet Rasch model was used. When the findings of the study were examined, it was determined that the rater behavior of teacher candidates had statistically significant differences at the group level over time. It was found that 26 out of 48 peer raters had rater drift in their evaluations. It was also found that the majority of rater drift over time was positive, meaning that evaluators became more generous over time. Another result obtained in the study was that teacher assessment did not show rater drift over time, with similar ratings for six weeks. The study’s findings were discussed with previous studies in the literature, and recommendations were made to researchers.

Examining The Rater Drift in The Assessment of Presentation Skills in Secondary School Context

The alternative assessment, including peer assessment, helps students develop metacognition among the sub-categories of assessment types. Despite the advantage of alternative assessment, reliability and validity issues are the most significant problems in alternative assessment. This study investigated the rater drift, one of the rater effects, in peer assessment. The performance of 8 oral presentations based on group work in the Science and Technology course was scored by 7th-grade students (N=28) using the rubric researchers developed. The presentations lasted for four days, with two presentations each day. While examining the time-dependent drift in rater severity in peer assessment, the many-Facet Rasch Measurement model was used. Two indexes (interaction term and standardized differences) were calculated with many-facet Rasch measurement to determine the raters who made rater drift either individually or as a group. The analysis examined the variance of scores in the following days compared to the first day’s scores. Accordingly, the two methods used to determine rater drift gave similar results, and some raters at the individual level tended to be more severe or lenient over time. However, no significant rater drift at the group level showed that drifts had no specific models.

Automated essay scoring (AES) of constructed responses in nursing examinations: An evaluation

Nursing students’ higher-level thinking skills are ideally assessed through constructed-response items. At the baccalaureate level in North America, however, this exam format has largely fallen into disuse owing to the labor-intensive process of scoring written exam papers. The authors sought to determine if automated essay scoring (AES) would be an efficient and reliable alternative to human scoring. Four constructed-response exam items were administered to an initial cohort of 359 undergraduate nursing students in 2016 and to a second cohort of 40 students in 2018. The items were graded by two human raters (HR1 & HR2) and an AES software platform. AES approximated or surpassed agreement and reliability measures achieved by the HR1 and HR2 with each other, and AES surpassed both human raters in efficiency. A list of answer keywords was created to increase the efficiency and reliability of AES. Low agreement between human raters may be explained by rater drift and fatigue, and shortcomings in the development of Item 1 may have reduced its overall agreement and reliability measures. It can be concluded that AES is a reliable and cost-effective means of scoring constructed-response nursing examinations, but further studies employing greater sample sizes are needed to establish this definitively.

A Mixed Method Program Evaluation of Annual Inspections Conducted in Childcare Programs in Washington State

This mixed method study used a program evaluation to assess the reliability and social validity of the focused childcare monitoring checklist used in Washington State, as well as its social validity in maintaining quality programming in licensed childcare centers. The focused monitoring checklist and interview responses were used to answer two specific research questions: (1) How do stakeholders describe the value, usefulness, and effects of state administrated focused monitoring?; and (2) What is the inter-rater reliability of the focused monitoring tool used to assess the foundational health and safety issues that must be met by state licensed early childhood programs? The study found that licensors and providers found the focused monitoring tool as more efficient and informative than the current differential monitoring system. The use of a checklist focusing on real time compliance increased the value placed on the relevance of the inspection with respect to meeting licensor and provider needs. The results also showed that even with a controlled tool, performance of onsite inspections can vary greatly along a continuum of reliability and objectivity due to licensor rater drift and individual perceptions of licensing procedures. Licensing agencies should consider further evaluation of the monitoring process and the reliability of the checklist tool as the process is implemented statewide, concentrating on the training content and training methods provided to licensors.
 https://doi.org/10.21423/jrs-v08stevens

Examining the Calibration Process for Raters of the GRE® General Test

AbstractOne of the challenges in scoring constructed‐response (CR) items and tasks is ensuring that rater drift does not occur during or across scoring windows. Rater drift reflects changes in how raters interpret and use established scoring criteria to assign essay scores. Calibration is a process used to help control rater drift and, as such, serves as a type of quality control during CR scoring. Calibration sets are designed to provide sufficient evidence that raters have understood and internalized the rubrics and can score accurately across all score points of the score scale. This study examined the calibration process used to qualify raters to score essays from the GRE® Analytical Writing measure. A total of 46 experienced raters participated in the study, and each rater scored up to 630 essays from 1 of 2 essay prompt types. Two research questions were evaluated: Does calibration influence scoring accuracy? and Does reducing the frequency of calibration impact scoring accuracy? While the distribution of score points represented by the essays used in the study did not necessarily reflect what raters see during operational scoring, results suggest that the influence of calibration on Day 1 remains with raters through at least 3 scoring days. Results further suggest that scoring accuracy may be moderated by prompt type. Nevertheless, study results indicate that daily calibration for GRE prompt types may not be necessary and that reducing the frequency of calibration is unlikely to reduce scoring accuracy.

On the Performance of the Marginal Homogeneity Test to Detect Rater Drift.

When constructed response items are administered repeatedly, "trend scoring" can be used to test for rater drift. In trend scoring, raters rescore responses from the previous administration. Two simulation studies evaluated the utility of Stuart's Q measure of marginal homogeneity as a way of evaluating rater drift when monitoring trend scoring. In the first study, data were generated based on trend scoring tables obtained from an operational assessment. The second study tightly controlled table margins to disentangle certain features present in the empirical data. In addition to Q, the paired t test was included as a comparison, because of its widespread use in monitoring trend scoring. Sample size, number of score categories, interrater agreement, and symmetry/asymmetry of the margins were manipulated. For identical margins, both statistics had good Type I error control. For a unidirectional shift in margins, both statistics had good power. As expected, when shifts in the margins were balanced across categories, the t test had little power. Q demonstrated good power for all conditions and identified almost all items identified by the t test. Q shows substantial promise for monitoring of trend scoring.

Inter-Rater Reliability and Intra-Rater Reliability of Assessing the 2-Minute Push-Up Test.

The purpose of this study was to assess inter-rater reliability and intra-rater reliability of the 2-minute, 90° push-up test as utilized in the Army Physical Fitness Test. Analysis of rater assessment reliability included both total score agreement and agreement across individual push-up repetitions. This study utilized 8 Raters who assessed 15 different videotaped push-up performances over 4 iterations separated by a minimum of 1 week. The 15 push-up participants were videotaped during the semiannual Army Physical Fitness Test. Each Rater randomly viewed the 15 push-up and verbally responded with a "yes" or "no" to each push-up repetition. The data generated were analyzed using the Pearson product-moment correlation as well as the kappa, modified kappa and the intra-class correlation coefficient (3,1). An attribute agreement analysis was conducted to determine the percent of inter-rater and intra-rater agreement across individual push-ups.The results indicated that Raters varied a great deal in assessing push-ups. Over the 4 trials of 15 participants, the overall scores of the Raters varied between 3.0 and 35.7 push-ups. Post hoc comparisons found that there was significant increase in the grand mean of push-ups from trials 1-3 to trial 4 (p < 0.05). Also, there was a significant difference among raters over the 4 trials (p < 0.05). Pearson correlation coefficients for inter-rater and intra-rater reliability identified inter-rater reliability coefficients were between 0.10 and 0.97. Intra-rater coefficients were between 0.48 and 0.99. Intra-rater agreement for individual push-up repetitions ranged from 41.8% to 84.8%. The results indicated that the raters failed to assess the same push-up repetition with the same score (below 70% agreement) as well as failed to agree when viewed between raters (29%). Interestingly, as previously mentioned, scores on trial 4 increased significantly which might have been caused by rater drift or that the Raters did not maintain the push-up standard over the trials. It does appear that the final push-up scores received by each participant was a close approximation of actual performance (within 65%) but when assessing physical performance for retention in the Army, a more reliable test might be considered.

Trends in Classroom Observation Scores.

Observations and ratings of classroom teaching and interactions collected over time are susceptible to trends in both the quality of instruction and rater behavior. These trends have potential implications for inferences about teaching and for study design. We use scores on the Classroom Assessment Scoring System-Secondary (CLASS-S) protocol from 458 middle school teachers over a 2-year period to study changes over time in (a) the average quality of teaching for the population of teachers, (b) the average severity of the population of raters, and (c) the severity of individual raters. To obtain these estimates and assess them in the context of other factors that contribute to the variability in scores, we develop an augmented G study model that is broadly applicable for modeling sources of variability in classroom observation ratings data collected over time. In our data, we found that trends in teaching quality were small. Rater drift was very large during raters' initial days of observation and persisted throughout nearly 2 years of scoring. Raters did not converge to a common level of severity; using our model we estimate that variability among raters actually increases over the course of the study. Variance decompositions based on the model find that trends are a modest source of variance relative to overall rater effects, rater errors on specific lessons, and residual error. The discussion provides possible explanations for trends and rater divergence as well as implications for designs collecting ratings over time.

The Effect of Observation Length and Presentation Order on the Reliability and Validity of an Observational Measure of Teaching Quality

Observational methods are increasingly being used in classrooms to evaluate the quality of teaching. Operational procedures for observing teachers are somewhat arbitrary in existing measures and vary across different instruments. To study the effect of different observation procedures on score reliability and validity, we conducted an experimental study that manipulated the length of observation and order of presentation of 40-minute videotaped lessons from secondary grade classrooms. Results indicate that two 20-minute observation segments presented in random order produce the most desirable effect on score reliability and validity. This suggests that 20-minute occasions may be sufficient time for a rater to observe true characteristics of teaching quality assessed by the measure used in the study, and randomizing the order in which segments were rated may reduce construct irrelevant variance arising from carry over effects and rater drift.

Using Automated Essay Scores as an Anchor When Equating Constructed Response Writing Tests

Assessments consisting of only a few extended constructed response items (essays) are not typically equated using anchor test designs as there are typically too few essay prompts in each form to allow for meaningful equating. This article explores the idea that output from an automated scoring program designed to measure writing fluency (a common objective of many writing prompts) can be used in place of a more traditional anchor. The linear-logistic equating method used in this article is a variant of the Tucker linear equating method appropriate for the limited score range typical of essays. The procedure is applied to historical data. Although the procedure only results in small improvements over identity equating (not equating prompts), it does produce a viable alternative, and a mechanism for checking that the identity equating is appropriate. This may be particularly useful for measuring rater drift or equating mixed format tests.

Training and Maintaining System-Wide Reliability in Outcome Management

The Child and Adolescent Functional Assessment Scale (CAFAS) is widely used for outcome management, for providing real time client and program level data, and the monitoring of evidence-based practices. Methods of reliability training and the assessment of rater drift are critical for service decision-making within organizations and systems of care. We assessed two approaches for CAFAS training: external technical assistance and internal technical assistance. To this end, we sampled 315 practitioners trained by external technical assistance approach from 2,344 Ontario practitioners who had achieved reliability on the CAFAS. To assess the internal technical assistance approach as a reliable alternative training method, 140 practitioners trained internally were selected from the same pool of certified raters. Reliabilities were high for both practitioners trained by external technical assistance and internal technical assistance approaches (.909–.995, .915–.997, respectively). 1 and 3-year estimates showed some drift on several scales. High and consistent reliabilities over time and training method has implications for CAFAS training of behavioral health care practitioners, and the maintenance of CAFAS as a global outcome management tool in systems of care.

P-641 - The importance of rigor in post-baseline assessments in cns clinical trials

Inappropriate subjects may be enrolled in a study when enrollment pressures cause inflated baseline severity scores. An increasing number of studies now include methods such as blinded independent centralized ratings (CR) to ensure that appropriate subjects are entered into the trial. Post-baseline factors such as functional unblinding, expectation bias and rater drift can also affect outcomes. Independent raters, blind to study visit, can minimize functional unblinding and expectation bias. Continuous calibration of CR can minimize rater drift. To examine studies with both site ratings (SR) and CR to determine how critical post-baseline blinding and continuous calibration are. A trial of acute schizophrenia used CR for the PANSS and SR for the BPRS on the same subjects. A Parkinson's psychosis study used CR in the US and SR ex-US to assess subjects using the SAPS. A GAD trial used CR of subjects enrolled by SRs’ SIGH-A evaluations. In the schizophrenia trial, CR separated the active comparator and one of two test arms. SR separated the active comparator but neither test arm. In the Parkinson's psychosis study, pimavanserin showed greater separation with CR than SR. In the GAD trial, CR had lower placebo response than SR, independent of subject selection. Data from several studies support the continued importance of rater blinding and independence, post subject selection. Results suggest that precision of ratings beyond baseline can increase the sensitivity of findings in a clinical trial, decrease placebo response rates and potentially eliminate Type II errors.

Rater Effects on Essay Scoring: A Multilevel Analysis of Severity Drift, Central Tendency, and Rater Experience

This study examined rater effects on essay scoring in an operational monitoring system from England's 2008 national curriculum English writing test for 14-year-olds. We fitted two multilevel models and analyzed: (1) drift in rater severity effects over time; (2) rater central tendency effects; and (3) differences in rater severity and central tendency effects by raters’ previous rating experience. We found no significant evidence of rater drift and, while raters with less experience appeared more severe than raters with more experience, this result also was not significant. However, we did find that there was a central tendency to raters’ scoring. We also found that rater severity was significantly unstable over time. We discuss the theoretical and practical questions that our findings raise.

P.2.a.007 Quantifying rater drift on the HAM-D: implications for reliability, sample size, and ongoing training strategy

P.2.a.007 Quantifying rater drift on the HAM-D: implications for reliability, sample size, and ongoing training strategy

Interrater Reliability of Using Brief Standardized Outcome Measures in a Community Mental Health Setting

Given psychiatry's need to implement measurement-based care, the study examined whether direct-care staff could reliably administer brief positive and negative symptom instruments to track symptom changes and inform clinical decision making. Raters (82 case managers) were assessed at baseline. Training was provided for individuals not meeting reliability criteria. These individuals were reassessed to determine the effect of training. In addition, rater drift was assessed for raters judged to be reliable at baseline. Seventy-seven percent of direct-care staff met criteria for reliability either at baseline or after they received additional training. A majority of direct-care staff can be trained to reliability on brief scales of positive and negative symptoms that can be used to guide clinical decision making.

Interrater Reliability of Using Brief Standardized Outcome Measures in a Community Mental Health Setting

Given psychiatry's need to implement measurement-based care, the study examined whether direct-care staff could reliably administer brief positive and negative symptom instruments to track symptom changes and inform clinical decision making.Raters (82 case managers) were assessed at baseline. Training was provided for individuals not meeting reliability criteria. These individuals were reassessed to determine the effect of training. In addition, rater drift was assessed for raters judged to be reliable at baseline.Seventy-seven percent of direct-care staff met criteria for reliability either at baseline or after they received additional training.A majority of direct-care staff can be trained to reliability on brief scales of positive and negative symptoms that can be used to guide clinical decision making.

P02-88 - Quantifying rater drift on the HAM-D in a sample of standardized rater training events: Implications for reliability and sample size calculations

Introduction/objectives/aimsThough rater drift in clinical trials has long been understood to negatively impact trial results, few studies have systematically quantified this. We examined training data for the HAM-D (Hamilton Depression Scale, 17-item version) at two time points to measure the impact.MethodsRaters participating in a standardized training scored the HAM-D based on two videotaped interviews of depressed patients. To assess drift, data from an initial, post-online training session was compared to data obtained 12 months later. Intra-class correlation coefficients (Shrout &amp; Fliess, 1979) and concordance with expert ratings were compared.ResultsIntra-class correlation coefficients (ICC) for raters (n = 167) following initial training were good to excellent for individual raters (.695–.976, p &lt; .0001) and good for the overall cohort (.752, p &lt; .0001). Concordance with expert ratings was excellent at 99.3%. The overall ICC fell to .730 at the second assessment and although the upper bound of individual performance remained in the good to excellent range, the frequency of scores in the poor to fair range (&lt; .65) increased. Concordance also fell slightly to 87%.ConclusionsRater drift occurred over 12 months, as gauged by the metrics of reliability and concordance. Drift was apparent in a limited portion of the cohort but resulted in a lower overall ICC at the second time point. Because studies are generally powered assuming that the ICC remains stable, there are implications for both this power calculation and the required sample size.

Inaccuracy in Clinical Trials: Effects and Methods to Control Inaccuracy

The increasing rate of failed trails found in mood and anxiety disorders is now being seen in Alzheimer's studies. Factors related to the administration of clinician rating scales, such as poor inter-rater reliability, poor interview quality and rater bias may be a contributing factor. Studies have found inter-rater reliability to be problematic in Alzheimer's studies, even with less subjective outcome measures. Lack of standardization of administration and scoring procedures has been identified as a major contributing factor. Remediation through better training procedures has been found to be successful, although ongoing calibration is needed to prevent rater drift. Expectancy bias and baseline score inflation is more difficult to remediate. Inflation of baseline scores increases placebo response, since lower severity has been found to be associated with higher placebo response. The use of centralized raters that are independent from study sites may help ameliorate these issues. Increased methodological research examining new approaches to these problems is warranted. The increased costs associated with this research should offset the time and expense of continuing with 'business as usual'.

P.3.f.002 Quantifying rater drift in a sample of standardised rater training events: Is PANSS reliability maintained over time?

P.3.f.002 Quantifying rater drift in a sample of standardised rater training events: Is PANSS reliability maintained over time?