Rater Judgments Research Articles

Investigating level descriptors and their impact on validity

This article presents a validation study which investigates the relationship between mastery level descriptors and item difficulty in the National Tests of English (NTE) in Norway. The aim is to establish the extent to which the descriptors indicate item difficulty and thus support the argument that the mastery levels are a reflection of the framework of the NTE. This argument strength has direct implications for validity and for the defensibility of moving from criterion-based descriptors to norm-based national data. The study involved a panel of 10 raters assigning level descriptors from 7 content categories to 80 test items, giving a total of 5,600 individual judgements. These judgements are compared to the real test scores of around 46,000 pupils to establish if the level descriptors assigned by raters can predict pupil performance on the tests. The results show a strong correlation between rater judgements and real test scores, meaning that the descriptors offer an indication of item difficulty. However, some individual descriptor categories contain deviations from the expected order. We conclude that, while the level descriptors reflect the test’s framework, and thus support the validity argument, the argument could be strengthened with the revision of some individual descriptors.

Detecting Rater Bias in Mixed-Format Assessments

ABSTRACT Mixed-format assessments made up of multiple-choice (MC) items and constructed response (CR) items that are scored using rater judgments include unique psychometric considerations. When these item types are combined to estimate examinee achievement, information about the psychometric quality of each component can depend on that of the other. For example, the presence of differential item functioning (DIF) in MC items could compromise the sensitivity of rater effect indicators, such as differential rater functioning (DRF) indices. Likewise, the presence of DRF could compromise the sensitivity of DIF indices. We used real data and a simulation study to consider the impact of DIF on DRF indices, and the impact of DRF on DIF indices. Results indicate some interaction between DIF and DRF that varies across data collection designs. We consider the implications of these results for research and practice.

Detecting Deception Using MCI in Twitter

The purpose of the study was to test whether Modified Cognitive Interviewing (MCI) is an effective method for detecting deceptive human eyewitness accounts in a computer-mediated computer platform with limited text space (such as in Twitter). 44 college students were the participants in this study, where they either had to perform, or pretend that they had performed a cognitive task. Of the 44 participants, 15 performed the task and reported truthfully about their activities; 16 performed the task and denied having participated in the task; 13 read the instructions about the cognitive task and when interviewed claimed to have actually performed the task. The transcripts of interviews, conducted in Twitter, were rated by individuals trained in cognitive interviewing; forensic speech variables (response length (RL), unique word (UW) count and type-token ratio (TTR) were coded from transcripts. Human rater judgments and computer-based speech analysis performed better than chance; computer based judgments were superior to the human judgments (i.e., 79% vs. 54%, respectively). Speech content variables derived from MCI differed significantly, and in different ways, between the truthful and false claimant participants and also between the truthful and denial type participants. MCI derived statement analysis methods are a scientifically valid method, when used in Twitter, that can be used by professionals tasked with distinguishing between true claims, false claims and denials.

Using the analytical hierarchical process for programme design decisions: A disability case study

Background: A care facility for people with disabilities struggles to obtain financial support for its Parent Education and Support Programme. The programme’s design includes two implementers, an occupational therapist and a community-based worker, increasing its core costs. To enhance the likelihood of donor support, the Facility considered choosing the best-suited implementer for the programme. To help inform this decision, a formal methodological approach to high-level decision-making called multi-criteria decision analysis (MCDA) was utilised.Objective: Through a case study, this paper demonstrates how the MCDA methodology, using the analytical hierarchical process (AHP), was applied in a programme evaluation context.Method: Decision models were constructed using the AHP MCDA method and elicited rater judgments. Raters were drawn from four stakeholder groups: Programme beneficiaries, management, donors, and experts in disability and rehabilitation. This was followed by assigning criteria weights, establishing local priorities for each alternative, and aggregating the judgments. The model was then synthesised, and a sensitivity analysis was conducted.Results: The findings revealed that specific outcomes were attributed to each implementer, and thus, deciding to employ only one implementer would have had serious consequences for the programme’s quality and the achievement of intended outcomes.Conclusion: The results confirmed the usefulness of AHP MCDA for programme design decisions.Contribution: This article contributes by enhancing the understanding of the AHP MCDA methodology. Secondly, it demonstrates the suitability of this methodology for programme designers, evaluators, or non-profit organisations (NPOs) who need to make informed decisions about the design and implementation of interventions.

Rater judgments and word difficulty: Conceptualizing the substantive validity of the VST

The substantive component of construct validity requires a confrontation between empirical test results and content relevance. The Vocabulary Size Test (VST) has been extensively validated in terms of empirical results. Less is known, however, about expert judgments of content relevance. The VST was constructed and validated according to the principle that frequency underlies vocabulary acquisition. This does not mean, however, that the two are equivalent. To better understand the construct of word difficulty as it is measured in VSTs, the results of two Rasch analyses, one on the VST, the other a Multi Facet Rasch Model (MFRM) of a questionnaire distributed to education practitioners were correlated (n = 80, r = 0.67, p < 0.001, 95% CI = [0.53, 0.77]). Semi-structured interviews were then conducted to explore how practitioners understand the concept of word difficulty. Findings indicate that word difficulty is understood to encompass more than frequency, validating recent research into the predictive power of lexical sophistication variables.

A sequential approach to detecting differential rater functioning in sparse rater-mediated assessment networks

Researchers frequently evaluate rater judgments in performance assessments for evidence of differential rater functioning (DRF), which occurs when rater severity is systematically related to construct-irrelevant student characteristics after controlling for student achievement levels. However, researchers have observed that methods for detecting DRF may be limited in sparse rating designs, where it is not possible for every rater to score every student. In these designs, there is limited information with which to detect DRF. Sparse designs can also exacerbate the impact of artificial DRF, which occurs when raters are inaccurately flagged for DRF due to statistical artifacts. In this study, a sequential method is adapted from previous research on differential item functioning (DIF) that allows researchers to detect DRF more accurately and distinguish between true and artificial DRF. Analyses of data from a rater-mediated writing assessment and a simulation study demonstrate that the sequential approach results in different conclusions about which raters exhibit DRF. Moreover, the simulation study results suggest that the sequential procedure results in improved accuracy in DRF detection across a variety of rating design conditions. Practical implications for language testing research are discussed.

Beyond Agreement: Exploring Rater Effects in Large-Scale Mixed Format Assessments

ABSTRACT Scoring procedures for the constructed-response (CR) items in large-scale mixed-format educational assessments often involve checks for rater agreement or rater reliability. Although these analyses are important, researchers have documented rater effects that persist despite rater training and that are not always detected in rater agreement and reliability analyses, such as severity/leniency, centrality/extremism, and biases. Left undetected, these effects pose threats to fairness. We illustrate how rater effects analyses can be incorporated into scoring procedures for large-scale mixed-format assessments. We used data from the National Assessment of Educational Progress (NAEP) to illustrate relatively simple analyses that can provide insight into patterns of rater judgment that may warrant additional attention. Our results suggested that the NAEP raters exhibited generally defensible psychometric properties, while also exhibiting some idiosyncrasies that could inform scoring procedures. Similar procedures could be used operationally to inform the interpretation and use of rater judgments in large-scale mixed-format assessments.

Adding Adaptation/Modification Size and Scope to Frameworks for Classifying Changes to an Intervention.

Frameworks for classifying adaptations and modifications to an intervention have been developed and continue to evolve. These frameworks capture a range of attributes of adaptations and modifications. However, the size and the scope of a change, relative to the intervention's overall size and structure, are not currently included. This omission can have negative consequences, especially for conclusions about how different kinds of changes to an intervention may influence implementation quality and intervention outcomes. Coding the relative size and scope of an adaptation or modification should be simple when the original intervention is described in detail. For interventions that are not manualized, coding may alternatively be based on rater judgments. Attending to the size and the scope of adaptations and modifications could have an ancillary benefit, generating increased conceptual and empirical work on procedures for identifying the distinct components and sub-elements of an intervention.

A Model‐Data‐Fit‐Informed Approach to Score Resolution in Performance Assessments

AbstractMany large‐scale performance assessments include score resolution procedures for resolving discrepancies in rater judgments. The goal of score resolution is conceptually similar to person fit analyses: To identify students for whom observed scores may not accurately reflect their achievement. Previously, researchers have observed that rater‐agreement methods and person fit analyses result in similar conclusions about which students’ achievement estimates warrant additional investigation, and that score resolution generally improves person fit. We consider the implications of using person fit analysis as an initial step to identify performances for score resolution, and of using fit indices to identify raters to provide additional ratings. We simulated student responses to multiple‐choice items and a writing task. We simulated various types of person misfit in the writing task and identified the persons who needed resolution using a person‐fit index. Results indicate larger improvements in person fit after resolution when the person fit approach was used compared to a rater‐agreement approach. With the fit‐informed approach, person fit improved for of the misfitting students; with the rater‐agreement approach, person fit improved for around one third of these students. We consider the implications of our findings for mixed‐format assessments.

Principals’ severity affects teacher evaluation: statistical adjustments mitigate effects

ABSTRACT Classroom observation is a common approach to teacher evaluation. Yet, concerns about differences in rater judgment are widespread. Despite this concern, few researchers have examined the practical impact of such differences in rater judgments on teachers’ judged effectiveness. This study fills that gap. Using data from a large-scale teacher evaluation system, we found substantial differences in principal severity that affected teachers’ classification within performance categories. We then demonstrate a technique that researchers and practitioners can use to control for differences in rater severity – thus limiting the degree to which rater severity differences can threaten the fairness of classroom observations. We discuss implications for research and practice.

Are raters influenced by prior information about a learner? A review of assimilation and contrast effects in assessment.

Understanding which factors can impact rater judgments in assessments is important to ensure quality ratings. One such factor is whether prior performance information (PPI) about learners influences subsequent decision making. The information can be acquired directly, when the rater sees the same learner, or different learners over multiple performances, or indirectly, when the rater is provided with external information about the same learner prior to rating a performance (i.e., learner handover). The purpose of this narrative review was to summarize and highlight key concepts from multiple disciplines regarding the influence of PPI on subsequent ratings, discuss implications for assessment and provide a common conceptualization to inform research. Key findings include (a) assimilation (rater judgments are biased towards the PPI) occurs with indirect PPI and contrast (rater judgments are biased away from the PPI) with direct PPI; (b) negative PPI appears to have a greater effect than positive PPI; (c) when viewing multiple performances, context effects of indirect PPI appear to diminish over time; and (d) context effects may occur with any level of target performance. Furthermore, some raters are not susceptible to context effects, but it is unclear what factors are predictive. Rater expertise and training do not consistently reduce effects. Making raters more accountable, providing specific standards and reducing rater cognitive load may reduce context effects. Theoretical explanations for these findings will be discussed.

Monotonicity as a Nonparametric Approach to Evaluating Rater Fit in Performance Assessments

ABSTRACT Rater fit analyses provide insight into the degree to which rater judgments correspond to expected properties, as defined within a measurement framework. Parametric models such as the Rasch model provide a useful framework for evaluating rating quality; however, these models are not appropriate for all assessment contexts. The purpose of this study is to explore numeric and graphical rater monotonicity analyses as a nonparametric approach to evaluating rater fit, and to consider whether these indices lead to similar decisions as Rasch fit indices in contexts with complete and incomplete data. Using simulated and real data, results indicate that nonparametric rater monotonicity indices can be used to identify raters whose ratings exhibit acceptable and problematic psychometric properties.

Client attendance measures in counselling psychology trainees

AbstractCounselling supervisors and educators have yet to reach a consensus about standard measures for evaluating the performance of graduate student therapists. Compared to self‐reports and rater judgements, client attendance records provide low‐inference behavioural data partially reflective of clients’ engagement with therapists. This naturalistic study evaluates client attendance data using archival records of 92 doctoral students in beginning and advanced counselling practica who produced data for 771 clients, 3,949 scheduled sessions and 3,186 attended sessions. Providing evidence of the sample representativeness for clinicians, results indicated that the mean number of sessions attended per client for student therapists (5.65) was very similar to that reported in the literature for outpatient therapists (<6). As expected because of structural differences in client assignment, beginning and advanced students evidenced statistically significant differences on attendance variables such as number of clients and scheduled sessions. In contrast, attendance ratios were equivalent across counsellor experience, suggesting that these individual difference variables could be useful for evaluation of therapist competencies related to client engagement.

Exploring the Impact of Rater Effects on Person Fit in Rater‐Mediated Assessments

AbstractResearchers have documented the impact of rater effects, or raters’ tendencies to give different ratings than would be expected given examinee achievement levels, in performance assessments. However, the degree to which rater effects influence person fit, or the reasonableness of test‐takers’ achievement estimates given their response patterns, has not been investigated. In rater‐mediated assessments, person fit reflects the reasonableness of rater judgments of individual test‐takers’ achievement over components of the assessment. This study illustrates an approach to visualizing and evaluating person fit in assessments that involve rater judgment using rater‐mediated person response functions (rm‐PRFs). The rm‐PRF approach allows analysts to consider the impact of rater effects on person fit in order to identify individual test‐takers for whom the assessment results may not have a straightforward interpretation. A simulation study is used to evaluate the impact of rater effects on person fit. Results indicate that rater effects can compromise the interpretation and use of performance assessment results for individual test‐takers. Recommendations are presented that call researchers and practitioners to supplement routine psychometric analyses for performance assessments (e.g., rater reliability checks) with rm‐PRFs to identify students whose ratings may have compromised interpretations as a result of rater effects, person misfit, or both.

Does effectiveness in performance appraisal improve with rater training?

Performance appraisal is a complex process by which an organization can determine the extent to which employees are performing their work effectively. However, this appraisal may not be accurate if there is no reduction in the impact of problems caused by possibly subjective rater judgements. The main objective of this work is to check the effectiveness—separately and jointly—of the following four training programmes in the extant literature aimed at improving the accuracy of performance assessment: 1) Performance Dimension Training, 2) Frame-of-Reference, 3) Rater Error Training, and 4) Behavioural Observation Training. Based on these training strategies, three programmes were designed and applied separately. A fourth programme was a combination of the other three. We analyzed two studies using different samples (85 students and 42 employees) for the existence of differences in the levels of knowledge of performance and its dimensions, rater errors, observational accuracy, and accuracy of task and citizenship performance appraisal, according to the type of training raters receive. First, the main results show that training based on performance dimensions and the creation of a common framework, in addition to the training that includes the four programmes (Training_4_programmes), increases the level of knowledge of performance and its dimensions. Second, groups that receive training in rater error score higher in knowledge of biases than the other groups, whether or not they have received training. Third, participants’ observational accuracy improves with each new moment measure (post-training and follow-up), though not because of the type of training received. Fourth, participants who receive training through the programme that combine the other four gave a task performance appraisal that was closer to the one undertaken by the judges-experts than the other groups. And finally, students’ citizenship performance appraisal does not vary according to type of training or to different moment measures, whereas the group of employees who received all four types of training gave a more accurate citizenship performance assessment.

Conceptualizing Rater Judgments and Rating Processes for Rater‐Mediated Assessments

AbstractRater‐mediated assessments exhibit scoring challenges due to the involvement of human raters. The quality of human ratings largely determines the reliability, validity, and fairness of the assessment process. Our research recommends that the evaluation of ratings should be based on two aspects: a theoretical model of human judgment and an appropriate measurement model for evaluating these judgments. In rater‐mediated assessments, the underlying constructs and response processes may require the use of different rater judgment models and the application of different measurement models. We describe the use of Brunswik's lens model as an organizing theme for conceptualizing human judgments in rater‐mediated assessments. The constructs vary depending on which distal variables are identified in the lens models for the underlying rater‐mediated assessment. For example, one lens model can be developed to emphasize the measurement of student proficiency, while another lens model can stress the evaluation of rater accuracy. Next, we describe two measurement models that reflect different response processes (cumulative and unfolding) from raters: Rasch and hyperbolic cosine models. Future directions for the development and evaluation of rater‐mediated assessments are suggested.

A nonparametric procedure for exploring differences in rating quality across test-taker subgroups in rater-mediated writing assessments

Differences in rater judgments that are systematically related to construct-irrelevant characteristics threaten the fairness of rater-mediated writing assessments. Accordingly, it is essential that researchers and practitioners examine the degree to which the psychometric quality of rater judgments is comparable across test-taker subgroups. Nonparametric procedures for exploring these differences are promising because they allow researchers and practitioners to examine important characteristics of ratings without potentially inappropriate parametric transformations or assumptions. This study illustrates a nonparametric method based on Mokken scale analysis (MSA) that researchers and practitioners can use to identify and explore differences in the quality of rater judgments between subgroups of test-takers. Overall, the results suggest that MSA provides insight into differences in rating quality across test-taker subgroups based on demographic characteristics. Differences in the degree to which raters adhere to basic measurement properties suggest that the interpretation of ratings may vary across subgroups. The implications of this study for research and practice are discussed.

Evaluating rater judgments on ETIC Advanced writing tasks: An application of generalizability theory and Many-Facet

Developed by China Language Assessment (CLA), the English Test for International Communication Advanced (ETIC Advanced) assesses one’s ability to perform English language tasks in international workplace contexts. ETIC Advanced is only composed of writing and speaking tasks, featured with authentic constructed response format. However, the elicitation of extended responses from candidates would call for human raters to make judgments, thus raising a critical issue of rating quality. This study aimed to evaluate rater judgements on the writing tasks of ETIC Advanced. Data in the study represented scores from 186 candidates who performed all writing tasks: Letter Writing, Report Writing, and Proposal Writing (n=3,348 ratings). Rating was conducted by six certified raters based on a six-point three-category analytical rating scale. Generalizability theory (GT) and Many-Facets Rasch Model (MFRM) were applied to analyse the scores from different perspectives. Results from GT indicated that raters’ inconsistency and interaction with other aspects resulted in a relatively low proportion of overall score variance, and that the ratings sufficed for generalization. MFRM analysis revealed that the six raters differed significantly in severity, yet remained consistent in their own judgements. Bias analyses indicated that the raters tended to assign more biased scores to low-proficient candidates and the Content category of rating scale. The study serves to demonstrate the use of both GT and MFRM to evaluate rater judgments on language performance tests. The findings of this study have implications for ETIC rater training.

Identifying Linguistic Markers of Collaboration in Second Language Peer Interaction: A Lexico‐grammatical Approach

Although there is consensus that collaboration refers to two or more learners working together to accomplish a task (Davin & Donato, 2013; Ohta, 2001), debate remains about how to assess collaboration. Researchers have pursued two approaches to evaluate collaboration during peer interaction: rater judgments (e.g., Ahmadi & Sedeghi, 2016; Winke, 2013) and qualitative coding of interactional patterns (e.g., Galaczi, 2008; Storch, 2002a). Largely absent, however, has been any attempt to describe the linguistic features of collaboration. Therefore, the present study uses corpus linguistic techniques to identify the linguistic markers of collaborative and noncollaborative peer interactions. Students of English as a second language (N = 80) enrolled in an intensive English program carried out a paired oral test as part of the program's formative assessment procedures. Their interactions were audio‐recorded and rated using an analytic rubric with three categories (collaboration, task completion, and style), and transcripts were analyzed for 146 linguistic features using the Biber Tagger (Biber, 1988). Linguistic features associated with high collaboration included first‐ and second‐person pronouns, wh‐questions, that deletion, and subordinate conjunctions, whereas low‐collaboration interactions were characterized by nominal forms. The collaborative and noncollaborative functions served by these linguistic features are discussed.

Explicit and implicit aptitude effects on second language speech learning: Scrutinizing segmental and suprasegmental sensitivity and performance via behavioural and neurophysiological measures

The current study examines the role of cognitive and perceptual individual differences (i.e., aptitude) in second language (L2) pronunciation learning, when L2 learners’ varied experience background is controlled for. A total of 48 Chinese learners of English in the UK were assessed for their sensitivity to segmental and suprasegmental aspects of speech on explicit and implicit modes via behavioural (language/music aptitude tests) and neurophysiological (electroencephalography) measures. Subsequently, the participants’ aptitude profiles were compared to the segmental and suprasegmental dimensions of their L2 pronunciation proficiency analyzed through rater judgements and acoustic measurements. According to the results, the participants’ segmental attainment was associated not only with explicit aptitude (phonemic coding), but also with implicit aptitude (enhanced neural encoding of spectral peaks). Whereas the participants’ suprasegmental attainment was linked to explicit aptitude (rhythmic imagery) to some degree, it was primarily influenced by the quality and quantity of their most recent L2 learning experience.