Rater Behavior Research Articles

Operational Rater Types in Writing Assessment: Linking Rater Cognition to Rater Behavior

This research investigated the relation between rater cognition and rater behavior, focusing on differential severity/leniency regarding criteria. Participants comprised a sample of 18 raters examined in a previous rater cognition study (Eckes, 2008b). These raters, who were known to differ widely in their perceptions of criterion importance, provided ratings of live examinee writing performance. Based on these ratings, criterion-related bias measures were estimated using many-facet Rasch measurement. A cluster analysis of bias measures yielded four operational rater types. Each type was characterized by a distinct pattern of differentially severe or lenient ratings on particular criteria. The observed bias patterns were related to differential perceptions of criterion importance: Criteria perceived as highly important were more closely associated with severe ratings, and criteria perceived as less important were more closely associated with lenient ratings. Implications of the demonstrated link between rater cognition and rater behavior for future research into the nature of rater bias are discussed.

Effectiveness of rater consensus training in objective structured clinical examination using Kolb’s experiential learning

conceptualization 1. Discuss immediately the concrete experience using the immediate response system 2. Discuss rating scores for two cycles under modified Delphi consensus training 3. Observation of inconsistent rater behaviors Active experimentation 1. Rating during consensus training 2. Rating at high-stake OSCE OSCE1⁄4 objective structured clinical examination. Table 2 Inter-rater reliability among each station. Station Paired t-test Pearson correlation Mean (n1⁄4 58) CI SD (95%) p value * r p value ** 1. Leg pain 0.17 1.11w 0.76 3.560 0.71 0.55 0.000 2. Headache 0.17 1.06w 1.40 4.676 0.78 0.49 0.000 3. Dizziness 0.97 2.56w 0.02 3.770 0.06 0.56 0.000 4. Palpitation 0.43 0.33w 1.19 2.878 0.26 0.70 0.000 5. Pancreatitis 0.62 1.86w 0.62 4.720 0.32 0.45 0.000 6. Menopause 0.50 0.38w 1.38 3.336 0.26 0.70 0.000 7. Cough 0.28 0.53w 1.08 3.048 0.49 0.65 0.000 8. H1N1 0.88 1.73w 0.02 3.250 0.04 0.34 0.009 *value obtained from the paired t-test (a1⁄4 0.05). **value obtained from the Pearson correlation (a1⁄4 0.01). CI1⁄4 confidence interval; SD1⁄4 standard deviation. S.-Y. Chu et al. / Tzu Chi Medical Journal 24 (2012) 155e156 156 concrete experience, and trainers can immediately fill in gaps in knowledge. We found behavior level changes, as determined by the self-evaluation, to be high. However, raters’ scoring variations remained high on two stations in this OSCE examination. Further investigation is needed to determine the reasons contributing to these raters’ persistent scoring variations. Nonetheless, these early results support the idea that Kolb’s cycle of experiential learning model can facilitate rater consensus training in a high-stakes OSCE setting.

Ratings of counterproductive performance: the effect of source and rater behavior

PurposeThe purpose of this study was to examine inter‐rater agreement on counterproductive performance between self‐ and peer‐ratings, and the factors that moderate this agreement. The factors investigated included self‐reported levels of counterproductive performance and known antecedents of counterproductive performance: conscientiousness and integrity values.Design/methodology/approachData were gathered (three to five peer ratings per individual) from 108 undergraduate students.FindingsThe paper finds that there was a significantly low correlation between self‐ and peer‐ ratings of counterproductive performance. Ratings given by peers were much higher than ratings given by oneself. Individuals and peers who are similar in the extent to which they engage in counterproductive behaviors were in agreement with respect to ratings of counterproductive performance.Practical implicationsThis study provided evidence that rater disagreement is a consistent phenomenon across dimensions of performance. In addition, rater perceptions of counterproductive performance have a significant impact on overall performance ratings; therefore individual differences between the rater and ratee may have a large influence on overall ratings in an organizational setting. There is some evidence in this study that peer ratings of counterproductive behavior vary depending on the rater's own counterproductive behaviors. The fact that rater agreement is influenced by the rater's own behavior implies that individual rater effects are influencing counterproductive performance measurement.Originality/valueThis study adds value by extending the literature on inter‐rater agreement to counterproductive performance. In addition, this study is unique in that it shows that a rater's own level of counterproductive performance can impact their ratings of others.

Investigating the effectiveness of individualized feedback to rating behavior — a longitudinal study

The effectiveness of individualized feedback on rater behavior has been investigated in several previous studies (e.g. Elder, Knoch, Barkhuizen, & von Randow, 2005; Lunt, Morton, & Wigglesworth, 1994; O’Sullivan & Rignall, 2007; Wigglesworth, 1993). The findings of these studies are somewhat inconclusive. However, all of these studies investigated the value of such feedback on a one-off basis, and it is thus not clear how raters use this type of feedback over several administrations of a test. Furthermore, previous research has focused only on one language skill at a time, and we do not know if raters can incorporate the feedback in a similar way when rating speaking and writing. This study tracks the rating behavior of 19 raters assessing a large-scale English for specific purposes (ESP) assessment for the health professions over eight administrations. After each administration, raters received detailed performance profiles of their rating behavior which were generated using many-faceted Rasch measurement. Raters also completed a questionnaire and a subset of raters were interviewed to ascertain their views on the effectiveness of this feedback. The findings showed that the raters were rating no better when receiving the feedback than when they had not received feedback and neither speaking nor writing raters were able to incorporate the feedback more successfully. Although the raters were generally positive about the success of the feedback, there was no relationship between their perceptions of the feedback and the success of the feedback.

Factors That Influence Data Quality in Caries Experience Detection: A Multilevel Modeling Approach

Caries experience detection is prone to misclassification. For this reason, calibration exercises which aim at assessing and improving the scoring behavior of dental raters are organized. During a calibration exercise, a sample of children is examined by the benchmark scorer and the dental examiners. This produces a 2 × 2 contingency table with the true and possibly misclassified responses. The entries in this misclassification table allow to estimate the sensitivity and the specificity of the raters. However, in many dental studies, the uncertainty with which sensitivity and specificity are estimated is not expressed. Further, caries experience data have a hierarchical structure since the data are recorded for the surfaces nested in the teeth within the mouth. Therefore, it is important to report the uncertainty using confidence intervals and to take the clustering into account. Here we apply a Bayesian logistic multilevel model for estimating the sensitivity and specificity. The main goal of this research is to find the factors that influence the true scoring of caries experience accounting for the hierarchical structure in the data. In our analysis, we show that the dentition type and tooth or surface type affect the quality of caries experience detection.

Playing with the stakes: A consideration of an aspect of the social context of a gatekeeping writing assessment

Abstract In high-stakes writing assessments, rater training in the use of a rating scale does not eliminate variability in grade attribution. This realisation has been accompanied by research that explores possible sources of rater variability, such as rater background or rating scale type. However, there has been little consideration thus far of variability in rater behaviour that may be brought about by the socio-political context of the testing situation. In addition, studies of rater behaviour are undertaken in either research conditions or authentic high-stakes testing conditions, but never both. This report is a “stakes study” in the context of a writing test for teacher certification in Quebec. The research objective was to see if raters grade differently when the consequences to test takers are changed but if all else remains the same (text, rater, rating scale and rater training). Scores given by the raters in the “high-stakes” (authentic testing) and “low-stakes” (research) conditions were compared. Raters were also questioned immediately after their final grading session to allow them to discuss any perceived differences in their rating behaviour in the two conditions. It was hypothesised that raters might be more severe in their grading if there were no stakes involved for the test takers. This case study, while exploratory and small in scale, demonstrates the potential in including the social context as a factor in future research on variability in rater scoring behaviour.

Comment

Comment

Native and nonnative rater behavior in grading Korean students’ English essays

The purpose of the study is to investigate rating behavior between Korean and native English speaking (NES) raters. Five Korean English teachers and five NES teachers graded 420 essays written by Korean college freshmen and completed survey questionnaires. The grading data were analyzed with FACETS program. The results revealed Korean raters’ inferiority in measuring linguistic components. Furthermore, the Korean raters were more severe in scoring grammar, sentence structure, and organization, whereas the NES raters were stricter toward content and overall scores. In addition, the analysis of the raters’ responses on survey discovered that the NNS raters’ perception spread into content and grammar as the most difficult feature to grade, while all NES raters thought content as the most difficult. Based on these research findings, future research suggestions and implications are discussed.

Diagnostic assessment of writing: A comparison of two rating scales

Alderson (2005) suggests that diagnostic tests should identify strengths and weaknesses in learners' use of language and focus on specific elements rather than global abilities. However, rating scales used in performance assessment have been repeatedly criticized for being imprecise and therefore often resulting in holistic marking by raters (Weigle, 2002). The aim of this study is to compare two rating scales for writing in an EAP context; one `a priori' developed scale with less specific descriptors of the kind commonly used in proficiency tests and one empirically developed scale with detailed level descriptors. The validation process involved 10 trained raters applying both sets of descriptors to the rating of 100 writing scripts yielded from a large-scale diagnostic assessment administered to both native and non-native speakers of English at a large university. A quantitative comparison of rater behaviour was undertaken using FACETS. Questionnaires and interviews were administered to elicit the raters' perceptions of the efficacy of the two types of scales. The results indicate that rater reliability was substantially higher and that raters were able to better distinguish between different aspects of writing when the more detailed descriptors were used. Rater feedback also showed a preference for the more detailed scale. The findings are discussed in terms of their implications for rater training and rating scale development.

Examining Rater Behavior on a Revised Version of the Short Child Occupational Profile (SCOPE)

A previous study of the Short Child Occupational Profile (SCOPE), an assessment of occupational participation, supported the psychometric soundness of the instrument overall, but pointed to some potential problems in practitioners' use of the SCOPE in practice. Specific revisions were made to the SCOPE to address the rating behaviors of leniency/severity, halo effect, and restriction of range. A many-faceted Rasch model analysis was conducted after 39 practitioners working in eight practice sites learned to administer the SCOPE using a variety of methods and rated 168 clients using the SCOPE (version 2.1). Practitioners exhibited three significantly different levels of leniency/severity, 35 practitioners did not exhibit any halo effects, and the top three rating categories were used 91% of the time. Findings suggest that a variety of methods can be used to learn to administer and rate the SCOPE in an appropriate manner to assess the personal and environmental factors affecting the occupational participation of children.

A New Approach to Rater Training and Certification in a Multicenter Clinical Trial

Recent evidence demonstrates that the quality of raters' applied clinical skills is directly related to study outcome. As such, the training and evaluation of raters' clinical skill in administering symptom-rating scales is essential before being certified to rate patients in clinical trials. This study examined a novel approach to rater training and certification that focused on both conceptual knowledge and applied skills. Forty-six raters (MDs = 14; PhDs = 7; MA = 5; BA/LPN/RN = 20) in a large multicenter depression study went through a 2-step Hamilton Rating Scale for Depression (HAMD) certification process: didactic training, administered online via an interactive Web tutorial, and live, applied training, where raters interviewed depressed patients while being remotely observed via 3-way teleconference. Raters' applied skills were evaluated using the Rater Applied Performance Scale (RAPS), designed specifically to evaluate critical rater behaviors associated with good clinical interviews. Raters received feedback immediately following the interviews; those receiving a failing score were given 2 more opportunities to pass. Each subsequent session was accompanied by feedback, and was conducted by a different trainer, who was blind to the results of the previous session as well as to which session number it was, to avoid bias. Raters who failed on the third attempt were excluded from rating patients in the trial. All training and testing occurred prior to the startup meeting. Results found a significant improvement pre-to-post Web training in raters knowledge of scoring conventions, P < 0.001. On the applied component, raters' RAPS scores improved significantly on the second attempt following feedback, from 9.05 to 11.58, P < 0.001, and from their second to their third session (from 9.00 to 11.00, P = 0.033. Three raters failed all 3 attempts and were excluded from the study. Results support the efficacy of the approach in improving both conceptual knowledge and applied interviewing skill.

A Model of Rater Behavior in Essay Grading Based on Signal Detection Theory

An approach to essay grading based on signal detection theory (SDT) is presented. SDT offers a basis for understanding rater behavior with respect to the scoring of construct responses, in that it provides a theory of psychological processes underlying the raters' behavior. The approach also provides measures of the precision of the raters and the accuracy of classifications. An application of latent class SDT to essay grading is detailed, and similarities to and differences from item response theory (IRT) are noted. The validity and utility of classifications obtained from the SDT model and scores obtained from IRT models are compared. Validity coefficients were found to be about equal in magnitude across SDT and IRT models. Results from a simulation study of a 5‐class SDT model with eight raters are also presented.

A Reputation Mechanism for Business-to-Business Electronic Commerce That Accounts for Rater Credibility

This article investigates whether source credibility theory can support reputation mechanisms in business-to-business (B2B) electronic commerce. In contrast to consumer electronic marketplaces, the raters in B2B communities are skilled and connected, necessitating a reputation mechanism to account for the relationship between the user and the rater. To solve this problem, TrustBuilder, a prototype rating tool, incorporates a methodology to calculate a weighted rating based on source credibility theory. This solution offers several advantages over existing models. First, source credibility theory provides tested frameworks for aggregating ratings from different sources. Second, there are validated scales for measuring a source's (rater's) credibility. Finally, the weights of a rater's ratings depend on user preferences instead of rater behavior, which decreases the amount of data required to calibrate the model. In an experiment, industry practitioners evaluated bids from service providers using a credibility-weighted tool as well as a standard unweighted tool. The experiment showed that the use of a credibility-weighted tool led to increased user confidence as well as more varied evaluations. This study therefore provides evidence that incorporating source credibility theory in a rating tool adds value in the process of evaluating service providers by increasing the decision maker's confidence in the accuracy of the information.

Effects of Accountability on Rating Behavior and Rater Accuracy

We tested whether improvements in rater accuracy on a performance‐appraisal task attributed to rater accountability could be explained by variance in rater behavior. Data from Mero and Motowidlo (1995) were used initially to test whether observed improvements in rater accuracy could be explained by rating process behaviors of attending to relevant subordinate performance and taking better notes to record that performance. Results showed that both behaviors were correlated with accuracy and that these behaviors partially mediated the relationship between accountability and improved decision quality for participants who were held accountable for their rating decision and did not have any preconceived information about that rating from their supervisor. We then tested this process model on a new sample of participants and found that attentiveness and notetaking mediated the relationship between accountability and accuracy.

The Effect of Scoring Criteria Specificity on Peer and Self-assessment

The value of peer and self-assessments is commonly diminished by scoring range restriction by the raters. This investigation studied the effect of different levels of scoring criteria specificity and written feedback requests on the distribution of scores, the correlation between peer and self-assessments and the quantity and nature of written feedback. Increasing the number of criteria decreased the mean scores and increased the standard deviations of the peer and self-assessments, providing a wider range of scores and increasing the sensitivity of the instrument. Correlation between peer and self-assessment was improved with more specific criteria, depending on the statistic used. However, analysis revealed that the more specific written feedback requests elicited more peer feedback. Educators should consider the effects of criteria specificity and written feedback solicitation on rater behaviour when designing these instruments.

Rater characteristics and rater bias: implications for training

Recent developments in multifaceted Rasch measurement (Linacre, 1989) have made possible new kinds of investigation of aspects (or 'facets') of performance assessments. Relevant characteristics of such facets (for exam ple, the relative harshness of individual raters, the relative difficulty of test tasks) are modelled and reflected in the resulting person ability measures. In addition, bias analyses, that is interactions between elements of any facet, can also be analysed. (For the facet 'person', an element is an individual candidate; for the facet 'rater', an element is an individual judge, and so on.) This permits investigation of the way a particular aspect of the test situation (type of candidate, choice of prompt, etc.) may elicit a consistently biased pattern of responses from a rater. The purpose of the research is to investigate the use of these analytical techniques in rater training for the speaking subtest of the Occupational English Test (OET), a specific-purpose ESL performance test for health professionals. The test involves a role-play based, profession-specific inter action, involving some degree of choice of role-play material. Data are presented from two rater training sessions separated by an 18-month interval and a subsequent operational test administration session. The analysis is used to establish 1) consistency of rater characteristics over different occasions; and 2) rater bias in relation to occasion of rating. The study thus addresses the question of the stability of rater characteristics, which has practical implications in terms of the accreditation of raters and the requirements of data analysis following test administration sessions. It also has research implications concerning the role of multifaceted Rasch measurement in understanding rater behaviour in performance assessment contexts.

The effect of writers' personalities and raters' personalities on the holistic evaluation of writing

Recent work in reading and writing theory, research and pedagogy has raised questions about relationships between Fluent reading processes and holistic scoring of essays (e.g., Huot 1993). In holistic scoring settings, are the raters behaving as normal Fluent readers (i.e., readers interacting critically and personally with the text) or, are they somehow disconnected From their normal reader responses because they are using reliable scoring guides? Related questions concern the behavior of such holistic raters when they are teachers (e.g., Barritt, Stock, & Clark, 1986), and when those teachers respond to student writing (Connors & Lunsford, 1993). How are teachers/raters behaving, and what are they responding to in judging the writing? Previous research has suggested a role for personality type in the study of the process of writing evaluation (Jensen & DiTiberio, 1984, 1989). Thus, it seems reasonable to ask what role personality types play in the holistic evaluation of writing. This empirical study addressed the general question: What role, if any, do personality types of writers and of raters play in the holistic rating of writing? Moreover, is there a relationship between writers' personalities and raters' personalities? Writers were native English-speaking university freshman composition students; raters were native English-speaking university freshman composition instructors. Results indicate that the personality types of writers affect the ratings their essays receive, and the personality types of raters affect the ratings they give to essays. However, there is no significant relationship between writers' styles and raters' styles. Implications for future research, as well as classroom implications of these results are discussed.

Patterns of rater behaviour in the assessment of an oral interaction test

Abstract Lack of inter-rater agreement in the assessment of oral tests is wellknown. In this paper, multi-faceted Rasch analysis was used to determine whether any bias was evident in the way a group of raters (N=13) rated two different versions of an oral interaction test, undertaken by the same candidates (N=83) under the two conditions – direct and semi-direct. Rasch measurement allows analysis of the interaction between ‘facets’; in this case, raters, items and candidates are all facets. In this study, the interaction between rater and item was investigated in order to determine whether particular tasks in the test were scored in a consistently biased way by particular raters. The results of the analysis indicated that certain raters consistently assessed the tape-version of the test more harshly whilst others consistently rated the live version more harshly. This type of approach also allowed a finer analysis at the level of individual items with respect to harshness and consistency across ratings. The implications for rater training and feedback are discussed.

How do subject specialists construe classroom language proficiency?

Recent research on rater variation (e.g., Barnwell, 1989; Douglas and Selinker, 1990; Hadden, 1991) raises the question of whether 'linguistically naive' subject specialists may be better equipped than language experts to judge the effectiveness of particular areas of non-native speaker communica tion. This question is investigated with reference to a classroom-based observation schedule which was developed to assess the English language proficiency of non-native speaker graduates training as secondary mathema tics and science teachers. The article examines aspects of rater behaviour as evidenced in recent trials. The schedule was applied to observations of actual performance in the maths and science classroom as well as to the viewing of a number of videoed segments of classroom interaction. Ratings were elicited from two groups of assessors: nine ESL teachers and eight subject specialists (maths/science teachers/teacher trainers). Findings reveal significant correlations between subject specialists' and language teachers' overall judgements of communica tive effectiveness, but differences between the two groups with respect to their ratings of particular dimensions of language use and to the weighing of these dimensions in relation to global proficiency assessments.

Self‐appraisal and perceptions of the appraisal discussion: A field experiment

AbstractPerformance evaluation based on self‐appraisal has been widely advocated because of its potential for increasing the effectiveness of the performance appraisal discussion. A field experiment was conducted to assess the effects of a formal ’both‐rate‘ self appraisal (where both manager and subordinate independently complete appraisals before the discussion) on perceptions of ratee and rater behaviors and outcomes. One hundred fifty‐one ratees and 81 raters randomly assigned to self‐appraisal and control groups participated in the study. Results indicated that the self‐appraisal treatment had no main effects on ratee perceptions of their contributions to the discussion or satisfaction with the appraisal. Self‐appraising ratees perceived less influence over the appraisal discussion, and less agreement with their manager's rating than did non‐self appraisers. However, informal self‐appraisal behavior was significantly and positively correlated with most dependent variables. Results suggest major differences between formal and informal self‐appraisal which warrant future research.