Effect Of Raters Research Articles

Effects of listeners’ dialectal variation on comprehensibility and accentedness judgements of second language speech

Abstract This study investigated the impact of listeners’ dialectal variations within the same L1 on perceptions of L2 speech comprehensibility and accentedness. Thirty untrained British and American English-L1 raters assessed recorded L2 speech samples from 29 Finnish learners of English and provided scalar judgments for their comprehensibility and accentedness. Four linguistically trained graduate students evaluated the samples for segmental accuracy, speech rate, word stress and intonation. Results of linear mixed models indicated no significant effect of raters’ L1 variety on global comprehensibility and accentedness ratings. While American and British raters relied on same pronunciation features in their comprehensibility judgments, they placed different levels of importance on pronunciation features when making accentedness judgments. Particularly, both groups primarily exhibited sensitivity towards segmental errors in their assessments of accentedness, whereas American raters demonstrated an additional sensitivity to intonation. The findings underscore the complexity of accent perception based on the L1 variety of listeners.

Twins Assessing Their Own and Parental Intelligence: Examining the Raters' Agreement and the Effect of Raters' and Targets' Gender.

The goal of this study was to explore the raters’ agreement and the effect of raters’ and targets’ gender on self- and parental intelligence assessments in the sample of Croatian twins. Twins were asked to assess their own and their parents’ overall intelligence, as well as specific abilities from the Gardner’s theory of multiple intelligences. Data was analysed to explore: i) twins’ agreement in parental assessments and behavioural genetic analysis of the overall intelligence estimates; ii) gender differences in self- assessments; and iii) raters’ and targets’ gender effects on parental assessments. The twins’ mean correlation in their assessments of overall parental intelligence was .60. The differences between monozygotic and dizygotic twin correlations were nonsignificant for all of the estimated abilities, and model fitting analysis indicates that hypothesis about genetic effect on parental assessment of intelligence should be rejected. The hypotheses about males’ higher self-assessments for overall intelligence and for the masculine types of abilities - logical-mathematical, body-kinesthetic and spatial abilities - were confirmed. For the feminine types of abilities - verbal/linguistic, inter- and intra- personal intelligences - there were no significant gender effects. Both target and rater effect were found for the parental estimates of intelligence. Fathers were estimated higher on overall intelligence, logical-mathematical, body-kinesthetic and spatial abilities, while mothers were estimated higher on interpersonal and intrapersonal intelligence. The effect of the raters’ gender was found for overall intelligence as well as for inter- and intra- personal intelligences, where males gave higher estimates of parental intelligences than females.

Reproducibility of the DrGoniometer app for field-based assessment of the break-point angle in Nordic Hamstring exercise.

Smartphone-based measurement could represent a portable and cheap solution to assess the Nordic Hamstring break-point angle (BPA). We investigated the following: (1) the reproducibility of the DrGoniometer app, (2) its agreement with a computerized video analysis software used as a reference method, and (3) the effects of raters' experience. Fifty recordings were processed by four independent raters. Intraclass correlation coefficient (ICC), SEM, and minimum detectable change (MDC) were calculated. Bland-Altman plots verified the limits of agreement (LoA) with the reference method. The effect of raters' clinical experience was investigated by unpaired t-tests. The app showed excellent intra-rater (ICC = 0.89, 95% CI, 0.84-0.92) and inter-rater reliability (ICC = 0.82, 95% CI, 0.74-0.89), low error (SEM = 2°; MDC = 5.5°), and substantial agreement with video analysis [95% LoA = ±6.7°; mean difference = 1.0°]. Rater experience did not affect results (P > 0.05). The smartphone-based app offers a reliable, valid, quick, and portable method for measuring the BPA in Nordic Hamstring exercise.

Fairness and objectivity of a multiple scenario objective structured clinical examination.

Introduction: The aim of the Objective Structured Clinical Examination (OSCE) is a standardized and fair assessment of clinical skills. Observing second clinical year medical students during a summative OSCE assessing a General Practice clerkship, we noticed that information exchange with peers led to a progressively faster and overly focused management of simulations. Therefore, we established a Multiple Scenario-OSCE (MS-OSCE) where all students had to manage the same chief complaint at a station but it’s underlying scenarios being randomly changed during students’ rotation through their parcours. We wanted to ensure they fully explore differential diagnosis instead of managing their task influenced by shared information. We wanted to assess if a MS-OSCE violates the assumption of objectivity and fairness given that students are not tested with the same scenarios.Methods: We developed and piloted five OSCE stations (chest pain, abdominal pain, back pain, fatigue and acute cough) with two or three different underlying scenarios each. At each station these scenarios randomly changed from student to student. Performance was assessed with a checklist and global rating. The effect of scenarios and raters on students’ grades was assessed calculating the intraclass correlation coefficient with a fixed effect two level linear model.Results: A total of 169 students and 23 raters participated in the MS-OSCE. The internal consistency over all stations was 0.65 by Cronbach’s alpha. The difference of the mean grades between the scenarios of a given chief complaint ranged from 0.03 to 0.4 on a 1 to 5 grading scale. The effect of scenarios on the variance of the final grades at each station ranged from 4% to 9% and of raters from 20% to 50% when adjusted for students’ skills. Conclusions: The effect of different scenarios on the grades was relevant but small compared to the effect of raters on grades. Improving rater training is more important to ensure objectivity and fairness of MS-OSCE than providing the same scenario to all students.

Systematic analysis of measurement variability in lung cancer with multidetector computed tomography.

OBJECTIVE:To systematically analyze the nature of measurement variability in lung cancer with multidetector computed tomography (CT) scans.METHODS:Multidetector CT scans of 67 lung cancer patients were analyzed. Unidimensional (Response Evaluation Criteria in Solid Tumor criteria), bidimensional (World Health Organization criteria), and volumetric measurements were performed independently by ten radiologists and were repeated after at least 5 months. Repeatability and reproducibility measurement variations were estimated by analyzing reliability, agreement, variation coefficient, and misclassification statistically. The relationship of measurement variability with various sources was also analyzed.RESULTS:Analyses of 69 lung tumors with an average size of 1.1–12.1 cm (mean 4.3 cm) indicated that volumetric technique had the minimum measurement variability compared to the unidimensional or bidimensional technique. Tumor characteristics (object effect) could be the primary factor to influence measurement variability while the effect of raters (subjective effect) was faint. Segmentation and size in tumor characteristics were associated with measurement variability, and some mathematical function was established between the volumetric variability and tumor size.CONCLUSION:Volumetric technique has the minimum variability in measuring lung cancer, and measurement variability is associated with tumor size by nonlinear mathematical function.

A Cross-Linguistic Investigation of the Effect of Raters’ Accent Familiarity on Speaking Assessment

ABSTRACTThe project aimed to examine the effect of raters’ familiarity with accents on their judgments of non-native speech. Participants included three groups of raters who were either from Spanish Heritage, Spanish Non-Heritage, or Chinese Heritage backgrounds (n = 16 in each group) using Winke & Gass’s (2013) definition of a heritage learner as being someone who reports cultural or ethnic ties and has family members who are native speakers of that heritage language. Using a 1–7 Likert-type scale, all participants evaluated the “overall English proficiency” and “foreign accents” of 28 speech samples selected from Educational Testing Service’s TOEFL iBT public data set. Approximately half of the speech samples were from speakers with a Spanish language background, and the other half were from speakers with a Chinese language background and were matched on English language proficiency. Results revealed that familiarity with a particular accent facilitated the identification of that accent. Although the three rater groups did not differ reliably in the ratings assigned to the speakers, the majority self-reported that their accent familiarity affected their evaluations and possibly made them more lenient toward speakers with familiar accents. Furthermore, all rater groups assigned higher ratings to Spanish speakers than to Chinese speakers.

Board #119 - Research Abstract Quantifying Clinical Performance During Simulated Pediatric Cardiac Arrest (Submission #9887)

Hypothesis Methods to quantitatively measure performance during resuscitative care are lacking in published literature. Members of our investigative team have previously published psychometric analyses of task-based scoring instruments used in educational research in pediatric resuscitation. These published investigations used instruments that were designed for specific cases in pediatric resuscitation, rather than for a more generalizable application. We hypothesize that a novel scoring instrument will reliably assess clinical performance during simulated cardiac arrest. Methods This study was conducted at 11 pediatric centers in Canada and the US. Teams of pediatric healthcare providers performed a simulated arrest scenario (asystole for 6 minutes, VF for 6 minutes). A task-based scoring instrument was designed using a 0, 1, or 2 point scoring system to rate performance; items were chosen according to the pulseless arrest algorithm of the American Heart Association Pediatric Advanced Life Support course and include CPR performance parameters (chest compression rate, depth, release, pauses), defibrillation (dose, timing), and epinephrine administration (dose, timing). Multiple raters reviewed and scored a set of simulations. Overall interrater reliability was measured; a fully-crossed generalizability study with team and rater as facets was performed to determine the variance in scores ascribable to each facet; a decision study was done to determine the effect of additional raters and scenarios on the G coefficient. Results Three raters scored four videos. Overall scores ranged from 53/90 (59%) to 73/90 (81%) possible points. Intraclass correlation coefficient was 0.77 (F 3,8 = 4.46, p = 0.04). Variance components were 21% for rater, 57% for scenario. G coefficient was 0.80; by D study this increased to 0.91 and 0.93 with 8 and 10 raters, respectively. Conclusion A novel scoring instrument for quantifying performance during pediatric cardiac arrest showed modest reliability and generalizability. Future studies should examine the effect of a larger number of raters and/or scenarios on generalizability, as well as the utility of the instrument in assessing real clinical performance. Disclosures Adam Cheng receives grant support from the Heart and Stroke Foundation of Canada, and the Canadian Institute for Health Research. Nancy Tofil spoke at a Laerdal Conference, presenting her own original material. She did not receive honoraria, but was reimbursed travel expenses.

Abstract 268: Quantitative Performance Assessment of Simulated Pediatric Cardiopulmonary Resuscitation

Background: Methods to quantitatively measure performance during resuscitative care are lacking in published literature. Members of our group have previously published psychometric analyses of task-based scoring instruments used in educational research in pediatric resuscitation. These published investigations used instruments that were designed for specific cases in pediatric resuscitation, rather than for a more generalizable application. We hypothesize that a novel scoring instrument will reliably assess clinical performance during simulated cardiac arrest. Methods: This study was conducted at 11 pediatric centers in Canada and the US. Teams of pediatric providers performed a simulated cardiac arrest scenario (asystole for 6 minutes, VF for 6 minutes). A task-based scoring instrument was designed by investigator consensus using a 0, 1, or 2 point scoring system to rate performance during cardiac arrest. The items were chosen according to the essential steps in the pulseless arrest algorithm of the AHA Pediatric Advanced Life Support course and include CPR performance parameters (chest compression rate, depth, release, pauses), defibrillation (dose in J/kg, timing), and epinephrine (dose, timing). Multiple raters reviewed and scored a set of simulations. Overall interrater reliability was measured; a fully-crossed generalizability study with team and rater as facets was performed to determine the variance in scores ascribable to each facet; a decision study was done to determine the effect of additional raters and scenarios on the G coefficient. Results: Three raters scored four videos. Overall scores ranged from 53/90 (59%) to 73/90 (81%) possible points. Intraclass correlation coefficient was 0.77 (F 3,8 = 4.46, p = 0.04). Variance components were 21% for rater, 57% for scenario. G coefficient was 0.80; by D study this increased to 0.91 and 0.93 with 8 and 10 raters, respectively. Conclusions: A novel scoring instrument for quantifying performance during pediatric cardiac arrest showed modest reliability and generalizability. Future studies should examine the effect of a larger number of raters and/or scenarios on generalizability, as well as the utility of the instrument in assessing real clinical performance.

Circumferential Iris Transillumination Defects in Exfoliation Syndrome

We identified a pattern of concentric circular transillumination defects (TIDs) in a few patients with exfoliation syndrome (XFS) using an infrared detection system. This pattern of iris abnormality has also been observed in a mouse model of XFS. The objective of the current study is to determine whether concentric iris TIDs are specific to XFS and may have some diagnostic utility for identifying early cases of disease. A total of 68 volunteers from the University of Iowa Glaucoma Clinic with normal eyes (n=21) or diagnoses of either XFS (n=12), pigment dispersion syndrome (PDS) (n=8), or primary open-angle glaucoma (POAG) (n=27) were enrolled in the study. The irides of these subjects were each examined by 4 ophthalmologists masked to their diagnosis, using infrared videography. The presence of concentric, circular TIDs on the videos was graded as none (grade 0), possible (grade 1), definite (grade 2), or prominent (grade 3) by 4 examiners. We searched for an association between the presence of concentric bands of transillumination and the diagnosis of XFS after removing the effect of different raters was evaluated using the Cochran-Mentel-Haenszel test. We performed the same analysis for PDS and for POAG. The presence of any concentric, circular iris TIDs (grades 1 to 3) was detected in a mean of 38% normal subjects, 35% POAG patients, 53% PDS patients, and 77% of XFS patients. When the frequency of concentric, circular iris transillumination (grades 1 to 3 pooled) was compared between each of the patient groups and normal controls, a significant difference was detected between XFS patients and controls (P=0.000019). No significant difference was detected between POAG and controls (P=0.64) or between PDS and controls (P=0.20). Furthermore, prominent concentric, circular iris transillumination (grade 3) was only observed in XFS. Detection of concentric, circular iris TIDs with an infrared system is easy, inexpensive, rapid, and relatively specific in XFS. Future larger studies will be needed to confirm the findings of this small pilot study. Furthermore, this examination technique has the potential to help physicians to make earlier diagnoses of XFS and to better plan for future surgeries to minimize risk of complication.

Expertise is perceived from both sound and body movement in musical performance

Music is a rich form of nonverbal communication, in which the movements that expert musicians make during performance can influence the perception of expressive and structural features of the music. Whether the actual skill of a musician is perceivable from vision of movement was examined. In Experiment 1, musicians and non-musicians rated performances by novice, intermediate and expert clarinettists from point-light animations of their movements, sound recordings, or both. Performances by clarinettists of more advanced skill level were rated significantly higher from vision of movements, although this effect was stronger when sound was also presented. In Experiment 2, movements and sound from the novice and expert clarinettists’ performances were switched for half the presentations, and were matched for the rest. Ratings of novice music were significantly higher when presented with expert movements, although the opposite was not found for expert sound presented with novice movements. No perceptual effect of raters’ own level of musicianship was found in either experiment. These results suggest that expertise is perceivable from vision of musicians’ body movements, although perception of skill from sound is dominant. The results from Experiment 2 further indicate a cross-modal effect of vision and audition on the perception of musical expertise.

A Primer on the Validity of Assessment Instruments

A Primer on the Validity of Assessment Instruments

Generalizability Theory Applied to Reading Assessments for Students With Significant Cognitive Disabilities

Students with significant disabilities must participate in large-scale assessments, often using an alternate assessment judged against alternate achievement standards. The development and administration of this type of assessment must necessarily balance meaningful participation with accurate measurement. In this study, generalizability theory is used to estimate the dependability of reading items and tasks that have been administered using two formats of communication (receptive and expressive). The results reflect a trade-off between meaningful participation and accurate measurement of students with significant cognitive disabilities, particularly when considering the two formats. Significant variance is obtained for persons interacting with tasks, whereas the effect of raters is negligible. Furthermore, these results appear to vary across administrative format.

The Moderating Effect of Raters’ Opportunities to Observe Ratees' Job Performance on the Validity of an Assessment Centre

Previous research has suggested that the quality of judgmental validation criteria may improve depending on the increase of opportunity for the rater to observe ratees’ job performances. This study investigated whether the validity of an assessment centre increases with opportunity to observe. Subjects were engineers and applied scientists from the Research and Development (R&D) units of seven German high‐technology companies.A total of 6 personality scales, 10 cognitive ability tests and 9 work samples was administered to 155 subjects. Performance appraisals were collected from their supervisors. Assessors were trained organizational psychologists who were not members of the participating organizations. A total score (‘assessment centre score’) was computed by averaging the 25 predictors after standardization of each predictor. The (uncorrected) validity of this assessment centre was r = 0.37. In addition, opportunity to observe (i.e. amount of time the performance rater was the ratee’s supervisor) emerged as a moderator: for example validities were r = 0.09 (opportunity to observe ≤ 2 years) and r = 0.50 (opportunity to observe> 2 years).

Feature approaches to suspect identification: effect of multiple raters on system performance

Abstract In suspect identification, facial features of offenders are used to aid witnesses in identifying suspects. Feature approaches have been proposed as an alternative to the mug-shot album for identifying suspects. All feature approaches are based, at least in part, on subjective ratings by police or suspects on such facial features as length of hair: short 1 2 3 4 5 long. There is a strong, widely-held belief among researchers that since people vary considerably in their ratings of any given person's facial features, ratings from at least 10-12 police raters per suspect must be averaged to obtain an accurate estimate that can be used for comparison with witness feature descriptions. It is argued here that this is a myth, untested and unchallenged. Results from five experiments, and theoretical considerations support the contention that one, or at most two, police raters/suspect guarantee good feature system performance. For single-rater systems witnesses examine, depending on the experiment, an average of 15-55 suspect photographs before the actual offenders photograph is displayed (for police files containing 1000 mug-shots). For two-rater systems, witnesses examine an average of 2-12 photographs before retrieving the offender's photograph. More raters do not improve system performance further.

The effect of rater variables in the development of an occupation-specific language performance test

There has been little research into the effect of raters' background on assessments made in language tests. This article explores the effect that the rater's occupational and linguistic background has on assessments made in an occupation-specific oral language test, the Japanese Language Test for Tour Guides. Assessments of 51 test candidates made by 33 such assessors, including native and near-native speakers of Japanese with backgrounds either in teaching Japanese as a foreign language or in tour guiding in Japanese, were compared in order to determine what effect background has on assessments made on both linguistic and 'real-world' criteria. Multifaceted Rasch analysis was used, as this allows a range of facets to be modelled; for the purposes of this research we were interested in the facet 'raters', and in particular how the scale was applied for each of the assessment categories. These were then compared across the different types of rater. While differences were found in the assessments according to rater type, these were minor and did not point to the unsuitability of any group as a whole; there were no significant differences between the different types of rater in terms of the overall grade awarded. However, there were significant differences in ratings awarded for some individual criteria, and the fact that these reveal different perceptions of what constitutes good performances has obvious implications not only for the selection and training of raters for occupation- specific language performance tests but also for the development of assess ment procedures.

The effect of writers' personalities and raters' personalities on the holistic evaluation of writing

Recent work in reading and writing theory, research and pedagogy has raised questions about relationships between Fluent reading processes and holistic scoring of essays (e.g., Huot 1993). In holistic scoring settings, are the raters behaving as normal Fluent readers (i.e., readers interacting critically and personally with the text) or, are they somehow disconnected From their normal reader responses because they are using reliable scoring guides? Related questions concern the behavior of such holistic raters when they are teachers (e.g., Barritt, Stock, & Clark, 1986), and when those teachers respond to student writing (Connors & Lunsford, 1993). How are teachers/raters behaving, and what are they responding to in judging the writing? Previous research has suggested a role for personality type in the study of the process of writing evaluation (Jensen & DiTiberio, 1984, 1989). Thus, it seems reasonable to ask what role personality types play in the holistic evaluation of writing. This empirical study addressed the general question: What role, if any, do personality types of writers and of raters play in the holistic rating of writing? Moreover, is there a relationship between writers' personalities and raters' personalities? Writers were native English-speaking university freshman composition students; raters were native English-speaking university freshman composition instructors. Results indicate that the personality types of writers affect the ratings their essays receive, and the personality types of raters affect the ratings they give to essays. However, there is no significant relationship between writers' styles and raters' styles. Implications for future research, as well as classroom implications of these results are discussed.

The Effect of Raters' Background and Training on the Reliability of Direct Writing Tests

THROUGHOUT SCHOOL, STUDENTS ARE continually assessed on their ability to express themselves in writing.' This claim is no less true when working with a foreign language. Assessments often influence decisions made by institutions of higher education since the admission requirements of many universities include also a score in writing. Major examinations such as the TOEFL (Test of English as a Foreign Language) now require foreign students to demonstrate their writing proficiency in the second language (9).

The Effect of Raters' Background and Training on the Reliability of Direct Writing Tests

The Effect of Raters' Background and Training on the Reliability of Direct Writing Tests

Interrater Reliability of the Koppitz Developmental Scoring Method in the Clinical Evaluation of the Single Case

Using the Koppitz method, Bender-Gestalt protocols of three patients (8-12 yr.), representing three commonly occurring disorders, and of one normal child were evaluated by 12 raters of varying experience with this system. Raters were asked to make provisional diagnoses based on Bender protocols alone, and these were compared with actual clinical diagnoses. Despite significant interrater reliability of Koppitz Total Developmental Scores and emotional indices, no significant agreement was found in subscores or in diagnoses made, although the effect of raters' experience was noted. Results suggest that the type of dysfunction present is a factor in raters' reliability. Results are seen, however, to support the potential of the Bender-Gestalt for use in relatively subtle diagnosis of children and highlight the differentiation of "organic" versus "developmental" factors as an area for further study.