Use Of Global Rating Scale Research Articles

Use of global rating scales and checklists in clinical simulation-based assessments: a protocol for a scoping review

IntroductionAssessment in health sciences education remains a hotly debated topic, with measures of competency and how to determine them in simulation-based assessments enjoying much of the focus. Global rating scales...

The use of global rating scales for OSCEs in veterinary medicine.

OSCEs (Objective Structured Clinical Examinations) are widely used in health professions to assess clinical skills competence. Raters use standardized binary checklists (CL) or multi-dimensional global rating scales (GRS) to score candidates performing specific tasks. This study assessed the reliability of CL and GRS scores in the assessment of veterinary students, and is the first study to demonstrate the reliability of GRS within veterinary medical education. Twelve raters from two different schools (6 from University of Calgary [UCVM] and 6 from Royal (Dick) School of Veterinary Studies [R(D)SVS] were asked to score 12 students (6 from each school). All raters assessed all students (video recordings) during 4 OSCE stations (bovine haltering, gowning and gloving, equine bandaging and skin suturing). Raters scored students using a CL, followed by the GRS. Novice raters (6 R(D)SVS) were assessed independently of expert raters (6 UCVM). Generalizability theory (G theory), analysis of variance (ANOVA) and t-tests were used to determine the reliability of rater scores, assess any between school differences (by student, by rater), and determine if there were differences between CL and GRS scores. There was no significant difference in rater performance with use of the CL or the GRS. Scores from the CL were significantly higher than scores from the GRS. The reliability of checklist scores were .42 and .76 for novice and expert raters respectively. The reliability of the global rating scale scores were .7 and .86 for novice and expert raters respectively. A decision study (D-study) showed that once trained using CL, GRS could be utilized to reliably score clinical skills in veterinary medicine with both novice and experienced raters.

Raters’ decisions, rating procedures and rating scales

This special issue of Language Testing explores raters’ evaluations of L2 proficiency and possible causes of variability of rater judgments. In addition to the analysis of rater behavior and rater consistency, we investigate the relationship between general measures of oral and written L2 performance concerning complexity, accuracy and fluency of L2 production and overall judgments of oral and written L2 performance by raters, based on holistic rating scales. Finally, the use of rating scales in different contexts and for different types of learners is also examined. In this introduction the three central themes presented in the various contributions are briefly discussed: rater behavior and rater consistency, rater judgments and measures of language performance, and the use of global rating scales.

Improving the content and face validity of OSCE assessment marking criteria on an undergraduate midwifery programme: A quality initiative

Objective Structured Clinical Examinations (OSCE's) have been adopted as a means of assessing midwifery students' clinical skills. The purpose of the OSCE is to provide a standardised method for the evaluation of clinical skill performance in a simulated environment. This paper describes how a quality improvement initiative using both internal and external expert review was utilised to improve OSCE assessment marking criteria. The purpose of the quality initiative was to review the content and face validity of the marking criteria for assessing performance. The design and choice of tools used to score students' performance is central to reliability and validity. 20 videos of students from year one of a midwifery preregistration programme undertaking an OSCE assessment on abdominal examination and 18 videos of students response to obstetric emergencies e.g. PPH, and shoulder dystocia were available for review. The quality initiative aimed to strengthen the reliability and validity of the OSCE in assessing student performance. Conclusion: the use of global rating scales allows for the capturing of elements of professional competency that do not appear on specific criteria for skills performance checklists.

Comparing the use of global rating scale with checklists for the assessment of central venous catheterization skills using simulation

The use of checklists is recommended for the assessment of competency in central venous catheterization (CVC) insertion. To explore the use of a global rating scale in the assessment of CVC skills, this study seeks to compare its use with two checklists, within the context of a formative examination using simulation. Video-recorded performances of CVC insertion by 34 first-year medical residents were reviewed by two independent, trained evaluators. Each evaluator used three assessment tools: a ten-item checklist, a 21-item checklist, and a nine-item global rating scale. Exploratory principal component analysis of the global rating scale revealed two factors, accounting for 84.1% of the variance: technical ability and safety. The two checklist scores correlated positively with the weighted factor score on technical ability (0.49 [95% CI 0.17-0.71] for the 10-item checklist; 0.43 [95% CI 0.10-0.67] for the 21-item checklist) and negatively with the weighted factor score on safety (-0.17 [95% CI -0.48-0.18] for the 10-item checklist; -0.13 [95% CI -0.45-0.22] for the 21-item checklist). A checklist score of <80% was strong indication of incompetence. However, a high checklist score did not preclude incompetence. Ratings using the global rating scale identified an additional 11 candidates (32%) who were deemed incompetent despite scoring >80% on both checklists. All these candidates committed serious errors. In conclusion, the practice of universal adoption of checklists as the preferred method of assessment of procedural skills should be questioned. The inclusion of global rating scales should be considered.

The Role of Family Emotional Expressiveness in the Development of Children's Social Competence

Social competence is important for obtaining and maintaining social and academic success, and for mental health and personal adjustment in childhood, adolescence, and early adulthood. Problematic peer relationships have been shown to predict such extreme outcomes as school drop out, criminality, and psychopathology (Parker & Asher, 1987). Several aspects of emotion, such as encoding and decoding, emotion regulation, and emotional expressiveness (Cassidy, Parke, Butkovsky, & Braungart, 1992; Hubbard & Coie, 1994; Parke, Cassidy, Burks, Carson, & Boyum, 1992) have been suggested as antecedents of peer competence. There has been increasing interest in exploring the antecedents of children's social behaviors, particularly in examining, family influences on the development of these behaviors (Parke & Ladd, 1992). The focus of this report is on the role of family emotion in the emergence of children's social competence. The family provides the first context for the recognition and communication of affective messages. Family members send affective messages to the child, with increasing expectation that the child will be able to interpret and respond to them. As the child moves outside of the immediate family environment, she will likely encounter others who do not share contextual cues with the child's family, do not share all of the same display rules, and who may be less willing to make the extra effort to understand the child. In addition, these people may communicate their own needs to the child in ways that may be less clear than those the child has experienced at home. Thus, it is to the child's advantage to be well versed in emotion skills as she moves beyond the confines of the family. Several studies have demonstrated a connection between parent-child interactions in structured game or physical play situations and children's social status with peers (e.g., Barth & Parke, 1993; MacDonald & Parke, 1984; Putallaz, 1987). It may be that through engaging in affectarousing encounters such as physical play, the child learns the communicative value of affect and how to use such displays to understand and regulate others (Parke et al., 1992). Physical play allows for many opportunities to learn to encourage or discourage interaction through affective expression. Past research has focused on the type of affective display used by parents during their interactions with their children. Several studies have found that children who are exposed to higher levels of parental negative affect are less well accepted by their peers, while high levels of parental positive affect are associated with greater peer acceptance (Carson, 1991; MacDonald Parke, 1984). It is assumed that children who learn to use high levels of negative affect in interaction with peers are less well accepted. Those who use more positive affect are more accepted by their age mates (Hubbard & Coie, 1994). Several limitations characterize these earlier studies. First, the studies have been conducted primarily in laboratory situations and have relied on structured games or physical play situations. One goal of the present study is to explore the generalizability of prior findings by observing family affective exchanges during an ecologically valid situation, namely a family dinner. Second, affect has been measured primarily through the use of global rating scales, rather than specific affect categories. Prior research has revealed the value of focusing on specific affective dimensions for understanding both marital and parent-child interaction patterns (Gottman, 1994; Gottman & Katz, 1989). This work and related studies (Roberts & Strayer, 1987) have suggested that not all forms of negative affect have similar effects. While harsh, angry affect is clearly associated with negative outcomes, moderate levels of negative affect may have more positive social outcomes. In this study, we assume that exposure to moderate levels of negative affect may allow children opportunities to learn to regulate negative affect, which is an important social skill (Gottman & Katz, 1989; Sroufe, 1979). …