Performance-based Examination Research Articles

An evaluation of the inter-rater reliability in a clinical skills objective structured clinical examination

Background. An objective structured clinical examination (OSCE) is a performance-based examination used to assess health sciences students and is awell-recognised tool to assess clinical skills with or without using real patients.Objectives. To determine the inter-rater reliability of experienced and novice assessors from different clinical backgrounds on the final mark allocationsduring assessment of third-year medical students’ final OSCE at the University of the Free State.Methods. This cross-sectional analytical study included 24 assessors and 145 students. After training and written instructions, two assessors per station(urology history taking, respiratory examination and gynaecology skills assessment) each independently assessed the same student for the same skill bycompleting their individual checklists. At each station, assessors could also give a global rating mark (from 1 to 5) as an overall impression.Results. The urology history-taking station had the lowest mean score (53.4%) and the gynaecology skills station the highest (71.1%). Seven (58.3%) ofthe 12 assessor pairs differed by >5% regarding the final mark, with differences ranging from 5.2% to 12.2%. For two pairs the entire confidence interval(CI) was within the 5% range, whereas for five pairs the entire CI was outside the 5% range. Only one pair achieved substantial agreement (weightedkappa statistic 0.74 ‒ urology history taking). There was no consistency within or across stations regarding whether the experienced or novice assessorgave higher marks. For the respiratory examination and gynaecology skills stations, all pairs differed for the majority of students regarding the globalrating mark. Weighted kappa statistics indicated that no pair achieved substantial agreement regarding this mark.Conclusion. Despite previous experience, written instructions and training in the use of the checklists, differences between assessors were found inmost cases.

Using Think-aloud Interviews to Examine a Clinically Oriented Performance Assessment Rubric

ABSTRACT Performance-based assessment is a common approach to assess the development and acquisition of practice competencies among health professions students. Judgments related to the quality of performance are typically operationalized as ratings against success criteria specified within a rubric. The extent to which the rubric is understood, interpreted, and applied by assessors is critical to support valid score interpretations and their subsequent use. Therefore, the purpose of this study was to examine evidence to support a scoring inference related to assessor ratings on a clinically oriented performance-based examination. Think-aloud data showed that rubric dimensions generally informed assessors’ ratings, but specific performance descriptors were rarely invoked. These findings support revisions to the rubric (e.g., less subjective, rating-scale language) and highlight tensions and implications of using rubrics for student evaluation and making decisions in a learning context.

Impact of test anxiety on pharmacy students' performance in Objective Structured Clinical Examination: a cross-sectional survey.

To assess the association between test-related anxiety and pharmacy students' performance in the Objective Structured Clinical Examination (OSCE). A cross-sectional survey was conducted among Year-5 Doctor of Pharmacy (PharmD) students enrolled at a public university in Saudi Arabia. Students completed Test Anxiety Inventory (TAI), a 20-item validated questionnaire used to assess test-related anxiety, prior to the commencement of a summative OSCE. TAI has two subscales: TAI-Emotionality (TAI-E) and TAI-Worry (TAI-W). The overall maximum possible total score (TAI-T) is 80, with a minimum possible score of 20. All 25 students, 10 males and 15 females completed the survey. The overall mean scores for TAI-E, TAI-W and TAI-T were 20.2, 19.6 and 50.2, respectively. There were no statistically significant differences between males and females with respect to the TAI-E score (P=0.43), TAI-W score (P=0.25) and TAI-T score (P=0.34). Females had higher marks in the OSCE compared with males (P=0.01), however. After adjusting for gender, multiple linear regression analysis showed a statistically significant negative association between TAI-W score and marks obtained in OSCE (P=0.02; 95% CI=-0.42, -0.03). Assessment-related worry may negatively affect students' scores in performance-based examination and could lead to overall underperformance.

Multiple mini-interviews as a predictor of academic achievements during the first 2years of medical school.

BackgroundRecently, conventional interviews have been replaced with the multiple mini-interviews (MMI) for medical student selection in Korea. We first introduced the MMI as a new admissions tool in Korea. The aim of this study is to determine whether the MMI accurately predicts academic achievement on both written and performance-based examinations during the first 2 years of medical school.MethodsThe original scores of each station were standardized to T-scores in the candidates group. Three cohorts of students were included depending upon the year they entered medical school. Pearson’s correlations were calculated to estimate the correlations between MMI scores and academic achievements. Additional correlated factors were run through multiple stepwise linear regression analysis to estimate predictive validity.ResultsThere were no differences between T-scores or grade point averages (GPA) among the cohorts. The correlation coefficients between total MMI scores and academic achievement in Year 1 and the Year 2 performance-based examinations ranged from 0.17 to 0.43. Station 1 significantly predicted academic achievement over the second year. Station 3 significantly predicted only performance-based examination performance over the second year.ConclusionMMI is a useful tool to assist with medical student selection. In particular, critical thinking, professionalism, and presentation and communication skills may be meaningful topics for predicting academic achievements, especially in performance-based subjects.

Scenarios and Standardized Professionals From the Objective Structured System-Interaction Examinations (OSSIEs) of an Internal Medicine Residency Program

Abstract The purpose of this study is to make available information regarding standardized professionals, as used by the Southern Illinois University School of Medicine Internal Medicine Residency ...

Systems-Based Practice Assessed With a Performance-Based Examination Simulated and Scored by Standardized Participants in the Health Care System: Feasibility and Psychometric Properties

Background: Systems-based practice is one of the six general competencies proposed by the Accreditation Council for Graduate Medical Education in their Outcome Project. However, little has been published on its assessment—possibly because the systems-based practice competency has been viewed as difficult to define and measure. Purpose: The purpose of this study was to determine whether a full performance-based examination of systems-based practice cases simulated and scored by standardized participants in the health care system could feasibly be constructed and implemented that would provide reliable and valid measurements. Methods: In the 1st year of the project (2008), four systems-based practice cases were developed and pilot tested with 13 residents. Videotapes of residents were studied to develop an instrument for subsequent assessment of performance by standardized participants. In the 2nd year (2009), the examination was expanded to a full 12 cases, which were completed by 11 second-year residents, and psychometric analyses were performed on the scores. Results: The generalizability coefficient for the full 12-case examination based on scoring by standardized participants was .71, which is nearly equal to that based on scoring by faculty physician observers, which was .78. The correlation between total scores obtained with standardized participants and physician observers was .78. Conclusions: A performance-based examination can provide a feasible and reliable assessment of systems-based practice. However, attempts to evaluate convergent validity and discriminant validity—by correlating systems-based practice performance assessments with mean global ratings of residents on the 6 competencies by faculty throughout training—were unsuccessful, due to a lack of independence between the rated dimensions.

Who Should Be an OSCE Examiner?

In this issue, Whelan et al. (1) address the question of who (standardized patients or physicians) should grade the performance of students taking Objective Structured Clinical Examinations (OSCEs). This is an interesting issue that has educational as well as socioeconomic dimensions. The last three decades of the 20th century were characterized by a significant shift in the way physician competence in Anglo-Saxon countries has been defined and assessed. The adoption of performance-based frameworks, such as the influential “Millers Pyramid” (2), placed more emphasis on what physicians could do rather than what they know. During the same period, the influence of medical educators with training in psychometrics led to a much greater emphasis on standardization, reliability, and validity in assessment. Together, the adoption of performance and psychometric discourses created a fertile ground for new assessment technologies such as the OSCE. No longer a novelty at the end of the first decade of the 21st century, OSCEs have been widely implemented by health professions around the world, including psychiatry (3). Mental health professionals need no convincing that one of the core competencies tested in an OSCE, or in any performance-based examination for that matter, is communication skills. However, whether communication skills are a unified construct is less clear. Our group (4) has reported that the appropriateness of specific communication skills (e.g., open-ended versus directed questioning) varies greatly according to the clinical problem encountered. For example, the often-taught communication style that gives priority to open-ended questions and listening is appropriate to a passive and withdrawn patient but entirely inadequate for an agitated manic patient. And although the first item of most communication scales is “makes eye contact,” we know that in some cultures direct eye contact is considered intrusive and uncomfortable. Therefore, to some degree, what is “appropriate” competence in a performance-based examination is a matter of perspective. So who is best positioned to assess the adequacy of student competence in a performance-based examination? Whelan et al. (1) follow the tradition of addressing this question from a psychometric perspective of “accuracy” (5–7), that is, the rater who is best able to reliably and consistently (psychometrically) score performances is considered the most appropriate examiner. This raises the interesting issue of what it actually means to be an examiner. At one extreme, we have interviewed individuals who argued, “There are no evaluators in the room, there are merely observers” (8). The implication is that the markers are “merely identifying behaviors that individuals perform and [that] it is the responsibility of the test administrators to compile those records into evaluations and numbers.” This idea arises from the often-made, but seldom-explicated, distinction between “assessment” and “evaluation” (9). Assessment is a process by which information is obtained relative to some known objective or goal. Assessment of skill attainment is rather straightforward. Either the skill exists at some acceptable level or it does not. Skills are readily demonstrable. Inherent in the idea of evaluation is “value.” When we evaluate, what we are doing is engaging in some process that is designed to provide information that will help us make a judgment about a given situation. When we evaluate, we are saying that the process will yield information regarding the worthiness, appropriateness, goodness, validity, legality, etc., of something for which a reliable measurement or assessment has been made. From this perspective, the “veracity” of the recording of a dispassionate and neutral observer is all that matters for “reliable assessment.” This view is congruent with a positivist conception that there is a reality/truth that can be Received September 8, 2008; accepted September 18, 2008. The authors are affiliated with the Department of Psychiatry at the Wilson Centre for Research in Education, University of Toronto, in Toronto, Ontario. Address correspondence to Brian David Hodges, M.D., Ph.D., Wilson Centre for Research in Education, Department of Psychiatry/University of Toronto, University Health Network-Toronto General Hospital, 200 Elizabeth St., 8 Easton, Rm 212, Toronto, Ontario M5G-2C4, Canada; brian. hodges@utoronto.ca (e-mail). Copyright © 2009 Academic Psychiatry

Design and Implementation of a Video-based Clinical Examination

Objectives: To (1) describe the development of a Video-based Clinical Examination (VCE) as a formal testing format to evaluate student ability to make an accurate pharmaceutical assessment and recommendation, and (2) determine student perception of the VCE testing format.Methods: Descriptive study of first-year pharmacy students. Results: One hundred and twenty-nine students were included in the study. Students perceived that the VCE testing format provided a real life/interactive environment but felt rushed as the video segments of the patient/pharmacist interaction occurred quickly.Implications: Based on the findings of this project, we will continue to pursue further research related to validity, reliability and application of VCEs. However, the University of Colorado will continue to incorporate VCEs in the performance based evaluations in the Professional Skills Development 1 course, as it appears to be an effective stepping-stone for first-year students to begin developing their active listening, higher level learning and problem-solving skills. Results of this project will be shared with the faculty and curriculum committee at the University of Colorado School of Pharmacy to encourage further use and research of VCEs in other courses. Keywords: Knowledge retention, clinical competency, performance-based examination, problembased examination

An Assessment of the Impact of Multimedia, Technology-Based Learning Tools on the Cardiac Auscultation Skills of Third-Year Medical Students

:Background: Previous studies have shown that medical students and post-graduate trainees need to improve their proficiency in cardiac auscultation. Technologic advances have created new learner-centered opportunities to enhance proficiency in this important physical examination skill.Objectives: We sought to determine if technology-based, self-directed learning tools improved the cardiac auscultation skills of third-year medical students.Methods: Sixteen (16) third-year medical (M3) students were exposed to three educational inter-ventions: a one-hour cardiac auscultation lecture that featured computer-generated heart sounds, a PDA-based heart sounds/murmur form and a web-based cardiac auscultation program. Thirteen (13) internal medicine (IM) residents who served as a comparison group attended a cardiac auscultation lecture identical in content and format to the student lecture. At the end of the study period, we evaluated the ability of both groups to accurately identify heart sounds and cardiac murmurs via a twelve-item performance-based examination utilizing computer-generated heart sounds.Results: Following our teaching interventions, findingsM3 students correctly identified 80% of the computer-simulated heart sounds/murmurs while the comparison group of IM residents accurately detected 60% of the same cardiac findings (p<. 005).Conclusions: The combination of traditional lecture and multi-media, technology-based, self-directed learning tools appears to be an effective and efficient strategy for teaching and reinforcing cardiac auscultation skills to third year medical students.

Evaluating the clinical skills of osteopathic medical students.

Because clinical skills play an important role in health services, many medical credentialing organizations are making performance-based assessments part of the board-certification and licensure processes. While clinical skills are taught and evaluated at colleges of osteopathic medicine, the development and validation of standardized assessment methodologies is far from complete. The purpose of this study was to gather data to support the use of a performance-based assessment of osteopathic clinical skills. A sample of 121 fourth-year osteopathic medical students was tested using the Comprehensive Osteopathic Medical Licensing Examination-USA performance-based clinical skills examination (COMLEX-USA-PE) prototype, a standardized patient performance evaluation that involves a series of 12 simulated encounters. Students were evaluated in a number of domains that included history taking, physical examination, osteopathic manipulative treatment techniques, written communication and clinical problem solving, and physician-patient communication. The analysis of data from 1452 standardized patient encounters suggests that reliable and valid scores can be obtained using the current prototype. The use of COMLEX-USA-PE to assess the readiness of osteopathic medical students to provide patient care in supervised graduate medical education training programs is supported.

Performance assessment of community-based physicians: evaluating the reliability and validity of a tool for determining CME needs.

To evaluate the reliability, validity, and feasibility of the Physician Assessment in Medical Practice (PAMP) as a means of determining the CME needs of practicing, community-based physicians. A group of 45 randomly selected community-based physicians (19 certified family physicians and 26 general practitioners) affiliated with the Department of Family Medicine at Bruce Rappaport Faculty of Medicine, Technion Institute of Technology, Haifa, Israel, volunteered to participate in the study, conducted in 1997. All participants took a ten-station, performance-based examination designed to closely represent the physicians' work settings. At each station, a different medical problem was presented by a standardized patient. Physician-candidates' performances were assessed by physician-examiners using global ratings. The following performance domains were assessed: information gathering, diagnosis and management plan, and communication skills. A CME needs assessment score was determined for each of the participants and a CME level to meet the needs of the physician was recommended. Overall reliability of the examination was high (.87), with domain reliabilities ranging from.76 to.87. Reliability of the examiners' judgments of the physicians' competence was.66. All the stations' validity scores were significant, and differences in performances between family physicians and general practitioners demonstrated construct validity of the test results. Overall, the cost of running the examination was U.S. $250 per physician-candidate. Using the PAMP to determine CME needs of community-based physicians was found to reliable, valid, and feasible, and the cost per physician-candidate was not excessive. Performance results provided indepth information for use by both the individual physician and providers of CME programs.

Exporting a technical skills evaluation technology to other sites

Background: The Objective Structured Assessment of Technical Skill (OSATS) is a multistation performance-based examination that assesses the technical skills of surgery residents. This study explores the implementation issues involved in remote administration of the OSATS focusing on feasibility and the psychometric properties of the examination. Methods: An eight-station OSATS was administered to surgical residents in Los Angeles and Chicago. The University of Toronto and the local institutions shared responsibility for organization and administration of the examination. Results: There was good reliability for both the checklist (α = 0.68 for LA, 0.73 for Chicago) and global rating forms (α = 0.82 for both sites). Both iterations also showed evidence of construct validity, with a significant effect of training year for the checklist and global rating forms at both sites (analysis of variance: F = 8.66 to 19.93, P <0.01). Despite some challenges, the model of central organization and peripheral delivery was effective for the administration of the examinations. Conclusions: Two iterations of the OSATS at remote sites demonstrated psychometric properties that are highly consistent with previously reported data suggesting that the examination is portable. Both faculty and residents indicated satisfaction with the examination experience. A model of central administration with peripheral delivery was feasible and effective.

Assessment of Empathy in a Standardized-Patient Examination

Background: Empathy has long been thought to be an important characteristic of a good physician, and a measure of empathy is needed to provide feedback to medical students, residents, and physicians on this important aspect of their clinical performance. The standardized-patient-testing format provides a simple but intuitively convincing approach to the assessment of empathy. Purpose: To determine the extent to which 4th-year medical students were checked "empathic" by standardized patients (SPs) on a performance-based examination, to evaluate the psychometric properties of this simple empathy measure, and to see whether empathy was related to clinical performance on history taking and physical examination. Method: Analyses were performed on examination data for 1,048 senior medical students in the 8 member schools of the New York City Consortium tested at The Morch and Center of Mount Sinai School ofMedicine. Results: The percentage of students who were checked "empathic" ranged from 59% to 98% (M = 79%) across the 7 SP cases. Of the 1,048 students, 268 (26%) were checked "empathic" on all 7 cases; however, 221 (21%)were checked on ≤4 cases, 90 (9%)on ≤3 cases, and 26 (3%)on ≤2 cases. The generalizability coefficient of the overall empathy scores was. 43; the dependability index with cutoff was. 81 for detecting students checked "empathic" on fewer than half of the 7 cases. The correlations of the empathy item with the other checklist items suggest that the empathy construct refers to behaviors that make the patient feel comfortable and important. Also, students who were checked "empathic" on fewer than half of the 7 cases performed lower on history taking and physical examination. Conclusions: Empathy appears to be reasonably acceptable in this sample of students, although it is of concern that, on average, more than 200 students per case were not seen as empathic, and more than 200 were checked "empathic" on ≤4 of the 7 cases. These results show the potential usefulness of this simple measure of empathy and illustrate the need for feedback to address any problems.

High, usual and impaired functioning in community-dwelling older men and women: Findings from the MacArthur Foundation Research Network on successful aging

The objective of this study is to determine the range of complex physical and cognitive abilities of older men and women functioning at high, medium and impaired ranges and to determine the psychosocial and physiological conditions that discriminate those in the high functioning group from those functioning at middle or impaired ranges. The subjects for this study were drawn from men and women aged 70–79 from 3 Established Populations for the Epidemiologic Study of the Elderly (EPESE) programs in East Boston MA, New Haven CT, and Durham County NC screened on the basis of criteria of physical and cognitive function. In 1988, 4030 men and women were screened as part of their annual EPESE interview. 1192 men and women met criteria for “high functioning”. Age and sex-matched subjects were selected to represent the medium (n = 80) and low (n = 82) functioning groups. Physical and cognitive functioning was assessed from performance-based examinations and self-reported abilities. Physical function measures focused on balance, gait, and upper body strength. Cognitive exams assessed memory, language, abstraction, and praxis. Significant differences for every performance-based examination of physical and cognitive function were observed across functioning groups. Low functioning subjects were almost 3 times as likely to have an income of ⩽$5000 compared to the high functioning group. They were less likely to have completed high school. High functioning subjects smoked cigarettes less and exercised more than others. They had higher levels of DHEA-S and peak expiratory flow rate. High functioning elders were more likely to engage in volunteer activities and score higher on scales of self-efficacy, mastery and report fewer psychiatric symptoms.

Station-length requirements for reliable performance-based examination scores

To directly compare the generalizability of medical students' performance scores under systematically varied station times in two surgery end-of-clerkship performance-based examinations. The participants were 36 third-year students randomly assigned to the first two rotations of the core surgery clerkship during 1991-92 at Southern Illinois University School of Medicine. The students rotated through a 12-station examination that employed standardized patients (SPs). In the first rotation, the student took six five-minute stations and six ten-minute stations. In the second rotation, the time lengths were reversed for the same stations. The students' total scores were based on (1) subscores on checklists that were completed by the SPs and (2) subscores on the students' written responses to short questions about each station (these responses were provided at station couplets that were five minutes long, regardless of station length). Generalizability coefficients were computed from the pooled rotation results to provide reliabilities for scores from the two station lengths. Generalizability decreased in the ten-minute stations, mostly attributable to less variability among students' performances. The checklist subscores accounted for most of this variability, while couplet subscores remained stable between station lengths. The longer station length actually decreased the generalizability of the scores by decreasing the variability among students' performances; thus, allocating different times to stations can affect the score reliability, as well as impact on the overall testing time, of performance-based examinations.

Effects of examinee gender, standardized-patient gender, and their interaction on standardized patients' ratings of examinees' interpersonal and communication skills.

To assess the effects of examinee gender, standardized-patient (SP) gender, and, in particular, their interaction on ratings made by SPs of examinees' interpersonal and communication skills in a performance-based examination of clinical competence. The examination was administered to four classes of senior medical students (about 70 per class) at Southern Illinois University School of Medicine, 1988-1991. The skill dimensions tested were clarity of communication, thoroughness of explanation, professional manner, personal manner, and overall patient satisfaction. Split-plot analyses of variance were used. There was no interaction of examinee gender and SP gender for any of the five rating scales. There was no main effect of examinee gender for four of the five scales; however, for personal manner, women students were rated slightly higher than men students. There was a main effect of SP gender, but the effect was not consistent from rating scale to rating scale or from class to class. Nevertheless, differences in ratings given by men and women SPs should not be of psychometric concern, since the ratings of men and women examinees are necessarily affected alike. Except for the women examinee's higher performance in personal manner, the men and women examinees generally performed equally well with respect to interpersonal and communication skills, and they performed equally well regardless of the gender of the SP.

A survey of study methods, preparation time, test-taking strategies, and perceptions of test validity on a clinical performance-based examination.

A survey of study methods, preparation time, test-taking strategies, and perceptions of test validity on a clinical performance-based examination.

Screening test length for sequential testing with a standardized-patient examination: a receiver operating characteristic (ROC) analysis.

The feasibility of using sequential testing (i.e., using a screening test) to reduce the length and expense of a performance-based examination with standardized-patient cases was demonstrated previously. In the present study, quantitative criteria rather than practical considerations were used to determine optimal values for the length of the screening test (i.e., number of cases) and the location of the screening pass-fail cutoff (i.e., its relation to the mean of the pass levels for the different cases). Data were derived from five classes of senior students at the Southern Illinois University School of Medicine, 1987-1991. Specifically, receiver operating characteristic (ROC) curves were plotted for screening tests of varying lengths, with the points on each ROC curve corresponding to different pass-fail cutoffs on the screening test. The results showed that good accuracy can be attained with a screening test that is only one-third the length of the full examination and that the cutoff for this screen should be set slightly above the mean of the case pass levels to maximize sensitivity and specificity. The authors conclude that their study demonstrates the value of an ROC analysis in evaluating the psychometric properties of a screening test in sequential testing.

Six years of comprehensive, clinical, performance-based assessment using standardized patients at the Southern Illinois University School of Medicine

By the end of 1990-91, the Southern Illinois University School of Medicine had had six years of experience with comprehensive, performance-based examinations of senior medical students' levels of clinical competence; this report assesses the psychometric aspects of the six examinations given during that period. The examinations were aimed at determining the students' readiness for postgraduate training. Compared with other clinical performance-based assessments that use standardized patients (SPs), these examinations had two important and unique features: (1) the examinations assessed a comprehensive range of clinical skills and reasoning; and (2) they approximated the challenges of real clinical practice wherein a practitioner's skills need to be orchestrated and prioritized in order to meet the challenges of the case encountered. Each year, the performance-based assessment given was an intensive clinical examination requiring each student to work up 13 to 18 SP problems over a three-day period. To administer an examination to an entire class of students took three weeks. Because all students after the first year of administration (1986) were required to pass these examinations, the fairness of test design and scoring and the setting of performance standards for the examinations became important issues for the faculty. The results, accumulated over six years and based on a total of 6,804 student-patient encounters involving 405 students, indicate that this kind of clinical performance-based examination can discriminate a wide range of students' clinical performances. The results provide evidence for the examinations' test security, content validity, construct validity, and reliability.

Effects of using two or more standardized patients to simulate the same case on case means and case failure rates

The effects of using two or more standardized patients (multiple SPs) to simulate the same case in a performance-based examination were studied at the case level by comparing case means and case failure rates for multiple SPs simulating the same case, using data from the classes of 1988, 1989, and 1990 at the Southern Illinois University School of Medicine. For total scores and scores on the students' written answers, the effects on means and failure rates were negligible and could be explained as due to sampling error. For scores on the checklists completed by the SPs, there were more significant differences than would be expected by chance alone, even though the number of significant differences was relatively small. The results demonstrate a need for caution in the interpretation of scores obtained from a case checklist completed by multiple SPs, particularly in regard to making pass-fail decisions.