High-stakes Objective Structured Clinical Examination Research Articles

Exploring the Relationship Between Examination-Induced Anxiety and Cardiovascular Responses Among PT Students

This study explores the relationship between examination-induced anxiety and cardiovascular responses among physical therapy (PT) students. A quasi-experimental repeated measures design was utilized to examine heart rate (HR), blood pressure (BP), and heart rate variability (HRV) at three stages: before, during, and after high-stakes Objective Structured Clinical Examinations (OSCEs). A total of 120 PT students (60 male, 60 female) were recruited using purposive sampling, with data collection extended over two academic years to ensure balanced gender representation. Anxiety was measured using the State-Trait Anxiety Inventory (STAI), while cardiovascular responses were continuously monitored using wearable devices. Results revealed significant increases in HR and BP, alongside decreases in HRV during the examination, reflecting heightened autonomic stress. Post-exam, cardiovascular markers demonstrated partial recovery but remained elevated compared to pre-exam levels, indicating lingering physiological effects of anxiety. Qualitative data from student interviews revealed cognitive impairments and lingering anxiety post-exam, corroborating the physiological findings. Repeated measures ANOVA indicated significant changes in cardiovascular responses over time, with no significant interaction effects for gender. These findings underscore the impact of examination-induced anxiety on both physiological and psychological outcomes in PT students. The persistent elevation in cardiovascular markers post-exam suggests that students may not fully recover from stress immediately, highlighting the need for targeted interventions, such as mindfulness or biofeedback, to help manage anxiety. Addressing examination stress could enhance both student well-being and academic performance, mitigating the long-term risks associated with chronic stress.

Validity evidence for the clinical communication skills assessment tool (CCSAT) from 9 years of implementation in a high stakes medical student OSCE

ObjectivesCommunication and other clinical skills are routinely assessed in medical schools using Objective Structured Clinical Examinations (OSCEs) so routinely that it can be difficult to monitor and maintain validity. We report on the accumulation of validity evidence for the Clinical Communication Skills Assessment Tool (CCSAT) based on its use with 9 cohorts of medical students in a high stakes OSCE. MethodsWe describe the implementation of the CCSAT including information on the underlying model, the tool’s items, domains, scales and scoring, and its role in curriculum. Internal structure is explored through item, internal consistency, and confirmatory factor analyses. Evidence for CCSAT validity is synthesized within prevailing frameworks (Messick12 and Kane13) based on continuous quality improvement and use of the CCSAT for feedback, remediation, curricular design, and research. ResultsImplementation of the CCSAT over time has facilitated our communication skills curriculum and training. Thoughtful case development and investment in standardized patient training has contributed to data quality. Item analysis supports our behaviorally anchored scale (not done, partly and well done) and the skills domains suggested by an a priori evidence-based clinical communication model were confirmed via analysis of actual student data. Evidence synthesized across the frameworks suggests consistent validity of the CCSAT for generalization inferences (that it captures the construct), responsiveness (sensitivity to change/difference), content validity/internal structure, relationships to other variables, and consequences/implications. More evidence is needed to strengthen validity of CCSAT scores for understanding extrapolation inferences and real-world implications. Conclusions and Practice ImplicationsThis pragmatic approach to evaluating validity within a program of assessment serves as a model for medical schools seeking to continuously monitor the quality of clinical skill assessments, a need made particularly relevant since the US NBME no longer requires the Step 2 Clinical Skills exam, leaving individual schools with the responsibility for ensuring graduates have acquired the requisite core clinical skills. We document strong evidence for CCSAT validity over time and across cohorts as well as areas for improvement and further examination.

Implementation of standard setting for high-stakes objective structured clinical examinations

Background and purposeTo describe one institution's approach to transformation of high-stakes objective structure clinical examinations (OSCEs) from norm-referenced to criterion-referenced standards setting and to evaluate the impact of these changes on OSCE performance and pass rates. Educational activity and settingThe OSCE writing team at the college selected a modified Angoff method appropriate for high-stakes assessments to replace the two standard deviation method previously used. Each member of the OSCE writing team independently reviewed the analytical checklist and calculated a passing score for active stations on OSCEs. Then the group met to determine a final pass score for each station. The team also determined critical cut points for each station, when indicated. After administration of the OSCEs, scores, pass rates, and need for remediation were compared to the previous norm-referenced method. Descriptive statistics were used to summarize the data. FindingsOSCE scores remained relatively unchanged when switched to a criterion-referenced method, but the number of remediators increased up to 2.6 fold. In the first year, the average score increased from 86.8% to 91.7% while the remediation rate increased from 2.8% to 7.4%. In the third year, the average increased from 90.9% to 92% while the remediation rate increased from 6% to 15.6%. Likewise, the fourth-year average increased from 84.9% to 87.5% while the remediation rate increased from 4.4% to 9%. SummaryTransition to a modified Angoff method did not impact average OSCE score but did increase the number of remediations.

Using cultural historical activity theory to reflect on the sociocultural complexities in OSCE examiners’ judgements

Examiners’ judgements play a critical role in competency-based assessments such as objective structured clinical examinations (OSCEs). The standardised nature of OSCEs and their alignment with regulatory accountability assure their wide use as high-stakes assessment in medical education. Research into examiner behaviours has predominantly explored the desirable psychometric characteristics of OSCEs, or investigated examiners’ judgements from a cognitive rather than a sociocultural perspective. This study applies cultural historical activity theory (CHAT) to address this gap in exploring examiners’ judgements in a high-stakes OSCE. Based on the idea that OSCE examiners’ judgements are socially constructed and mediated by their clinical roles, the objective was to explore the sociocultural factors that influenced examiners’ judgements of student competence and use the findings to inform examiner training to enhance assessment practice. Seventeen semi-structured interviews were conducted with examiners who assessed medical student competence in progressing to the next stage of training in a large-scale OSCE at one Australian university. The initial thematic analysis provided a basis for applying CHAT iteratively to explore the sociocultural factors and, specifically, the contradictions created by interactions between different elements such as examiners and rules, thus highlighting the factors influencing examiners’ judgements. The findings indicated four key factors that influenced examiners’ judgements: examiners’ contrasting beliefs about the purpose of the OSCE; their varying perceptions of the marking criteria; divergent expectations of student competence; and idiosyncratic judgement practices. These factors were interrelated with the activity systems of the medical school’s assessment practices and the examiners’ clinical work contexts. Contradictions were identified through the guiding principles of multi-voicedness and historicity. The exploration of the sociocultural factors that may influence the consistency of examiners’ judgements was facilitated by applying CHAT as an analytical framework. Reflecting upon these factors at organisational and system levels generated insights for creating fit-for-purpose examiner training to enhance assessment practice.

Improving the assessment of communication competencies in a national licensing OSCE: lessons learned from an experts\u2019 symposium

BackgroundAs the communication competencies of physicians are crucial for providing optimal patient care, their assessment in the context of the high-stakes Objective Structured Clinical Examination (OSCE) is of paramount importance. Despite abundant literature on the topic, evidence-based recommendations for the assessment of communication competencies in high stakes OSCEs are scarce. As part of a national project to improve communication-competencies assessments in the Swiss licensing exam, we held a symposium with national and international experts to derive corresponding guidelines.MethodsExperts were invited on account of their recognized expertise either in teaching or assessing communication competencies, or in conducting national high-stakes OSCEs. They were asked to propose concrete solutions related to four potential areas for improvement: the station design, the rating tool, the raters’ training, and the role of standardized patients. Data gene.rated in the symposium was available for analysis and consisted of video recordings of plenary sessions, of the written summaries of group work, and the cards with participants’ personal take-home messages. Data were analyzed using a thematic analysis approach.ResultsNine major suggestions for improving communication-competencies assessments emerged from the analysis and were classified into four categories, namely, the roles of the OSCE scenarios, rating tool, raters’ training, and simulated patients.ConclusionIn the absence of established evidence-based guidelines, an experts’ symposium facilitated the identification of nine practical suggestions for improving the assessment of communication competencies in the context of high-stakes OSCEs. Further research is needed to test effectiveness of the suggestions and how they contribute to improvements in the quality of high-stakes communication-competencies assessment.

Impact of Structured Feedback on Examiner Judgements in Objective Structured Clinical Examinations (OSCEs) Using Generalisability Theory

PurposeIn the context of health professions education, the objective structured clinical examination (OSCE) has been implemented globally for assessment of clinical competence. Concerns have been raised about the significant influence of construct irrelevant variance arising from examiner variability on the robustness of decisions made in high-stakes OSCEs. An opportunity to explore an initiative to reduce examiner effects was provided by a secondary analysis of data from a large-scale summative OSCE of the final-year students (n > 350) enrolled in a graduate-entry four-year Bachelor of Medicine/Bachelor of Surgery (MBBS) program at one Australian research-intensive university. The aim of this study was to investigate the impact of providing examiners with structured feedback on their stringency and leniency on assessing the final-year students’ clinical competence in the pre-feedback (P1) OSCE and post-feedback (P2) OSCE. MethodThis study adopted a quasi-experimental design to analyse the scores from 141 examiners before feedback was provided for the P1 OSCE, and 111 examiners after feedback was provided for the P2 OSCE. This novel approach used generalisability theory to quantify and compare the examiner stringency and leniency variance (Vj) contributing to the examiners’ scores before and after feedback was provided. Statistical analyses conducted were controlled for differences in the examiners and OSCE stations. ResultsComparing the scores of the 51 examiners who assessed students in both P1 and P2 OSCEs, the Vj reduced by 35.65% and its contribution to the overall variation in their scores decreased by 7.43%. The results were more noticeable in the 26 examiners who assessed students in both OSCEs and in at least one station common across both OSCEs. The Vj reduced by 40.56% and its contribution to the overall variation in their scores was also decreased by 7.72%. ConclusionThe findings might be suggested that providing examiners with structured feedback could reduce the examiner stringency and leniency variation contributing to their scores in the subsequent OSCE, whilst noting limitations with the quasi-experimental design. More definitive research is required prior to making recommendations for practice.

Is it an impact factor for standardized patients with actor background to perform in the taiwan high-stake objective structured clinical examination?

Background: In the past decade, Tri-Service General Hospital has trained many actors-background standardized patients (SPs). However, we also have recruited some nonactor SPs since 2010. Reviewing the literature, the reliability of SPs with or without actor background would be the impact factor on high-stakes OSCE has not been well studied. Aim: The purpose of this retrospective observation is to clarify whether a SP with or without a professional actor background will affect the performances of the examination when participating in the Taiwan high-stakes objective structured clinical examination (OSCE). The result will be the policy for selecting which kind of SPs to participate in the high-stack OSCE in our hospital. Methods: In this retrospective observation, we analyzed 74 actor background SPs (A-SPs), and 70 no actor background SPs (NA-SPs) who have participated in each spring test of 2015–2017 Taiwan high-stakes OSCE. The data of SPs performance have come from two parts: one is from the examinee with the global rating of the SPs performance and the other is from the examiner with eight-item checklist for the SPs performance. The scoring of both examinee and examiner is a five-point Likert scale. Results: The results show that there is no significant difference in SPs performance from the examinee scoring in the subsequent 3 years, stable and satisfactory, and let examinees feel like real patients. The scores from examiners also showed that two groups of SPs performed equally well without significant differences. Their performance is reliable and consistent, simulating to be a real patient in both groups. However, only one interest finding, when compared to the different age subdivision of SPs, showed the aged subgroup of A-SPs and NA-SPs with the significant difference in the item of reliability and consistent (4.67 ± 0.53 vs. 4.41 ± 0.50, P

Competency-based Standard Setting for a High-stakes Objective Structured Clinical Examination (OSCE): Validity Evidence.

This article was migrated. The article was marked as recommended. Introduction Medical educators need to demonstrate that their trainees meet expected competency levels when progressing through medical education. This study aimed to develop competency-based pass/fail cut-scores for a graduation required Objective Structured Clinical Examination (OSCE), and examine validity evidence for new standards. Methods Six clinicians used the modified Angoff method to determine the cut-scores for an 8-station OSCE. The clinicians estimated the percentage of minimally competent students who would answer each checklist item correctly. Inter-rater reliability, differences in other academic achievements between pass/fail groups, educational impact, and response process were examined. Results One hundred seventy-four rising 4th-year medical students participated in the OSCE. The cut-scores determined for the OSCE resulted in a substantially lower failure rate (5% vs. 29% of the previous year). The inter-rater reliability across domains and cases was .98 (95% CI = .97 - .99). The pass/fail groups significantly differed in six of the eight measures of academic achievements included in the study. Discussion The impact of the standards setting was substantial as it significantly reduced the failure rate and burdens of remediation for both students and faculty. The very high inter-rater reliability indicates that the modified Angoff method produced reliable cut-scores. The significant differences between the pass/fail groups in other measures support external validity of the standards and ensure no false passes. The study also supports response process validity by including discussion among judges and check of previous student performances, as well as recruiting and training multiple clinician educators experienced in medical student teaching. Conclusion Findings of the study provide strong evidence supporting validity of the new cut-scores from a wide spectrum of validity metrics, including response process, internal structure, relations to other variables, and consequences. The study also added to the literature the value of the modified Angoff method in determining competency-based standards for OSCEs.

Incorporating patient partner scores into high stakes assessment: an observational study into opinions and attitudes

BackgroundVolunteer patients (also known as patient partners (PPs)) play a vital role in undergraduate healthcare curricula. They frequently take part in objective structured clinical examinations (OSCE) and rate aspects of students’ performance. However, the inclusion and weighting of PP marks varies, while attitudes and opinions regarding how (and if) they should contribute towards the pass/fail outcome are uncertain.MethodsA prospective observational study was conducted to explore beliefs of PPs regarding inclusion of their scores in a high stakes undergraduate OSCE in a single UK medical school. All PPs delivering components of the local MBChB curriculum were asked to participate in the questionnaire study. Quantitative and qualitative data were analysed using descriptive statistics and framework analysis respectively.ResultsFifty out of 160 (31% response rate) PPs completed the questionnaire; 70% had participated in a final year OSCE. Thirty (60%) felt their marks should be incorporated into a student’s overall score, while 28% were uncertain. The main reasons for inclusion were recognition of the patient perspective (31%) and their ability to assess attitudes and professionalism (27%), while reasons against inclusion included lack of PP qualification/training (18%) and concerns relating to consistency (14%). The majority of PPs were uncertain what proportion of the total mark they should contribute, although many felt that 5-10% of the total score was reasonable. Most respondents (70%) felt that globally low PP scores should not result in an automatic fail and many (62%) acknowledged that prior to mark inclusion, further training was required.ConclusionThese data show that most respondents considered it reasonable to “formalise their expertise” by contributing marks in the overall assessment of students in a high stakes OSCE, although what proportion they believe this should represent was variable. Some expressed concerns that using marks towards progress decisions may alter PP response patterns. It would therefore seem reasonable to compare outcomes (i.e. pass/fail status) using historical data both incorporating and not incorporating PP marks to evaluate the effects of doing so. Further attention to existing PP training programmes is also required in order to provide clear instruction on how to globally rate students to ensure validity and consistency.

An Objective Structured Clinical Exam to Assess Semiology Skills of Medical Students.

Mastery of history taking and physical exam skills is a key competence of medical students. Objective Structured ClinicalExaminations are the gold standard to assess these competencies, but their implementation in Portugal is poorly documented. Wedescribe the implementation and our seven years experience with a high-stakes Objective Structured Clinical Examination to assessthese skills in the School of Medicine, University of Minho. Our Objective Structured Clinical Examination is in place since 2010 and has been subject to continuousimprovements, including the adoption of a standard setting procedure and an increase in the number of stations. Grades in our exam are well distributed and discriminate among students. History taking grades are lower and have remainedstable throughout the years while physical examination scores have risen. The exam is reliable, with internal consistency above 0.45and a G-coefficient of 0.74. It is also feasible, with a total testing time of approximately 20 hours for 140 students, and the involvement of18 standardized patients and 18 faculty assessors. More importantly, it was able to engage the students, who recognize its importance. The most important validity criterion of our, and any Objective Structured Clinical Examination, would be predictive validity,the ability to predict the performance of students in the clinical context. Our approach to a high-stakes Objective Structured Clinical Examination shows that it is feasible, reliable, valid and fairand can be implemented with success in the Portuguese setting.

Standard Setting Methods for Pass/Fail Decisions on High-Stakes Objective Structured Clinical Examinations: A Validity Study

Construct: Authentic standard setting methods will demonstrate high convergent validity evidence of their outcomes, that is, cutoff scores and pass/fail decisions, with most other methods when compared with each other. Background: The objective structured clinical examination (OSCE) was established for valid, reliable, and objective assessment of clinical skills in health professions education. Various standard setting methods have been proposed to identify objective, reliable, and valid cutoff scores on OSCEs. These methods may identify different cutoff scores for the same examinations. Identification of valid and reliable cutoff scores for OSCEs remains an important issue and a challenge. Approach: Thirty OSCE stations administered at least twice in the years 2010–2012 to 393 medical students in Years 2 and 3 at Aga Khan University are included. Psychometric properties of the scores are determined. Cutoff scores and pass/fail decisions of Wijnen, Cohen, Mean–1.5SD, Mean–1SD, Angoff, borderline group and borderline regression (BL-R) methods are compared with each other and with three variants of cluster analysis using repeated measures analysis of variance and Cohen's kappa. Results: The mean psychometric indices on the 30 OSCE stations are reliability coefficient = 0.76 (SD = 0.12); standard error of measurement = 5.66 (SD = 1.38); coefficient of determination = 0.47 (SD = 0.19), and intergrade discrimination = 7.19 (SD = 1.89). BL-R and Wijnen methods show the highest convergent validity evidence among other methods on the defined criteria. Angoff and Mean–1.5SD demonstrated least convergent validity evidence. The three cluster variants showed substantial convergent validity with borderline methods. Conclusions: Although there was a high level of convergent validity of Wijnen method, it lacks the theoretical strength to be used for competency-based assessments. The BL-R method is found to show the highest convergent validity evidences for OSCEs with other standard setting methods used in the present study. We also found that cluster analysis using mean method can be used for quality assurance of borderline methods. These findings should be further confirmed by studies in other settings.

Clinical perceptions of radiation therapy undergraduate competency standards.

IntroductionThe multifactorial nature of clinical skills development makes assessment of undergraduate radiation therapist competence level by clinical mentors challenging. A recent overhaul of the clinical assessment strategy at Queensland University of Technology has moved away from the high‐stakes Objective Structured Clinical Examination (OSCE) to encompass a more continuous measure of competence. This quantitative study aimed to gather stakeholder evidence to inform development of standards by which to measure student competence for a range of levels of progression.MethodsA simple anonymous questionnaire was distributed to all Queensland radiation therapists. The tool asked respondents to assign different levels of competency with a range of clinical tasks to different levels of student. All data were anonymous and was combined for analysis using Microsoft Excel.ResultsFeedback indicated good agreement with tasks that specified the amount of direction required and this has been incorporated into the new clinical achievements record that the students need to have signed off. Additional puzzling findings suggested higher expectations with planning tasks than with treatment‐based tasks.ConclusionThe findings suggest that the amount of direction required by students is a valid indicator of their level and has been adopted into the clinical assessment scheme. Further work will build on this to further define standards of competency for undergraduates.

Going “high stakes” with a pharmacy OSCE: Lessons learned in the transition

Objectives(1) To describe the experiences of the Auburn University Harrison School of Pharmacy (AUHSOP) in implementing a high-stakes Objective Structured Clinical Examination (OSCE) into its curriculum, (2) to examine the extent to which student performance on OSCEs used in the skills laboratory course correlates with performance on the high-stakes Milestone OSCE, and (3) to examine differences in scores between Milestone OSCEs used for formative feedback versus high-stakes consequences. MethodsTwo consecutive years of third professional year pharmacy student (P3) scores on the skills laboratory OSCE and the Milestone OSCE were examined. Correlations and t-tests were calculated between scores using a matched-pairs design. Pass rates were compared between formative and high-stakes Milestone OSCE scores. ResultsStatistically significant although low positive correlations were found between the analytical checklist scores on the two OSCEs. The passing rate on the Milestone increased significantly when the examination became used in a high-stakes manner. ConclusionsResults indicate that, although there is room for improvement, the high-stakes Milestone OSCE is an appropriate assessment for the curriculum, complementing the OSCEs embedded in laboratory courses. A description of lessons learned is offered to assist other schools implementing OSCEs in their programs.

Role plays to build counselling competencies

Context and setting Counselling is an important but neglected aspect of comprehensive patient care and good interpersonal communication skills are required as a pre-requisite. At Fatima Memorial Hospital College of Medicine and Dentistry (FMH), in Lahore, Pakistan, the goal is to produce primary health care doctors who are also culturally sensitive counsellors. Why the idea was necessary Effective counselling incorporates awareness of preventive, socioeconomic, culturally sensitive issues and requires appropriate interpersonal communication and empathy from the doctor, but this skill is currently not taught at FMH. Faculty members recognised the importance of training medical students to develop competency in counselling. What was done Role-play was selected as the method to introduce the concepts of counselling and to reinforce appropriate counselling behaviours because it demonstrates the competencies involved. In 2006, 4 role plays were written and videotaped using instructors to play the roles. These were for use with Year 4 medical students. The role-plays demonstrated competencies associated with counselling mothers about breast-feeding infants, antenatal care, diarrhoea control and vaccination in children. Checklists were developed to correspond to the competencies demonstrated in role plays. Instructors in the Department of Community Medicine were trained in the skills necessary to facilitate role play and the assessment methods to use to determine if competencies had been achieved. A pilot exercise was conducted with a randomly selected group of students who had completed the community medicine course to determine if the role-play exercise was feasible. The test run demonstrated that the project was appropriate and feasible and the new Year 4 students completed a pre-course objective structured clinical examination (OSCE) in which they were evaluated on the identified competencies by the instructors who had played the roles in the videotapes. Students also enacted role-plays of the patient–doctor relationship by switching between the roles of patient and doctor. Feedback on student performances was provided by the instructors. A randomly selected group of students were observed in patient–student interactions in community clinics and rated on identified competencies by observers using the structured checklists (direct observation through structured checklist [DOS]). Finally, students participated in a post-course, high-stakes OSCE 6 months after the course, which was evaluated by the instructors who had assessed their skills in the pre-course OSCE in order to ensure test-retest reliability. Evaluation of results and impact A total of 54 students participated in the role-play exercises. The class overall and 83% of students individually showed statistically significant improvement between their mean pre- and post-course evaluations. There was no gender difference in results for 2 culturally sensitive competencies (antenatal care and breast-feeding), but this may be attributed to the small number of students. Of the students observed in real patient situations, 75% showed a statistically significant difference between performance in the clinic and performance on the post-course OSCE, with the real-life performance being poorer. However, DOS scores were higher for the post-course than the pre-course performance. Role-play was effective in improving competencies in counselling and will be included from the early stages in the medical school curriculum to allow the students to internalise the skills. Future plans include incorporating counselling OSCE stations into the final year curriculum. The programme will be presented to the curriculum committee for incorporation into the community medicine curriculum and in university-affiliated medical schools.

Applying Generalizability Theory to High-Stakes Objective Structured Clinical Examinations in a Naturalistic Environment

Purpose The purpose of this project was to determine if generalizability theory could be successfully applied to a high-stakes licensure objective structured clinical examination as part of its normal administrative procedures and whether the analysis could yield useful information with regard to sources of variance. Methods The anonymized data received from the Canadian Chiropractic Examining Board for its June 2005 Clinical Skills Examination were analyzed with generalizability theory. Variance components were estimated with SPSS 11.5 (SPSS Inc, Chicago, Ill) as partially nested data. The data included 182 candidates, 43 raters, 40 standardized patient actors, and 18 individual cases. Results Internal consistency estimates (Cronbach α) were .86 for day 1 and .91 for day 2. The α estimates for stations averaged .68 for day 1 and .74 for day 2. The generalizability-coefficient for the day 1 exam was .65 and for the day 2 was .42. G-coefficients for stations averaged .63 for day 1 and .74 for day 2. On day 1, the raters contributed 7% of the variance, and on day 2, the raters contributed 8%. Conclusions Generalizability theory can contribute to the understanding of sources of variance and provide direction for the improvement of individual stations. The size of the rater variance in a station may also indicate the need for increased training in that station or the need to make the scoring checklist more clear and definitive. Generalizability theory, however, must be cautiously applied, and it requires careful selection of the floating raters and vigorous training of the raters in each station.