Observed Structured Clinical Exam Research Articles

Development of a novel observed structured clinical exam to assess clinical ultrasound proficiency in undergraduate medical education

ObjectivesA pilot study was performed to develop and test an observed structured clinical exam (OSCE) for clinical ultrasound in second-year medical students. The goal was to assess a longitudinal clinical ultrasound curriculum for medical students and to help determine readiness to perform ultrasound during clinical clerkships.MethodsThe OSCE contained 40 tasks over 30 min in a one-to-one examiner to examinee environment using standardized patients covering cardiac, pulmonary, and inferior vena cava (IVC) ultrasound exams along with 6 critical diagnoses. Examinees were assessed using a binary checklist approach. A two-way ANOVA analysis was performed to determine if there were differences between the day and session the OSCE was administered. Results are presented as mean ± standard deviation.ResultsOne hundred fifty-two students were tested with an overall mean score of 64.9 ± 17.6%. Scores between the cardiac, IVC, and lung sections varied—67.8% ± 18.8%, 62.4% ± 26.2%, and 57.1% ± 20.6%, respectively. One hundred twenty-six (82.9%) answered at least one critical diagnosis incorrectly. Students in the late session performed better than the early session (1: 60% vs 2: 69%, p = .001).ConclusionsStudents performed better in later sessions. Additionally, the number of questions left blank at the end of the exam suggests that the length of the OSCE should be evaluated. Incorporating critical diagnoses was challenging for examinees. The proposed OSCE is a valuable assessment tool that could be adapted to assess student’s readiness to use clinical ultrasound prior to clerkships.

Does surgical observed structured clinical exam (OSCE) predict clerkship grade, shelf exam scores, and preceptor clinical evaluation?

BackgroundClinical evaluation of medical student performance has been criticized as variable and subjective. The aim of this study was to assess the correlation of a summative surgical OSCE exam to clinical faculty evaluations as well as surgery shelf exam score and final grades. MethodsThe performance of 392 students who completed the surgical clerkship between 2017 and 2019 was assessed via Pearson Coefficients comparing OSCE grades, clinical evaluations of Medical Knowledge and Patient Care, Communication and Professionalism, the National Board of Medical Examiners (NBME) shelf surgical subject exam, and final clerkship grade. ResultsResults demonstrate a statistically significant positive relationship between the OSCE, Shelf score and grade, final clerkship grade, and all clinical evaluations except Communication skills. The greatest correlation occurred between OSCE and shelf scores and grades. Although significant, the degree of correlation with clinical observation was significantly less. ConclusionThis study demonstrates that a surgical OSCE has a small positive correlation with clinical knowledge as measured by the NBME shelf exam. There is also an equal correlation with medical knowledge standards, with the OSCE better predicting NBME shelf outcome. This lower correlation to clinical assessment suggests that either the clinical grades contain elements not detected on an OSCE exam but could also support the hypothesis that variability in clinical grades do contain a significant degree of subjectivity.

Efficacy of an ultrasound training program for nurse midwives to assess high-risk conditions at labor triage in rural Uganda.

Many high-risk conditions of pregnancy are undetected until the time of delivery in low-income countries. We developed a point-of-care ultrasound training protocol for providers in rural Uganda to detect fetal distress or demise, malpresentation, multiple gestation, placenta previa, oligohydramnios and preterm delivery. This was a mixed-methods study to evaluate the 2-week training curriculum and trainees' ability to perform a standard scanning protocol and interpret ultrasound images. Surveys to assess provider confidence were administered pre-training, immediately after, and at 3-month follow up. Following lecture and practical demonstrations, each trainee conducted 25 proctored scans and were required to pass an observed structured clinical exam (OSCE). All images produced 8 weeks post course underwent blinded review by two ultrasound experts to assess image quality and to identify common errors. Key informant interviews further assessed perceptions of the training program and utility of point-of-care ultrasound. All interviews were audio recorded, transcribed, and reviewed by multiple readers using a content analysis approach. Twenty-three nurse/nurse midwives and two physicians from one district hospital and three health centers participated in the training curriculum. Confidence levels increased from an average of 1 point pre-course to over 6 points post-course for all measures (maximum of 7 points). Of 25 participants, 22 passed the OSCE on the first attempt (average score 89.4%). Image quality improved over time; the final error rate at week 8 was less than 5%, with an overall kappa of 0.8-1 for all measures between the two reviewers. Among the 12 key informant interviews conducted, key themes included a desire for more hands-on training and longer duration of training and challenges in balancing clinical duties with ability to attend training sessions. This study demonstrates that providers without previous ultrasound experience can detect high-risk conditions during labor with a high rate of quality and accuracy after training.

IMPROVING GERIATRIC CARE OF INTERPROFESSIONAL PRACTICING PROVIDERS

Practicing providers often struggle with the care of older adults due to knowledge, skill, and attitude barriers. In an attempt to improve employee engagement in the care of older adults, the Geri-EMPOWER (Empowering Medical Providers and Older adults With strategies to Escape Readmission) program was initiated. In this program, case managers and visiting VA rural geriatric scholars participated in a two days of intensive educational sessions including lectures, shadowing inpatient teams and clinic providers, a dementia simulation learning exercise, and an Observed Structured Clinical Exam (OSCE) with standardized patient encounters. The 15 initial participating trainees came from a variety of medical backgrounds including physicians, nurse practitioners, nurses, social workers, and psychologists. A 10 item knowledge based pretest and posttest was constructed using the learning objectives of the course. Skills of attendees were directly observed during 4 OSCE stations. Attitudes towards older adults were measured before and after the intervention using the Caroline Opinions on Care of Older Adults (COCOA) scale. Geriatric and palliative care knowledge improved with average knowledge test scores improving from 63% to 86% before and after the course. Participants obtained all minimum competencies during their OSCE exam, and rated this session very highly in their course feedback. Attitudes towards older adults were also found to improve with an average COCOA score increase of 9 points before and after the educational sessions. This innovative course based in adult-learning theory demonstrates that employed interprofessional providers can quickly improve knowledge, skills, and attitudes towards older adults.

Assessing a novel point-of-care ultrasound training program for rural healthcare providers in Kenya

BackgroundA novel point-of-care ultrasound (PoCUS) training program was developed to train rural healthcare providers in Kenya on the Focused Assessment with Sonography for Trauma (FAST), thoracic ultrasound, basic echocardiography, and focused obstetric ultrasonography. The program includes a multimedia manual, pre-course testing, 1-day hands-on training, post-testing, 3-month post-course evaluation, and scheduled refresher training. This study evaluates the impact of the course on PoCUS knowledge and skills. Competency results were compared based on number of previous training/refresher sessions and time elapsed since prior training.MethodsTrainees were evaluated using a computer-based, 30 question, multiple-choice test, a standardized observed structured clinical exam (OSCE), and a survey on their ultrasound use over the previous 3 months.ResultsThirty-three trainees were evaluated at 21 different facilities. All trainees completed the written exam, and 32 completed the OSCE. Nine trainees out of 33 (27.3%) passed the written test. Trainees with two or more prior training sessions had statistically significant increases in their written test scores, while those with only one prior training session maintained their test scores. Time elapsed since last training was not associated with statistically significant differences in mean written test scores. Mean image quality scores (95% confidence interval) were 2.65 (2.37–2.93) for FAST, 2.41 (2.03–2.78) for thoracic, 2.22 (1.89–2.55) for cardiac, and 2.95 (2.67–3.24) for obstetric exams. There was a trend towards increased mean image quality scores with increases in the number of prior training sessions, and a trend towards decreased image quality with increased time elapsed since previous training. Forty percent of trainees reported performing more than 20 scans in the previous 3 months, while 22% reported less than 10 scans in the previous 3 months. Second and third trimester focused obstetric ultrasound was the most frequently performed scan type. Frequency of scanning was positively correlated with written test scores and image quality scores.ConclusionThis novel training program has the potential to improve PoCUS knowledge and skills amongst rural healthcare providers in Kenya. There is an ongoing need to increase refresher/re-training opportunities and to enhance frequency of scanning in order to improve PoCUS competency.

The PULSES project: Teaching the vital elements of code status discussions to oncology residents.

46 Background: Discussions with cancer patients around cardiopulmonary resuscitation, or ‘code status,’ are often led by trainees in oncology, but formal education for this competency is lacking. In this study, we developed and tested a novel communication tool, the PULSES framework, for informed code status decision-making (a six-step approach summarized by the PULSES acronym [see Table]), through an educational workshop. Methods: A multicentre randomized controlled trial was carried out at 3 academic cancer centres in Ontario, Canada. Residents in medical oncology (MO) and radiation oncology (RO) programs completed a workshop and an observed structured clinical exam (OSCE). Participants were randomized to complete the training before the OSCE (experimental arm) or after the OSCE (control arm). Randomization was stratified for centre and oncology discipline. Expert raters evaluated communication with two rating tools: the novel PULSES scale and the communication skills assessment form (CSAF), a validated benchmark tool that is not specific to oncology content. The primary outcome was improvement in PULSES scores. Results: Forty-six residents consented to participate (28 RO and 18 MO). Groups were well balanced for program and year of training. Participants in the experimental group had higher mean PULSES score than those in the control group (80.4±13.5 vs 63.4±9.7; p<.001; maximum score = 108). There was no significant effect for oncology program and no significant interaction between program and training condition. Scores from the PULSES and CSAF scales were highly correlated (R = 0.864). Conclusions: The PULSES training improved performance among oncology residents for code status discussions. Improved communication scores were not scale-specific. The PULSES framework offers a standardized approach and can be incorporated into competency-based curricula for postgraduate oncology programs. Future work will explore whether communication training in this area impacts patient-level outcomes. [Table: see text]

The PULSES project: Teaching the vital elements of code status discussions to oncology residents.

10024 Background: Discussions with cancer patients around cardiopulmonary resuscitation (CPR), or ‘code status,’ are often led by trainees in oncology, but formal education for this competency is lacking. In this study, we developed and tested a novel communication tool, the PULSES framework, for informed code status decision-making (a six-step approach summarized by the PULSES acronym [Table 1]), through an educational workshop. Methods: A multicentre randomized controlled trial was carried out at 3 academic cancer centres in Ontario, Canada. Residents in medical oncology (MO) and radiation oncology (RO) programs completed a workshop and an observed structured clinical exam (OSCE). Participants were randomized to complete the training before the OSCE (experimental arm) or after the OSCE (control arm). Randomization was stratified for centre and oncology discipline. Expert raters evaluated communication with two rating tools: the novel PULSES scale and the communication skills assessment form (CSAF), a validated benchmark tool that is not specific to oncology content. The primary outcome was improvement in PULSES scores. Results: Forty-six residents consented to participate (28 RO and 18 MO). Groups were well balanced for program and year of training. Participants in the experimental group had higher mean PULSES score than those in the control group (80.4±13.5 vs 63.4±9.7; p < .001; maximum score = 108). There was no significant effect for program and no significant interaction between program and training condition. Scores for the PULSES and CSAF scales were highly correlated (R = 0.864). Conclusions: The PULSES training improved performance among oncology residents for code status discussions. Improved communication scores were not scale-specific. The PULSES framework offers a standardized approach and can be incorporated into competency-based curricula for postgraduate oncology programs. Future work will explore whether communication training in this area impacts patient-level outcomes. [Table: see text]

Evaluation of Qatari and Canadian Pharmacy Students Learning Together about Cultural Competency using Videoconference Educational Activities

Background: Pharmacists are under pressure to provide patient centered care within increasingly culturally diverse settings. Pharmacy schools play an important role in educating learners regarding culture and its impact on patient care.
 Objectives: The objectives of this study were to determine if a novel cultural competency learning activity, which involved students from two culturally and ethnically different pharmacy schools learning together using videoconference education activities, improved: (1) student knowledge and confidence pertaining to cultural competency concepts, (2) attitudes and perceptions towards being a culturally competent pharmacist, and (3) academic performance related to cultural competency case studies.
 Methods: Pharmacy students from Qatar University in Doha, Qatar (n=25) and the University of Saskatchewan in Saskatoon, Canada (n=85) participated in a cultural competency activity comprised of small group work on a patient case study, followed by tutorial discussions. Some Canadian students (n=31/85) worked collaboratively (via video conference) with the students from Qatar. The evaluation used a convergent mixed methods design comprised of: (1) a pre and post session survey measuring student knowledge and confidence; (2) pre and post session student self-reflections; and, (3) student academic performance on care plans and an observed structured clinical exam (OSCE).
 Results: The survey identified small but statistically significant (p<0.05) improvements in knowledge and confidence with respect to 11 of the 12 questionnaire items in the students from Canada and 2 of the 12 items in the students from Qatar. The self-reflections found that 44.4% (n=36/81) of students who completed the pre and post reflective questions reported a change in knowledge and attitudes regarding cultural competency, but a reason for the change was not evident. Student grades on the cultural competency care plans and the OSCE were not different between the students who worked collaboratively across the two schools compared with those who did not.
 Conclusion: The cultural competency student activity showed small improvements in pharmacy student knowledge, attitudes and confidence, but did not affect academic performance. Future research in this area could utilize a greater diversity of methodological approaches, including a focus on student self-reflection and qualitative assessment, to better capture student competency.
 
 Type: Original Research

Faculty Communication Knowledge, Attitudes, and Skills Around Chronic Non-Malignant Pain Improve with Online Training.

Many physicians struggle to communicate with patients with chronic, non-malignant pain (CNMP). Through the use of a Web module, the authors aimed to improve faculty participants' communication skills knowledge and confidence, use of skills in clinical practice, and actual communication skills. The module was implemented for faculty development among clinician-educators with university faculty appointments, outpatient clinical practices, and teaching roles. Participants completed the Collaborative Opioid Prescribing Education Risk Evaluation and Mitigation Strategy (COPE-REMS®) module, a free Web module designed to improve provider communication around opioid prescribing. Main study outcomes were improvements in CNMP communication knowledge, attitudes, and skills. Skills were assessed by comparing a subset of participants' Observed Structured Clinical Exam (OSCE) performance before and after the curriculum. Sixty-two percent of eligible participants completed the curriculum in 2013. Knowledge-based test scores improved with curriculum completion (75% vs. 90%; P < 0.001). Using a 5-point Likert-type scale, participants reported improved comfort in managing patients with CNMP both immediately post-curriculum and at 6 months (3.6 pre vs. 4.0 post vs. 4.1 at 6 months; P = 0.02), as well as improvements in prescribing opioids (3.3 vs. 3.8 vs. 3.9, P = 0.01) and conducting conversations about discontinuing opioids (2.8 vs. 3.5 vs. 3.9, P < 0.001). Additionally, CNMP-specific communication skills on the OSCE improved after the curriculum (mean 67% vs. 79%, P = 0.03). Experienced clinician-educators improved their communication knowledge, attitudes, and skills in managing patients with CNMP after implementation of this curriculum. The improvements in attitudes were sustained at six months. A Web-based curriculum such as COPE-REMS® may be useful for other programs seeking improvement in faculty communication with patients who have CNMP.

Beyond standard checklist assessment: Question sequence may impact student performance.

IntroductionClinical encounters are often assessed using a checklist. However, without direct faculty observation, the timing and sequence of questions are not captured. We theorized that the sequence of questions can be captured and measured using coherence scores that may distinguish between low and high performing candidates.MethodsA logical sequence of key features was determined using the standard case checklist for an observed structured clinical exam (OSCE). An independent clinician educator reviewed each encounter to provide a global rating. Coherence scores were calculated based on question sequence. These scores were compared with global ratings and checklist scores.ResultsCoherence scores were positively correlated to checklist scores and to global ratings, and these correlations increased as global ratings improved. Coherence scores explained more of the variance in student performance as global ratings improved.DiscussionLogically structured question sequences may indicate a higher performing student, and this information is often lost when using only overall checklist scores.ConclusionsThe sequence test takers ask questions can be accurately recorded, and is correlated to checklist scores and to global ratings. The sequence of questions during a clinical encounter is not captured by traditional checklist scoring, and may represent an important dimension of performance.

Musculoskeletal ultrasound in internal medicine residency – a feasibility study

IntroductionUltrasound has become indispensable in medicine for diagnosis and therapeutics. In Internal Medicine Residency Training Program (IMRTP), there is a deficiency of a structured, competency-based musculoskeletal ultrasound (MSKUS) training despite its growing popularity.MethodsWe conducted a feasibility study for incorporating MSKUS in IMRTP and competency assessment. We recruited internal medicine residents from all training levels. Rheumatology attending with expertise in MSKUS supervised curriculum-design and patient-based assessment. A structured curriculum was developed for knee MSKUS, including training material and a pre- and post-test. An Observed Structured Clinical Exam (OSCE) was used for competency assessment. Attending evaluations and participant feedbacks were collected.ResultsFifteen residents participated and 12 completed the OSCE. The pre–post tests showed a statistically significant increase (p<0.001) for all subcomponents (anatomy, pathology, and biophysics). The residents scored high on the OSCE [8.8±1.1 (range 7–10)]. Attending evaluations for cognitive and technical skills were rated ‘Excellent’ (66%) and ‘Good’ (87%), respectively. The training program was perceived very valuable with regard to time and resources spent by residents and faculty, although some felt a need for more patient assessments.DiscussionThe competency assessed with our OSCE and highly positive feedback reflects this preliminary study's importance and sets the platform for future studies of formal ultrasound training in internal medicine.

Teaching the Competencies: Using Observed Structured Clinical Examinations for Faculty Development

Gastroenterology (GI) training programs must develop the teaching skills of their faculty and provide feedback to their fellows. Many faculty feel uncomfortable offering feedback or identifying specific areas for improvement to the fellows. We developed an Observed Structured Clinical Exam (OSCE) to assess fellows' skills and provided faculty with specific criteria to rate the fellows' performance. We propose that OSCEs can serve as tools for faculty development in delivering effective feedback. Faculty completed a Web-based training module and received written guidelines on giving feedback. Four OSCE stations were completed by each fellow with faculty using standardized checklists to assess the fellows' skills. Afterwards, faculty rated each program component and assessed their comfort level with feedback. Eight faculty members and 10 fellows from 5 GI training programs in NYC participated. 100% of the faculty agreed that feedback is an important learning tool, should include the learner's self-assessment, and that feedback skills could improve with practice. Compared to faculty skills prior to the program, 87.5% of the faculty agreed that they focused more on specific behaviors and 75% agreed that giving negative feedback was now easier. OSCEs can serve as practicums for faculty development in giving constructive feedback.

Call for advice on observed, structured clinical exam

I am undertaking a specialist skills module in neonatal nursing and will be examined by the university in a two-hour written exam and a two-hour observed, structured clinical exam (OSCE).

2. Use of a health advocacy essay to improve competence

We set out to determine if essay writing on health advocacy could improve scores on a Health Advocacy observed structured clinical exam (OSCE) station.&#x0D; A Health Advocacy station was used as one of ten stations at the annual resident OSCE in 2006. Subsequently, residents were instructed to write an essay regarding their actions as a health advocate, either in general or relating to a specific situation. For the 2007 OSCE, a different Health Advocacy station was used that was similar to the previous year in terms of the issues that residents needed to address and the scoring structure. The results on the OSCE stations between the two years were compared using Student’s T-test. Regression analysis was used to identify any predictors for a higher score.&#x0D; The number of residents taking the OSCE in 2006 and 2007 were 17 and 13, respectively. Nine residents completed the OSCE in both years. The number of PGY-1’s taking the OSCE for the first time in 2006 and 2007 were 8 and 4, respectively. Compliance with the essay writing was 100%. The mean score on the Health Advocacy station was significantly higher in 2007 compared to 2006 (53.0 ± 14.6 compared to 65.6 ± 10.0 vs. 53.0 ± 14.6; p=0.01). The mean score of PGY-1’s in 2007 tended to be higher than PGY-1’s in 2006 (67.5 ± 11.4 vs. 54.5 ± 12.3; p=0.11), suggesting the improvement was not due to having taken the OSCE previously. Regression analysis did not identify and other predictors of higher score on the OSCE station.&#x0D; These results suggest that essay writing by residents about actions they have taken as health advocates improves knowledge about the CanMEDS competency of Health Advocacy, as measured in an OSCE station. The non-randomized design of the study does not exclude the possibility of other factors influencing the improvement in score.&#x0D; Oandasan I, Barker K. Educating for Advocacy: Exploring the Source and Substance of Community-Responsive Physicians. Acad Med 2003; 78(1):S16-S19.&#x0D; Frank J, Langer B. Collaboration, Communication, Management, and Advocacy: Teaching surgeons new skills through the CanMEDS project. World J Surg 2003; 27:972-978.&#x0D; Verma S, Flynn L, Seguin R. Faculty’s and residents’ perceptions of teaching and evaluating the role of health advocate: A study at one Canadian university. Acad Med 2005; 80:103-108.

Development of clinical reasoning exercises in a problem-based curriculum

The paper describes an approach to assessing clinical reasoning through the use of multiple short cases in an oral or written examination format. Two studies were performed in consecutive medical school curricular units. The first evaluation exercise tested understanding of pathophysiological mechanisms using both oral and written cases. This study demonstrated acceptable test reliability and a high correlation between written and oral formats. A second study emphasised clinical diagnostic reasoning and examined generalizability across cases, body-systems and occasions (test-retest). This study used the oral format alone, with the same students but a different set of tutors from those participating in the first study. In the second study, a moderate correlation with performance on a concurrent multiple choice test and an Observed Structured Clinical Exam (OSCE) was observed.