Objective Structured Clinical Examination Checklist Research Articles

Development of an Objective Structured Clinical Examination Checklist and a Post-Education Questionnaire for Musculoskeletal Ultrasound Training Focusing on Volar Wrist and Carpal Tunnel Syndrome

Background: The objective structured clinical examination (OSCE) is used in the colleges of Korean Medicine, but few studies have validated the OSCE evaluation criteria or post-education questionnaires. Diagnostic ultrasound is used in Korean medicine treatment including acupuncture, acupotomy, and pharmacopuncture to increase the safety and efficacy of treatment. We aimed to develop and validate a OSCE checklist and questionnaire for diagnostic musculoskeletal ultrasound training.Methods: A OSCE checklist and rubric for diagnostic ultrasound training, and questionnaire was developed using literature research. Eight expert panelists verified each draft item in a single-round survey. Items with a content validity ratio (CVR) < 0.75 were excluded or modified to reflect the experts’ opinions.Results: The OSCE checklist and rubric for diagnostic ultrasound training focusing on volar wrist and carpal tunnel syndrome included: 15 items revised according to CVR and expert opinions, the pre-examination procedure, structures to be identified by ultrasound, scans with 2 diagnostic criteria for carpal tunnel syndrome, an explanation of the exam results, and the post exam procedure. The questionnaire consisted of 15 items, including the overall evaluation of training, the effect of the OSCE, and the perception of the ultrasound. All 6 self-evaluation items were not revised, as they had a CVR of ≥ 0.75.Conclusion: An ultrasound OSCE for scanning the volar wrist and diagnosing carpal tunnel syndrome was developed using 15 validated tasks, 15 survey questions about ultrasound training, and 6 questions for selfevaluation. These results may be used in the future for education in diagnostic ultrasound.

The Effectiveness of Self-Instructional Video vs. Classroom Teaching Method on Focused Assessment With Sonography in Trauma Among House Officers in University Hospital.

Background: The aim of this study was to compare the effectiveness of self-instructional-video (SIV) and classroom training method (CTM) in learning Focus-Assessment with Sonography-in-Trauma (FAST) among house officers (HO).Method: A randomized controlled study involving house officers working in the university hospital in Malaysia was conducted where participants were randomized into SIV group (intervention) and CTM group (control). Each group had to undergo a 4 h hands-on training. The intervention group has undergone self-training using the video material without any facilitation while the control group received lecture and hands-on training with facilitators. Participants' performance was assessed using a validated Objective Structured Clinical Examination checklist for landmark identification and interpretation of images generated. Learning preference and confidence level were also assessed.Result: A total of 33 HO were enrolled in this study. Marks obtained in image acquisition by the intervention and control were 25.3 (SD = 5.3) and 25.6 (SD = 2.3) p > 0.05, respectively. While in image interpretation, the mean score for the intervention and control group was 10.3 (SD 1.7) and 9.8 (SD = 1.7) p > 0.05, respectively. Overall performance assessment, showed the intervention group obtained 35.6 (SD = 5.9) compared to control 35.3 (SD = 3.4), p > 0.05. Based on pre-specified determinant these scores difference falls within the 10% of non-inferiority margin. The absolute difference between both groups was 0.3 (CI = −3.75 to 3.21, p = 0.871), which proves non-inferiority but not superiority. In terms of learning preference and confidence to perform FAST, most of the participants preferred the control group approach.Conclusion: The SIV method is as effective as the CTM for learning FAST among the house officers and served as an alternative to classroom teaching. However, this technique needs improvement in promoting their confidence and preference. Perhaps incorporating a feedback session after going through the SIV would improve the confidence.

Use of a fundus photograph matching program in imparting proficiency in ophthalmoscopy

ObjectiveTo determine whether practice using an online fundus photograph program results in a long-term increase in proficiency with direct ophthalmoscopy skills in medical students. DesignThis study was a prospective medical education trial. Students were enrolled to participate in a voluntary Objective Structured Clinical Examination (OSCE) using 5 patients with ocular findings. Students who matched a minimum of 6 discs 16 months before the study were assigned to the intervention group and were compared with students who did not participate in the exercise. ParticipantsForty-six second-year medical students at Queen’s University: 15 in the intervention group and 31 in the control group. MethodsStudents were evaluated using the Queen’s University Ophthalmoscopy OSCE Checklist (QUOOC). Students were asked to calculate the cup-to-disc ratio, comment on disc margins, and whether there was any macular pathology. Students participated in a summative OSCE as part of the curriculum in which all students attempted to match fundus photographs. ResultsStudents in the intervention group performed significantly better on the QUOOC, with a mean score of 78.3% (± 4.2), compared with the control, who had a mean score of 69.4% (± 4.2) (p = 0.005). The intervention group was significantly more accurate at matching optic nerve photographs, with 100% (15/15) of the students correctly identifying the correct optic nerve on first attempt compared with 53.3% (16/30) in the control group (p = 0.0014). ConclusionsThe use of an online peer fundus photograph program leads to a long-term increase in examination technique, proficiency in ophthalmoscopy, and accuracy at matching optic nerve photographs.

Done or Almost Done? Improving OSCE Checklists to Better Capture Performance in Progress Tests

ABSTRACTConstruct: The impact of using nonbinary checklists for scoring residents from different levels of training participating in objective structured clinical examination (OSCE) progress tests was explored. Background: OSCE progress tests typically employ similar rating instruments as traditional OSCEs. However, progress tests differ from other assessment modalities because learners from different stages of training participate in the same examination, which can pose challenges when deciding how to assign scores. In an attempt to better capture performance, nonbinary checklists were introduced in two OSCE progress tests. The purposes of this study were (a) to identify differences in the use of checklist options (e.g., done satisfactorily, attempted, or not done) by task type, (b) to analyze the impact of different scoring methods using nonbinary checklists for two OSCE progress tests (nonprocedural and procedural) for Internal Medicine residents, and (c) to determine which scoring method is better suited for a given task. Approach: A retrospective analysis examined differences in scores (n = 119) for two OSCE progress tests (procedural and nonprocedural). Scoring methods (hawk, dove, and hybrid) varied in stringency in how they awarded marks for nonbinary checklist items that were rated as done satisfactorily, attempted, or not done. Difficulty, reliability (internal consistency), item-total correlations and pass rates were compared for each OSCE using the three scoring methods. Results: Mean OSCE scores were highest using the dove method and lowest using the hawk method. The hawk method resulted in higher item-total correlations for most stations, but there were differences by task type. Overall score reliability calculated using the three methods did not differ significantly. Pass–fail status differed as a function of scoring methods and exam type, with the hawk and hybrid methods resulting in higher failure rates for the nonprocedural OSCE and the dove method resulting in a higher failure rate for the procedural OSCE. Conclusion: The use of different scoring methods for nonbinary OSCE checklists resulted in differences in mean scores and pass–fail status. The results varied with procedural and nonprocedural OSCEs.

Calibration of communication skills items in OSCE checklists according to the MAAS-Global

BackgroundCommunication skills (CS) are commonly assessed using ’communication items’ in Objective Structured Clinical Examination (OSCE) station checklists. Our aim is to calibrate the communication component of OSCE station checklists according to the MAAS-Global which is a valid and reliable standard to assess CS in undergraduate medical education. MethodThree raters independently compared 280 checklists from 4 disciplines contributing to the undergraduate year 4 OSCE against the 17 items of the MAAS-Global standard. G-theory was used to analyze the reliability of this calibration procedure. ResultsG-Kappa was 0.8. For two raters G-Kappa is 0.72 and it fell to 0.57 for one rater. 46% of the checklist items corresponded to section three of the MAAS-Global (i.e. medical content of the consultation), whilst 12% corresponded to section two (i.e. general CS), and 8.2% to section one (i.e. CS for each separate phase of the consultation). 34% of the items were not considered to be CS. ConclusionA G-Kappa of 0.8 confirms a reliable and valid procedure for calibrating OSCE CS checklist items using the MAAS-Global. We strongly suggest that such a procedure is more widely employed to arrive at a stable (valid and reliable) judgment of the communication component in existing checklists for medical students’ communication behaviours. Practice implicationsIt is possible to measure the ‘true’ caliber of CS in OSCE stations. Students’ results are thereby comparable between and across stations, students and institutions. A reliable calibration procedure requires only two raters.

Effect of novel patient interaction on students’ performance of pregnancy options counseling

BackgroundAlthough options counseling is a fundamental skill for medical providers, previous research has identified gaps in medical school reproductive health education.PurposeTo determine if a 1-h novel patient interaction (NPI) improves student performance when caring for a standardized patient with an unintended pregnancy.MethodsFrom September 2012 to June 2013 we randomized third-year medical students at the University of Colorado School of Medicine to the standard curriculum plus an NPI, or the standard curriculum only. The NPI consisted of a 1-h small-group session with a patient who discussed her experiences with options counseling and her decision to terminate her pregnancy. Students completed an Objective Structured Clinical Examination (OSCE) at the rotation's end, which included options counseling. The primary outcome was the proportion of participants achieving ‘excellence’ on the OSCE checklist. ‘Excellence’ was defined as a score ≥90%. Examinations were flagged as ‘unsatisfactory encounters’ if core competencies were not addressed. OSCE standardized patients and evaluators were blinded to group assignment.ResultsIn total, 135 students were eligible and randomized: 75 to NPI; 60 to control. During the OSCE, few students achieved ‘excellence’ (24% NPI vs. 28% control, p=0.57).There were no differences between scores for components of options counseling. More students in the control group ‘appeared somewhat uncomfortable’ delivering the pregnancy test results (5% NPI vs. 18% control, p=0.006). More than half (54%) of the intervention group and 67% of controls had ‘unsatisfactory encounters’ (p=0.16), almost exclusively due to omission of adoption. Most students addressed abortion (96% NPI vs. 92% control, p=0.29).ConclusionsA 1-h NPI does not improve medical students’ performance of pregnancy options counseling and the option of adoption is routinely omitted. Adoption is clearly an area that needs greater attention when designing comprehensive reproductive health curriculum for medical students.

Validation of the Queen’s University Ophthalmoscopy Objective Structured Clinical Examination Checklist to predict direct ophthalmoscopy proficiency

To validate the Queen's University Ophthalmoscopy Objective Structured Clinical Examination (OSCE) Checklist (QUOOC) by correlating the score derived from this instrument to the ability of a candidate to correctly identify retinal pathology. Educational Instrument Validation Study. Thirteen second-year medical students and 4 ophthalmology residents participated in an OSCE station examining the posterior segment of a series of patients' left eyes with the direct ophthalmoscope. Participants were asked to evaluate the cup-disc ratio, disc margins, and the macula for the presence of pathology. They were also evaluated for proper direct ophthalmoscopy skills. Each participant was assigned a composite clinical proficiency score. Participants' skills were scored using the QUOOC. Based on this evaluation, we correlated clinical proficiency (QUOOC) score to diagnostic accuracy. QUOOC was completed successfully within a 3-minute time frame that was deemed to be practical given the physical and time constraints of OSCE stations. Significant (P<0.01) differences were noted between medical students and residents with regard to both technique and clinical proficiency. A strong correlation was found between QUOOC score and a participant's ability to correctly detect retinal pathology (r = 0.71; P = 0.001). QUOOC was successful in discriminating between more and less experienced and competent participants. Using the QUOOC to evaluate proficiency at direct ophthalmoscopy is a good predictor of correctly detecting retinal pathology when present. We believe that using the QUOOC with standardized patients without physical findings is an accurate and feasible way of measuring students' proficiency in using direct ophthalmoscope in OSCE stations.

Useful apps for OSCE practice

OSCE (objective structured clinical examination) time can be a stressful time for many medical students who might feel underprepared and inexperienced. In this article we give an overview of the most useful apps for OSCE practice. -Free (offers in-app purchases, £3.99 to unlock all content) -Android (4.4/5 from 238 reviews) -Apple (4/5 from 22 reviews) They say: “Learn on the go with our comprehensive collection of high quality OSCE guides covering a wide range of commonly tested clinical skills.” Features -Guides to the following OSCEs: cardiovascular, respiratory, gastrointestinal, neurology, musculoskeletal, endocrine, obstetrics & gynaecology, ear, nose & throat, breast examination, clinical procedures and communication skills -Video demonstrations and images -Checklist mark schemes -Customisable timer Student BMJ’s verdict: This is one of the few OSCE apps on the market which includes videos and images of the clinical examinations, as well as text. Most of the content is locked until paid for. -Free - offers in-app purchase of £1.49 -Android (4/5 from 61 reviews) -Not available for Apple devices They say “Revision aide for medical OSCE examinations. Contains 47 mock OSCE sheets, which provides a checklist for students to work through in pairs, checking off each item to ensure correct procedure” Features -47 mock OSCE checklists - Stations include history-taking, practical and clinical skills - Available in test and practical mode - Timer mode - Exam mode which runs consecutive, random stations Student BMJ’s verdict You have to pay for the clinical OSCE checklists, but the free material is good quality. The app is easy to use and well designed. Being able to …

Faculty development on item writing substantially improves item quality

The quality of items written for in-house examinations in medical schools remains a cause of concern. Several faculty development programs are aimed at improving faculty's item writing skills. The purpose of this study was to evaluate the effectiveness of a faculty development program in item development. An objective method was developed and used to assess improvement in faculty's competence to develop high quality test items. This was a quasi experimental study with a pretest-midtest-posttest design. A convenience sample of 51 faculty members participated. Structured checklists were used to assess the quality of test items at each phase of the study. Group scores were analyzed using repeated measures analysis of variance. The results showed a significant increase in participants' mean scores on Multiple Choice Questions, Short Answer Questions and Objective Structured Clinical Examination checklists from pretest to posttest (p < .0005). The effect sizes were 1.38, 3.84 and 2.20 for Multiple Choice Questions, Short Answer Questions and Objective Structured Clinical Examination, respectively. This study emphasizes that items written by faculty without faculty development are generally lacking in quality. It also provides evidence of the value of faculty development in improving the quality of items generated by faculty.

Die Operationalisierung der HNO-Spiegeluntersuchung

Standardized procedures support learning of complex psychomotoric activities. We standardized the clinical otolaryngology examination to provide an objective basis for its operationalization. The Impact of the standardization on the objective measureable learning success and the student's evaluation were determined.During their 1-week ENT internship 166 medical students learned the standardized clinical otolaryngology examination. At the end of this week they absolved an objective structured clinical examination (OSCE). The students evaluated both, the teaching (standardization) and examination method (OSCE) with 2 questionnaires.All students passed the exam with mean score of 60.3±3.5 (max. 65) points. By using the OSCE checklist 2 independent examiners appraised nearly identical examination performances. In both evaluations the students rated the teaching and examination design positive. All students greatly appreciated the standardized clinical examinations and considered it as are very important for learning practical skills.The standardization of clinical otolaryngology examination permits objective and reliable tests for learning progress and success. Furthermore transparency, process reliability and test quality are considerably improved. The student's evaluation clearly reflects the mentioned advantages.

Development and pilot testing of an objective structured clinical examination (OSCE) on hoarseness

To develop a valid and reliable tool for an objective structured clinical examination (OSCE) on hoarseness. To pilot-test the feasibility by assessing residents' clinical skills in various core competencies while assessing hoarseness on a standardized patient (SP). Educational tool development. The OSCE checklists were developed using modified Delphi technique after obtaining feedback from faculty involved in providing care to hoarseness patients. SP-based and rest stations were created to assess clinical skills. Twelve Otolaryngology-Head and Neck Surgery residents participated in the study. Video recordings of residents' performance and their written documentation were rated by faculty members. The OSCE that we developed is a valid method of assessing residents' clinical skills for evaluating hoarseness. Senior residents performed better in all of the tasks such as obtaining history and performing a physical exam on an SP, ability to perform flexible laryngoscopy on a mannequin, and interpretation of radiologic findings. Internal consistency assessed by Cronbach's alpha as measure of inter-item reliability was 0.92 for laryngoscopic station and 0.95 for radiology station. This OSCE can be effectively used for the objective assessment of clinical competency in hoarseness. Our pilot study evaluated multiple competencies on a single occasion, including medical knowledge, patient care, professionalism, and communication and interpersonal skills. Clinical competence in history taking, physical examination, flexible fiber-optic laryngoscopy, and ability to interpret radiologic findings improved with increasing year of training. This OSCE provides targeted assessment of practice-based learning and feedback for improvement of clinical performance.

“You can do it late at night or in the morning. You can do it at home, I did it with my flatmate.” The educational impact of an OSCE

Background: The use of an objective structured clinical examination (OSCE) has been a powerful influence on doctor training but assessments do not always drive study behaviour in predictable ways.Aims: To investigate the impact an OSCE has on study behaviours by exploring how 5th year medical students identify what to learn for a summative OSCE and the role of the clinical environment in their preparation.Methods: A semi-structured questionnaire survey asked about strategies used by students to prepare for the OSCE. Focus group interviews explored successful methods of preparation for the OSCE. Themes were identified and classified.Results: The questionnaire response rate was 84%. Topic identification was usually from the list of examinable problems, past OSCE papers and a booklet prepared by a previous student containing a series of OSCE checklists. The study behaviours of students preparing for the OSCE exam were predominantly to practise on each other, and to rehearse routines. Strategic and efficient study habits were favoured over conscious utilization of the clinical environment.Conclusion: The expectation that an OSCE drives learning into the clinical workplace was not supported by this study. This suggests the role of clinical experience in helping students prepare for the exam may be more subliminal, or that an OSCE is more as a test of psychomotor skills than a marker of clinical experience. An unexpected benefit may be to drive more collaborative learning.

The reliability of six faculty members in identifying important OSCE items.

To measure the agreement among faculty members about the importance of items on a checklist used to grade an objective structured clinical examination (OSCE) station. Six faculty members rated the importance of 47 items for an OSCE station in which students took the history of a patient with sore throat and hoarseness. Of the 47 items, the raters independently identified 15 items as very important. The reliability of each individual rater was fair (averaged value of alpha = .63.) The reliability of the mean rating of the six raters was high (alpha = .91). The results strongly suggest that when a group of faculty members cooperatively identifies the important items to be included in an OSCE checklist, the reliability of the checklist is superior to one created by a single author.