Assessment In Medical Education Research Articles

Advancing Medical Students’ Non-technical Skills in a Group-Based Setting

Aim of the study: Introduction of non-technical skill (NTS) modules in undergraduate medical curricula is essential to prevent errors that occur due to inadequate communication and malfunctioning teams. We primarily aimed to observe medical students’ NTS performance improvement in a group-based setting and compare it across participants of different countries of study. Secondarily, we aimed to assess students’ views on such initiatives. Material and Methods: We designed a pilot module for assessing and improving a group’s NTS performance. This was based on three main pillars: simulation-based learning, constructive feedback and objective assessment. Senior medical students from EU universities were invited to apply online. Delegates were divided according to their country of study and participated in two surgical scenarios. Groups had two attempts at each scenario and feedback was given in between. Performance was assessed using a modified version of the Individual Teamwork Observation and Feedback Tool (iTOFT). Participant feedback was collected at the end of the module. Results: Eighty delegates were recruited and finished the study. A mean increase performance of 0.81 out of 3 was noted between attempts in trauma-case scenarios and 0.88 out of 3 in ward round scenarios. Overall performance improvement was similar across all groups (p > 0.05). Feedback of delegates was encouragingly positive. Conclusions: The reported improvement and positive feedback received highlight that such an intervention can be implemented in medical school curricula of all countries. Our module design, along with the use of iTOFT, should act as the basis for future development of group-based learning and assessment in undergraduate medical education.

Gender Bias in Resident Assessment in Graduate Medical Education: Review of the Literature.

Competency-based medical education relies on meaningful resident assessment. Implicit gender bias represents a potential threat to the integrity of resident assessment. We sought to examine the available evidence of the potential for and impact of gender bias in resident assessment in graduate medical education. A systematic literature review was performed to evaluate the presence and influence of gender bias on resident assessment. We searched Medline and Embase databases to capture relevant articles using a tiered strategy. Review was conducted by two independent, blinded reviewers. We included studies with primary objective of examining the impact of gender on resident assessment in graduate medical education in the USA or Canada published from 1998 to 2018. Nine studies examined the existence and influence of gender bias in resident assessment and data included rating scores and qualitative comments. Heterogeneity in tools, outcome measures, and methodologic approach precluded meta-analysis. Five of the nine studies reported a difference in outcomes attributed to gender including gender-based differences in traits ascribed to residents, consistency of feedback, and performance measures. Our review suggests that gender bias poses a potential threat to the integrity of resident assessment in graduate medical education. Future study is warranted to understand how gender bias manifests in resident assessment, impact on learners and approaches to mitigate this bias.

Collecting Validity Evidence: A Hands-on Workshop for Medical Education Assessment Instruments.

There is an increasing call for developing validity evidence in medical education assessment. The literature lacks a practical resource regarding an actual development process. Our workshop teaches how to apply principles of validity evidence to existing assessment instruments and how to develop new instruments that will yield valid data. The literature, consensus findings of curricula and content experts, and principles of adult learning guided the content and methodology of the workshop. The workshop underwent stringent peer review prior to presentation at one international and three national academic conferences. In the interactive workshop, selected domains of validity evidence were taught with sequential cycles of didactics, demonstration, and deliberate practice with facilitated feedback. An exercise guide steered participants through a stepwise approach. Using Likert-scale items and open-response questions, an evaluation form rated the workshop's effectiveness, captured details of how learners reached the objectives, and determined participants' plans for future work. The workshop demonstrated generalizability with successful implementation in diverse settings. Sixty-five learners, the majority being clinician-educators, completed evaluations. Learners rated the workshop favorably for each prompt. Qualitative comments corroborated the workshop's effectiveness. The active application and facilitated feedback components allowed learners to reflect in real time as to how they were meeting a particular objective. This feasible and practical educational intervention fills a literature gap by showing the medical educator how to apply validity evidence to both existing and in-development assessment instruments. Thus, it holds the potential to significantly impact learner and, subsequently, patient outcomes.

Identifying the narrative used by educators in articulating judgement of performance.

IntroductionModern assessment in medical education is increasingly reliant on human judgement, as it is clear that quantitative scales have limitations in fully assessing registrars’ development of competence and providing them with meaningful feedback to assist learning. For this, possession of an expert vocabulary is essential.AimThis study aims to explore how medical education experts voice their subjective judgements about learners and to what extent they are using clear, information-rich terminology (high-level semantic qualifiers); and to gain a better understanding of the experts’ language used in these subjective judgements.MethodsSix experienced medical educators from urban and rural environments were purposefully selected. Each educator reviewed a registrar clinical case analysis in a think out loud manner. The transcribed data were analyzed, codes were identified and ordered into themes. Analysis continued until saturation was reached.ResultsFive themes with subthemes emerged. The main themes were: (1) Demonstration of expertise; (2) Personal credibility; (3) Professional credibility; (4) Using a predefined structure and (5) Relevance.DiscussionAnalogous to what experienced clinicians do in clinical reasoning, experienced medical educators verbalize their judgements using high-level semantic qualifiers. In this study, we were able to unpack these. Although there may be individual variability in the exact words used, clear themes emerged. These findings can be used to develop a helpful shared narrative for educators in observation-based assessment. The provision of a rich, detailed narrative will also assist in providing clarity to registrar feedback with areas of weakness clearly articulated to improve learning and remediation.

Leveraging Natural Language Processing: Toward Computer-Assisted Scoring of Patient Notes in the USMLE Step 2 Clinical Skills Exam.

The United States Medical Licensing Examination Step 2 Clinical Skills (CS) exam uses physician raters to evaluate patient notes written by examinees. In this Invited Commentary, the authors describe the ways in which the Step 2 CS exam could benefit from adopting a computer-assisted scoring approach that combines physician raters' judgments with computer-generated scores based on natural language processing (NLP). Since 2003, the National Board of Medical Examiners has researched NLP technology to determine whether it offers the opportunity to mitigate challenges associated with human raters while continuing to capitalize on the judgment of physician experts. The authors discuss factors to consider before computer-assisted scoring is introduced into a high-stakes licensure exam context. They suggest that combining physician judgments and computer-assisted scoring can enhance and improve performance-based assessments in medical education and medical regulation.

Nelson-Denny Reading Test Scores as a Predictor of Student Success in Osteopathic Medical Education

Reading skills are crucial in medical school, where students are expected to absorb an onslaught of new and complex material. Studies on reading assessment in osteopathic medical education are lacking. To address gaps in the literature related to reading assessment and to investigate the correlation of the Nelson-Denny Reading Test with various performance indicators in osteopathic medical education. The West Virginia School of Osteopathic Medicine administered the Nelson-Denny Reading Test to first- and second-year students between 2015 and 2017. Raw scores were translated into the percentile rank, scale score, grade equivalent score, and stanine score based on guidelines supplied with the Nelson-Denny Reading Test. These translated scores were compared with Medical College Admission Test (MCAT) scores, first- and second-year performance on course examinations, Comprehensive Osteopathic Medical Licensing Examination-USA (COMLEX-USA) Level 1 scores, and scores provided in a 2002 study by Haught and Walls. A total of 623 students took the first-year Nelson-Denny Reading Test, and 408 took both the first- and second-year Nelson-Denny Reading Test. Findings showed a large correlation between the Nelson-Denny Reading Test and the verbal reasoning section (r=0.56 for the class of 2020 and 0.46 for the class of 2021) of the old MCAT (before 2015) and the reasoning skills section (r=0.42 for the class of 2020 and 0.49 for the class of 2021) of the new MCAT (released in 2015). There were no correlations with first- and second-year course examination scores or COMLEX-USA Level 1 scores. The Nelson-Denny Reading Test scores reported by Haught and Walls for medical students and health professional students were slightly higher than those found for osteopathic medical students in this study. The reasoning skills section of the new MCAT could serve as a good proxy for a reading test. There were no correlations between the Nelson-Denny Reading Test and performance in the first 2 years of medical school or COMLEX-USA Level 1 performance. Further research can strengthen the findings and determine whether correlations exist with clinical performance.

The Effect of the Dr. LEE Jong‐wook—Seoul Project on Medical Student Academic Performance in Laos

The Dr. LEE Jong-wook—Seoul Project (DLSP) in the Lao People’s Democratic Republic (PDR) is a long-term development cooperation initiative aimed at upgrading the medical education capacity at the University of Health Sciences (UHS) in the Lao PDR. Under the project, eight faculty members from UHS were trained at the Seoul National University College of Medicine for a 12-month period during 2010–2011. Using a training-of-trainers model, we estimated the effect of DLSP on the learning outcomes of students, which was measured using the standard test scores of the students. The questionnaire developed by the Medical Education Assessment Consortium in Korea was given to the students and both the percent-correct and standardized scores were examined. The evaluation strategy was to compare the learning outcomes of the students taught by the trainees with those that were not and to compare the medical topics taught by the trainees with other topics. It was found that the first group of trainees in the DLSP was associated with an increase of 1.5 in the percent-correct test scores for basic medicine. While the interpretation requires caution, it is consistent with the view that a training program for faculty members could be effective in improving the teaching capacity of medical schools in developing countries.

Development of multiple choice question bank in otorhinolaryngology by item analysis: a cross-sectional study

<p class="abstract"><strong>Background:</strong> Multiple choice questions (MCQs) are routinely used for formative and summative assessment in medical education. Item analysis is a process of post validation of MCQ tests, whereby items are analyzed for difficulty index, discrimination index and distractor efficiency, to obtain a range of items of varying difficulty and discrimination indices. This study was done to understand the process of item analysis and analyze MCQ test so that a valid and reliable MCQ bank in otorhinolaryngology is developed.</p><p class="abstract"><strong>Methods:</strong> 158 students of 7<sup>th</sup> Semester were given an 8 item MCQ test. Based on the marks achieved, the high achievers (top 33%, 52 students) and low achievers (bottom 33%, 52 students) were included in the study. The responses were tabulated in Microsoft Excel Sheet and analyzed for difficulty index, discrimination index and distractor efficiency. </p><p class="abstract"><strong>Results:</strong> The mean (SD) difficulty index (Diff-I) of 8 item test was 61.41% (11.81%). 5 items had a very good difficulty index (41% to 60%), while 3 items were easy (Diff-I >60%). There was no item with Diff-I <30%, i.e. a difficult item, in this test. The mean (SD) discrimination index (DI) of the test was 0.48 (0.15), and all items had very good discrimination indices of more than 0.25. Out of 24 distractors, 6 (25%) were non-functional distractors (NFDs). The mean (SD) distractor efficiency (DE) of the test was 74.62% (23.79%).</p><p class="abstract"><strong>Conclusions:</strong> Item analysis should be an integral and regular activity in each department so that a valid and reliable MCQ question bank is developed.</p>

Simulation Training in Medical Education\u2014an Exploration Through Different Theoretical Lenses

Different theoretical frameworks offer specific, but separate, understandings of the same phenomenon. With the increasing use of simulation for training and assessment in medical education, it is vital to consider how different frameworks grant various insights into the pedagogical value of simulation. In this article, the author evaluated three exemplar theoretical frameworks, cultural-historical activity theory, cognitive load theory, and grounded theory, considering their ontological and epistemological stances, their limitations, and their application to simulation training. The greater understanding offered by this article will inform research design and interpretation of results, enabling a more theoretically poised construction of pedagogical techniques.

Use of E-learning tools for assessment in medical education

Use of E-learning tools for assessment in medical education

Rationale for Programmatic Assessment in Medical Education and Overcoming the Anticipated Challenges During Its Implementation

Rationale for Programmatic Assessment in Medical Education and Overcoming the Anticipated Challenges During Its Implementation

Assessment in competency-based medical education: Significance and the existing challenges

Assessment in competency-based medical education: Significance and the existing challenges

A Novel Method of Setting Performance Standards in Surgery Using Patient Outcomes.

To describe a novel, outcome-based method of standard setting that differentiates between clinical outcomes rather than arbitrary educational goals. Standard setting methods used in assessments of procedural skill are currently not evidence-driven or outcome-based. This represents a potential obstacle for the broad implementation of these evaluations in summative assessments such as certification and credentialing. The concept is based on deriving a receiver operating characteristic curve from a regression model that incorporates measures of intraoperative surgeon performance and confounding patient characteristics. This allows the creation of a performance standard that best predicts a clinically significant outcome of interest. The discovery cohort used to create the predictive model was derived from pilot data that used the Global Evaluative Assessment of Robotic Skill assessment tool to predict patient urinary continence 3 months following robotic-assisted radical prostatectomy. A receiver operating characteristic curve with an area under the curve of 0.75 was created from predicted probability statistic generated by the predictive model. We chose a predicted probability of 0.35, based on an optimal tradeoff in sensitivity and specificity (Youden Index). Rearranging the regression equation, we determined the performance score required to predict a 35%, patient-adjusted probability of postoperative urinary incontinence. This novel methodology is context, patient, and assessment-specific. Current standard setting methods do not account for the heterogeneity of the clinical environment. Workplace-based assessments in competency-based medical education require standards that are credible to the educator and the trainee. High-stakes assessments must ensure that surgeons have been evaluated to a standard that prioritizes satisfactory patient outcomes and safety.

A Graduated Responsibility Supervising Resident Experience Using Mastery Learning Principles [Correction

MedEdPublish (ISSN 2312–7996) is a highly visible, open access, specialist practitioner e-journal that enables academics, teachers, clinicians, researchers and students to publish their experiences, views and research findings relating to teaching, learning and assessment in medical and health professions education. An innovative and key feature of MedEdPublish is that the peer review process begins following publication.

Benefits of a dedicated ultrasound education for medical students: A 3 year experience [Correction

MedEdPublish (ISSN 2312–7996) is a highly visible, open access, specialist practitioner e-journal that enables academics, teachers, clinicians, researchers and students to publish their experiences, views and research findings relating to teaching, learning and assessment in medical and health professions education. An innovative and key feature of MedEdPublish is that the peer review process begins following publication.

Self-explanation to support knowledge development for clinical reasoning: Perspectives from third year medical clerks [Correction

MedEdPublish (ISSN 2312–7996) is a highly visible, open access, specialist practitioner e-journal that enables academics, teachers, clinicians, researchers and students to publish their experiences, views and research findings relating to teaching, learning and assessment in medical and health professions education. An innovative and key feature of MedEdPublish is that the peer review process begins following publication.

Getting Value from Open Educational Repositories [Correction

MedEdPublish (ISSN 2312–7996) is a highly visible, open access, specialist practitioner e-journal that enables academics, teachers, clinicians, researchers and students to publish their experiences, views and research findings relating to teaching, learning and assessment in medical and health professions education. An innovative and key feature of MedEdPublish is that the peer review process begins following publication.

Summative Objective Structured Clinical Examination Assessment: A Mini Review

The end of any form of rigorous, specific, and high-stakes process of systematic and structured instruction is characterized by a summative assessment to evaluate the learner’s ability to apply the body of knowledge or clinical skills encountered and interacted with over a specific duration and to progress to the next phase of training. The aim of this project is to analyze the design, validity, delivery, supervision, and feedback of a summative objective structured clinical examination (OSCE) assessment. The results of this analysis showed a satisfactory and widely accepted method of assessment in medical education despite a few shortcomings. All forms of assessment have their inherent strengths and weaknesses, but it is essential that these assessments encourage future learning. Summative assessment has been proven to be a valid, comprehensive, and reliable method, and most importantly, it allows direct observation and evaluation of procedural and clinical skills.

A pilot survey of student perceptions on the benefit of the OSCE and MCQ modalities.

Objective: The objective structured clinical examination (OSCE) has become widely accepted as a form of assessment in medical education. At the same time, the more traditional multiple choice question (MCQ) examinations remain a central modality of student assessment. This pilot survey aimed to investigate students’ perceptions about the benefits of the OSCE and MCQs to yield data supporting the implementation of this assessment strategy into the national medical licensing examination in Germany.Methods: A questionnaire was delivered electronically to 34 German medical schools. Students in years 3-6 were invited to rate 11 items about objectives of good medical assessment. All items were presented for both the OSCE and MCQs using a 5-point Likert Scale (1=strongly disagree to 5=strongly agree). Factor analysis was used to identify underlying components in the ratings. Average scores of items that belonged to a component were computed.Results: Data analysis included 1,082 students from 32 medical schools. For the OSCE, factor analysis revealed two components, which were labelled “educational impact” and “development of clinical competence”. The average scores of items were 3.37 and 3.55, respectively. For the MCQ modality, also two components emerged. These were labelled “perceived weaknesses of MCQs” and “perceived strengths of MCQs” (consisting of items such as “promotes my theoretical knowledge”). The average scores for these components were 1.85 and 3.62.Conclusion: The results of this pilot survey indicate that students consider both OSCE and MCQs as useful assessments for the purposes for which they were designed. The assessment strategy thus appears appropriate and it should be used in the national licensing examination.

Association of a Competency-Based Assessment System With Identification of and Support for Medical Residents in Difficulty

Competency-based medical education is now established in health professions training. However, critics stress that there is a lack of published outcomes for competency-based medical education or competency-based assessment tools. To determine whether competency-based assessment is associated with better identification of and support for residents in difficulty. This cohort study of secondary data from archived files on 458 family medicine residents (2006-2008 and 2010-2016) was conducted between July 5, 2016, and March 2, 2018, using a large, urban family medicine residency program in Canada. Introduction of the Competency-Based Achievement System (CBAS). Proportion of residents (1) with at least 1 performance or professionalism flag, (2) receiving flags on multiple distinct rotations, (3) classified as in difficulty, and (4) with flags addressed by the residency program. Files from 458 residents were reviewed (pre-CBAS: n = 163; 81 [49.7%] women; 90 [55.2%] aged >30 years; 105 [64.4%] Canadian medical graduates; post-CBAS: n = 295; 144 [48.8%] women; 128 [43.4%] aged >30 years; 243 [82.4%] Canadian medical graduates). A significant reduction in the proportion of residents receiving at least 1 flag during training after CBAS implementation was observed (0.38; 95% CI, 0.377-0.383), as well as a significant decrease in the numbers of distinct rotations during which residents received flags on summative assessments (0.24; 95% CI, 0.237-0.243). There was a decrease in the number of residents in difficulty after CBAS (from 0.13 [95% CI, 0.128-0.132] to 0.17 [95% CI, 0.168-0.172]) depending on the strictness of criteria defining a resident in difficulty. Furthermore, there was a significant increase in narrative documentation that a flag was discussed with the resident between the pre-CBAS and post-CBAS conditions (0.18; 95% CI, 0.178-0.183). The CBAS approach to assessment appeared to be associated with better identification of residents in difficulty, facilitating the program's ability to address learners' deficiencies in competence. After implementation of CBAS, residents experiencing challenges were better supported and their deficiencies did not recur on later rotations. A key argument for shifting to competency-based medical education is to change assessment approaches; these findings suggest that competency-based assessment may be useful.