National Board Of Medical Examiners Research Articles

Effectiveness of Near-Peer-Taught Case Reviews on Students' Confidence in National Board of Medical Examiners (NBME) Exams.

Introduction A key element to a first-year medical student's (MS1) education is guidance and practice in applying anatomy, pathophysiology, diagnosis, and treatment concepts to clinical vignettes. One potential solution to providing effective clinical reasoning training is the involvement of second-year medical students (MS2s) in small group sessions as teachers to provide more personalized instruction via case-based learning sessions. Near-peer teaching has been shown to benefit students' confidence in learning and improve test scores. Similarly, case-based learning is heavily associated with an improved understanding of complex topics. As such, this study assessed the efficacy of near-peer teaching with concomitant case-based style presentations on improving the comfort of MS1s with their understanding of the curriculum content and their comfort with applying their knowledge to clinical scenarios. Methods This randomized controlled crossover trial included several small-group study sessions, each consisting of five MS1s and led by an MS2 who reviewed selected clinical cases in a standardized slide decks. The control arm was provided the same slide decks but did not participate in the MS2-led sessions. During the first course, students were assigned to either the control or intervention group and then crossed over to the opposite group (control to intervention and vice versa) in every subsequent course. Comfort with the curriculum material was then assessed through pre and post surveys, with the post surveys administered after the MS1s took their final NBME examination for that course. Results The study was cut short due to COVID-19 precautions limiting in-person sessions. Nevertheless, the post survey demonstrated an increased understanding of pathological concepts for the intervention group compared to the control group. Conclusions Future work on near-peer group study sessions should enroll a larger sample size with measures to improve the response rate to better test whether near-peer-led case reviews had a significant effect on students' understanding of anatomical concepts and confidence during NBME examinations.

Consider implications of ruling that deemed gifted student eligible for testing modifications

A New York federal court granted a preliminary injunction to a gifted student with learning disabilities and attention deficit hyperactivity disorder, holding that the National Board of Medical Examiners had to provide the student with double time for the Step 1 U.S. Medical Licensing Examination. The court rejected the NBME's argument that the student wasn’t an individual with a disability due to his academic success. (Sampson v. Nat’l Bd. of Med. Exam’rs, 22‐CV‐05120 (E.D.N.Y. 2022).

Medical Student Duty Hours and Shelf Performance: Is There a Correlation?

To date, there is limited information about medical student duty hours, shelf scores, and overall clerkship performance in obstetrics and gynecology (OB/GYN). As a result, we were curious to know whether spending more time in the clinical environment translated to an improved learning experience or, in contrast, translated to decreased study time and worse overall clerkship performance. A retrospective cohort analysis was performed at a single academic medical center of all medical students on the OB/GYN clerkship from August 2018 to June 2019. Recorded student duty hours were tabulated per day and per week, by student. National Board of Medical Examiners (NBME) Subject Exam (Shelf) equated percentile scores for the quarter of year were used. Our statistical analysis showed that working long hours did not translate to higher or lower shelf score, or higher overall clerkship grade. However, working longer hours in the last 2 weeks of the clerkship was associated with high shelf score. Longer medical student duty hours did not correlate to higher shelf scores or overall clerkship grades. Future multicenter studies are necessary to evaluate the importance of medical student duty hours and continue optimizing the educational experience of the OB/GYN clerkship. · Clinical hours were not associated with shelf examination scores.. · Clinical hours were not associated with overall clerkship grade.. · Longer clinical hours at the end of clerkship are correlated with higher examination scores..

The importance of USMLE step 2 on the screening and selection of applicants for general surgery residency positions

BackgroundAs announced by the Federation of State Medical Boards (FSMB) and National Board of Medical Examiners (NBME), the United States Medical Licensing Examination (USMLE) Step 1 score reporting has transitioned to pass/fail outcomes instead of the traditional numeric score after January 26, 2022. USMLE Step 1 scores have been used widely as a crucial tool in screening and selecting applicants for residency programs. This study aims to determine the role of USMLE Step 2 in the selection of applicants for general surgery residency. MethodsA retrospective study was conducted over six recruiting cycles from 2016 to 2021. The data from 334 interviewed applicants from one general surgery residency program were assessed. Data analyzed included USMLE Step 1 and Step 2 scores, applicant gender, Alpha Omega Alpha (AOA) status, letters of recommendation (LOR), and research/publications (RS). ResultsOf the 334 interviewed applicants, 209 (62.6%) were male. The mean [SD] USMLE Step 1 and USMLE Step 2 C K (Clinical Knowledge) scores were 239.6 [±10.4] and 249.2 [±11.4], respectively. The mean (SD) LOR and RS scores were 4.24 [±0.4] and 3.9 [±0.7], respectively. A positive correlation was observed between USMLE Step 1 and USMLE Step 2 C K (Clinical Knowledge) scores (r = 0.60, p < .001), LOR scores (r = 0.24, p = .008), and AOA status (r = 0.19, p = .038). There was a negligible correlation between USMLE scores and applicant gender. ConclusionTransitioning USMLE Step 1 to pass/fail will make the initial screening and selection process of applications challenging for residency programs. In the short term, USMLE Step 2 scores, LOR, and AOA status are important as screening assessments. Valid measures to ensure appropriate, equitable, and fair assessments are needed.

Medical Students' Socioeconomic Status and Academic Performance in Medical School.

Background Students from lower socioeconomic groups tend to underestimate their chances of acceptance to medical school and their likelihood of success once admitted. Objective The objective of this study is to determine if socioeconomic disadvantage status is linked to lower medical college admission test (MCAT) scores and academic performance in medical school. Methods Using the Association of American Medical Colleges (AAMC) education/occupation (EO) indicator, we compared economically disadvantaged students to students with no financial disadvantage on the MCAT, Phase 1 National Board of Medical Examiners (NBME), United States Medical Licensing Examination (USMLE) Step 1, Phase 2 NBME, and USMLE Step 2 test scores. Results Medical students in the disadvantaged group scored significantly lower on the MCAT than students with no financial disadvantage. The disadvantaged group showed a non-significant lower trend in performance until USMLE Step 2. Conclusions Applicants from lower socioeconomic backgrounds may perform lower on their MCAT and early medical school benchmark exams, but they appear to catch up with and even surpass their peers on their USMLE Step 2 examination.

Exploring Radiation Oncology Representation on the National Board of Medical Examiners (NBME) Official Practice Material for the Undergraduate United States National Standardized Medical Board Examinations

Background The portrayal of oncology-related topics on the National Board of Medical Examiners (NBME) published practice material for undergraduate medical education (UGME) standardized examinations including the United States Medical Licensing Exams (USMLE) Step 1, Step 2 Clinical Knowledge (CK), and NBME Clinical Science Subject Examinations ('Shelf') is largely unknown. The goal of this study is to explore the presence and characterization of radiation oncology topics, including radiation therapy (RT), on these practice examinations. Methods All current and commercially accessible NBME published self-assessments and sample questions for the aforementioned examinations will be queried for content pertaining to oncology and associated treatment modalities of RT, surgery, and systemic therapy. The total volume of oncology-related questions and its distribution across all 3 major practice examination types will be recorded. The questions will be cataloged by selected attributes including context, treatment modality appearing as the correct answer for "next best step" (NBS) prompt, and answers appearing as the incorrect answer for NBS prompt. Results 1320 USMLE Step 1 practice questions are available for analysis. In preliminary analysis, minimal questions about RT were noted; all appearing to involve concepts of associated toxicity. 920 USMLE Step 2CK practice questions are available for analysis. RT-related questions appeared with higher frequency on the clinical science preparation material (n=10) compared to basic science (n=4), while also notably appearing as a listed answer choice for NBS prompt against other treatment modalities. 1325 practice questions from relevant NBME shelf examinations were available. RT appeared sparingly on the practice NBME shelf examinations of Surgery (n=5), Medicine (n=4), Ob/Gyn (n=2), and Clinical Neurology (n=2). RT questions were absent from Pediatrics, Emergency Medicine, and Family Medicine Shelf practice examination material. In the final analysis of these practice questions from all 3 major examination types, we seek to conduct a comparative analysis to other treatment modalities. Discussion This work represents the first attempt, to our knowledge, to characterize RT topic appearance within the NBME published practice questions and their accompanying explanations for the USMLE Step 1, Step 2CK, and select NBME shelf examinations. Our analysis is important in assessing the presence of RT topics on medical student board preparation material, a potential avenue of RT content exposure in UGME. Careful review and comparative analyses of all oncology-related content can provide important insight for UGME-related RT education. Our anticipated analyses and future discussion may identify areas for improvements in the important education of oncology-related topics in UGME. The portrayal of oncology-related topics on the National Board of Medical Examiners (NBME) published practice material for undergraduate medical education (UGME) standardized examinations including the United States Medical Licensing Exams (USMLE) Step 1, Step 2 Clinical Knowledge (CK), and NBME Clinical Science Subject Examinations ('Shelf') is largely unknown. The goal of this study is to explore the presence and characterization of radiation oncology topics, including radiation therapy (RT), on these practice examinations. All current and commercially accessible NBME published self-assessments and sample questions for the aforementioned examinations will be queried for content pertaining to oncology and associated treatment modalities of RT, surgery, and systemic therapy. The total volume of oncology-related questions and its distribution across all 3 major practice examination types will be recorded. The questions will be cataloged by selected attributes including context, treatment modality appearing as the correct answer for "next best step" (NBS) prompt, and answers appearing as the incorrect answer for NBS prompt. 1320 USMLE Step 1 practice questions are available for analysis. In preliminary analysis, minimal questions about RT were noted; all appearing to involve concepts of associated toxicity. 920 USMLE Step 2CK practice questions are available for analysis. RT-related questions appeared with higher frequency on the clinical science preparation material (n=10) compared to basic science (n=4), while also notably appearing as a listed answer choice for NBS prompt against other treatment modalities. 1325 practice questions from relevant NBME shelf examinations were available. RT appeared sparingly on the practice NBME shelf examinations of Surgery (n=5), Medicine (n=4), Ob/Gyn (n=2), and Clinical Neurology (n=2). RT questions were absent from Pediatrics, Emergency Medicine, and Family Medicine Shelf practice examination material. In the final analysis of these practice questions from all 3 major examination types, we seek to conduct a comparative analysis to other treatment modalities. This work represents the first attempt, to our knowledge, to characterize RT topic appearance within the NBME published practice questions and their accompanying explanations for the USMLE Step 1, Step 2CK, and select NBME shelf examinations. Our analysis is important in assessing the presence of RT topics on medical student board preparation material, a potential avenue of RT content exposure in UGME. Careful review and comparative analyses of all oncology-related content can provide important insight for UGME-related RT education. Our anticipated analyses and future discussion may identify areas for improvements in the important education of oncology-related topics in UGME.

Prior Enlisted Medical Students at the Uniformed Services University: Outcomes During Four Years of Medical School

The Uniformed Services University (USU) implemented the Enlisted to Medical Degree Preparatory Program (EMDP2) with the goal of enhancing the diversity of the military physician corps. Programs like EMDP2 can assist students in making the social and intellectual transition from undergraduate studies to medical school and beyond. These types of programs are also opportunities to reduce health disparities and prepare students to work in multicultural settings. The purpose of this study was to evaluate whether there was any significant difference in performance between USU medical students who had attended the EMDP2 and those who had not.We compared the results of National Board of Medical Examiners (NBME) Clinical Science Subjects, United States Medical Licensing Examination (USMLE) Step 1, and USMLE Step 2 Clinical Knowledge exams of EMDP2 learners from the School of Medicine classes of 2020 to 2023 to those of four similarly sized cohorts of their peers who varied by age and prior military service.We found that the performance of EMDP2 graduates was comparable to their peers who followed more traditional and other alternative paths to medical school. For example, regression models showed that EMDP2 status was not a statistically significant predictor of average clerkship NBME exam score, nor of USMLE Step 1 failure.EMDP2 graduates performed on a par with their medical school peers, and EMDP2 status does not appear to influence NBME or USMLE performance. EMDP2 provides a focused curriculum and addresses the mandate to make medical education opportunities available to a more diverse population.

Assessing multiple-choice questions based on language precision and best practices to promote equity in the Dental Physiology course

Student assessment in professional schools is conducted mainly through examinations based on multiple-choice questions (MCQs). Although grading this type of assessments saves time, questions that are not well-written may impact students’ performance. Technical flaws in MCQs include those that provide irrelevant difficulty and those that provide an advantage to test-wise examinees. In addition, MCQs with these flaws may disproportionately affect students with weaker undergraduate science backgrounds and those from underrepresented groups including English Language Learners and first-generation college students. Inclusive teaching practices aim to create a level field by removing barriers and providing equal access to students regardless of their background.It is hypothesized that technical flaws in MCQs increase their difficulty. The objectives of this study are: 1) to rate the quality of the MCQs used in a Dental Physiology course at Boston University and 2) to examine the effect of questions with technical flaws on item performance. To measure the performance of specific items, two analyses will be conducted: item difficulty, defined as the percentage of students who choose an item correctly, and item discrimination which refers to the correlation of how well a test taker does on a particular item and their performance on the whole test.An evaluation instrument based on the one developed and validated by Breakall et al. (2019) was employed to identify item writing flaws that add irrelevant difficulty. Examples of item flaws that provide irrelevant difficulty based on the National Board of Medical Examiners (NBME) guidelines are: (a) items with complicated stems and lead-ins that include negative forms, (b) item options that are not written succinctly or include vague terms, (c) numerical data not presented consistently, (d) items that include nonparallel options, or (e) that include “none of the above.” This instrument was used to assess MCQs from a Dental School Physiology exam.The frequency of item flaws indicated that of all items analyzed, 56% items contained at least one flaw. The most common item flaws identified were those where the answer choices were not of approximately the same length (32%), did not have parallel grammatical form and structure (24%) or those that included negative phrasing (16%). In conclusion, this assessment indicates that the MCQs used in this Dental Physiology course have room for improvement. To better understand if the identified flaws affect item performance, exam data provided by Exam Soft Analytics will be analyzed. Based on those results, decisions could be made about modifying MCQs to better serve the needs of our diverse student population.

A Pilot Study Assessment of Medical Student Knowledge and System Citizenship Attitudes Pertaining to Health Systems Science.

Health system science (HSS) has been described as the third pillar of medical education. We introduced a new health system science and interprofessional practice (HSSIP) curriculum, and measured students' HSS knowledge and attitudes concerning health system citizenship. This pilot study involved first-year (M1) and fourth-year (M4) medical students in two cohorts across 2 years. Only M1 students in the second cohort participated in the new HSSIP curriculum. We compared student performance on a new National Board of Medical Examiners (NBME) HSS subject exam, and student attitudes toward system citizenship via a new attitudinal survey. Fifty-six eligible fourth-year students (68%) and 70 (76%) study eligible first-year students participated in the study. NBME HSS exam performance by M4 students was statistically significantly higher than M1 students for both cohorts, with moderate to large effect sizes. Exam performance for M1 students not experiencing the HSS curriculum was higher than for M1 students who received HSS curricular content. Attitudes toward HSS by M4 versus M1 students were statistically significantly different on several survey items with moderate effect sizes. Scale internal consistency for the HSS attitude survey was strong (0.83 or higher). There were differences among M4 and M1 medical students concerning knowledge of and attitudes toward HSS, with performance on the NBME subject exam similar to a national sample. Exam performance by M1 students was likely impacted by class size and other factors. Our results support the need for increased attention to HSS during medical education. Our health system citizenship survey has potential for further development and cross-institutional collaboration.

Investigating the Impact of Virtual Versus In-Person Small Group Discussion on Academic Performance in a First-Year Medical Neuroscience Course (S10.003)

<h3>Objective:</h3> We aimed to determine if academic performance was different in students who self-selected to participate in virtual small group discussions, compared to students who engaged in traditional in-person small group discussions, in a first-year medical school neuroscience course. <h3>Background:</h3> Incorporating virtual instruction into the medical curriculum has been a longstanding subject of study and has increasingly been required due to the COVID-19 pandemic. Previous literature suggests that exchange and communication patterns during learning may be stunted on virtual platforms. In 2021, for the pre-clinical neuroscience course, the Mayo Clinic Alix School of Medicine gave students the choice to engage in small group sessions either in-person or virtually. These sessions have been used to facilitate case discussions to reinforce course material. <h3>Design/Methods:</h3> This is a retrospective analysis of academic performance in the 2021 neuroscience course. Students self-selected to participate either in-person or virtually for small group discussions. Academic performance was measured by scores on the National Board of Medical Examiners (NBME) shelf exam and professor-written final exam, as well as weighted final course grades. Unpaired t-tests were used to compare data from the two cohorts. <h3>Results:</h3> Of the 51 students, 23 chose the in-person cohort and 27 chose the virtual cohort. One student switched cohorts part way through the course and was excluded from analysis. Overall, when comparing the in-person and virtual cohorts, there was no difference in scores for the NBME shelf exam (87.13% vs 85.33%, p = 0.48), professor-written final exam (87.13% vs 86.22%, p = 0.55), or weighted course grades (87.56% vs 87.09%, p = 0.71). <h3>Conclusions:</h3> Students who self-selected to engage in virtual discussions of neuroscience course material performed comparably to their in-person counterparts. This suggests that virtual platforms may be appropriate for holding small group discussions within the medical curriculum. <b>Disclosure:</b> Dr. Vanood has nothing to disclose. Dr. Chhabra has nothing to disclose. Dr. Hoerth has a non-compensated relationship as a Epilepsy.com Editorial Board member with the Epilepsy Foundation of America that is relevant to AAN interests or activities.

Neurology Clerkship Shelf Score Analysis Pre- and Post-Satisfactory/Unsatisfactory Grading System (P4-2.002)

<h3>Objective:</h3> This study examines the effects of changing the clerkship grading system at our institution from a traditional tiered scale to Satisfactory/Unsatisfactory grading. Scores on the National Board of Medical Examiners (NMBE) subject examination in Clinical Neurology are an objective measure that could reflect how changes in the grading system may impact learning. <h3>Background:</h3> Medical schools are continually undergoing innovative changes for curriculum improvement including revision of grading systems. In an attempt to reduce student anxiety and stress, many academic institutions have transitioned from traditional tiered grading to a satisfactory/unsatisfactory grading system. It is unclear what effect this change has had on learning and academic outcomes, specifically, end of clerkship national examinations. <h3>Design/Methods:</h3> A retrospective analysis was performed on available data two years prior to implementation of the Satisfactory/Unsatisfactory grading system from 2018 to 2020 and two years after from 2020 to 2022. The academic outcomes assessed were first-attempt fail rate, one standard deviation below equated percent correct national means and mean NMBE Clinical Neurology scores per academic year. <h3>Results:</h3> The first-attempt fail rate prior to implementation of the satisfactory/unsatisfactory evaluation system was zero, but afterwards the first-attempt fail rate increased to 2.4% in the 2021–2022 academic year. The one standard deviation below equated percent correct national mean score more than doubled between 2018–2019 and 2021–2022 academic years. Full results for mean NMBE score will be available once data collection is complete. <h3>Conclusions:</h3> Academic outcomes as measured by shelf exam scores appear to have been adversely affected following adoption of a satisfactory/unsatisfactory grading system. Future analysis will include United States Medical Licensing Examination board certification results to obtain additional objective data. <b>Disclosure:</b> Dr. Mayberry has nothing to disclose. Dr. Juel has received personal compensation in the range of $500-$4,999 for serving on a Scientific Advisory or Data Safety Monitoring board for Immunovant. Dr. Juel has received personal compensation in the range of $500-$4,999 for serving on a Scientific Advisory or Data Safety Monitoring board for Alexion. Dr. Juel has received personal compensation in the range of $0-$499 for serving on a Scientific Advisory or Data Safety Monitoring board for Accordant Health Services. The institution of Dr. Juel has received research support from argenx. The institution of Dr. Juel has received research support from Alexion. The institution of Dr. Juel has received research support from Janssen. The institution of Dr. Juel has received research support from NIH Rare Diseases Network. Dr. Gable has received personal compensation in the range of $500-$4,999 for serving as a Consultant for Medscape, Web MD. Dr. Gable has received personal compensation in the range of $500-$4,999 for serving as a Consultant for Pharmacy Times. Dr. Gable has received personal compensation in the range of $500-$4,999 for serving as a Consultant for Cowen. Dr. Gable has received personal compensation in the range of $500-$4,999 for serving on a Scientific Advisory or Data Safety Monitoring board for Argenx. Dr. Gable has received personal compensation in the range of $500-$4,999 for serving on a Scientific Advisory or Data Safety Monitoring board for Ultragenx. Dr. Gable has received personal compensation in the range of $5,000-$9,999 for serving as a Lecture series with Medscape. The institution of Dr. Lutz has received research support from NIH. Mrs. Berry has nothing to disclose.

Student with disabilities eligible for reasonable modifications from NBME

A New York federal court granted a preliminary injunction to a gifted student with learning disabilities and attention deficit hyperactivity disorder, holding that the National Board of Medical Examiners (NBME) had to provide the student with double time for the Step 1 United States Medical Licensing Examination (USMLE). The court rejected NBME's argument that the student was not an individual with a disability due to his academic success. Sampson v. Nat'l Bd. of Med. Exam'rs, 22‐CV‐05120 (E.D.N.Y. 2022).

The Early Impact of Deciding to Take the United States Medical Licensing Examination Step 1 for Osteopathic Medical Students in the Pass/Fail Era.

The United States Medical Licensing Examination (USMLE) Step 1 recently shifted from a three-digit scoring format to a pass/fail scoring rubric. Lake Erie College of Osteopathic Medicine (LECOM) is among a number of osteopathic medical schools that traditionally included passing Step 1 as a graduation requirement. However, LECOM removed this requirement following the change in scoring format. National Board of Medical Examiners (NBME) subject examinations have a significant impact on the clerkship grades of third-year medical students. Therefore, our pilot study compared NBME subject examination scores among third-year LECOM medical students that did and did not take and pass Step 1. We anticipate that both high pre-clinical grade point average (GPA) and having passed Step 1 will be associated with higher subject exam scores, but passing Step 1 will have a relationship with higher subject exam scores that is independent of pre-clinical GPA. Using voluntary response sampling, 201 osteopathic medical students from LECOM completed an online survey through Google Forms regarding their pre-clinical GPA, subject exam scores, whether they took and passed USMLE Step 1, and their study resources used throughout clerkships. Results: There was a positive correlation (p < 0.05) found between pre-clinical GPA and exam scores across all subjects among students that took Step 1. There was no relationship between pre-clinical GPA andexam scores across all subjects among students that did not take Step 1 (p> 0.05). Students that took Step 1 had a higher pre-clinical GPA than those that did not. Students that took and passed Step 1 scored higher on subject exams. Fifty-nine percent of respondents indicated they would have studied more for Step 1 if these exams were scored on the three-digit format, while zero respondents indicated they would have studied less. Although higher pre-clinical GPA and taking Step 1 were associated with higher scores on subject exams, taking Step 1 appears to have an independent influence on subject exams because there was no relationship found between pre-clinical GPA and subject exam scores among students that did not take Step 1. Therefore, there may be features related to preparing for this exam that better equip osteopathic medical students to perform well on subject exams.

Medical Education During COVID-19 Pandemic: A Comparative Effectiveness Study of Face-to-Face Traditional Learning Versus Online Digital Education of Basic Sciences for Medical Students.

During the coronavirus disease 2019 (COVID-19) pandemic, internet-based learning modalities and online classes became a tool for the continuous learning process for medical students. The aim of this study was to compare medical student performance in both online versus offline instructional methods. The study was conducted on 213 medical students of the basic science program at the American University of Antigua, College of Medicine (AUACOM), who completed the four semesters consecutively between Spring 2018 and Fall 2020. Two cohorts of students were considered in the study: cohort 1 (those who completed years 1 and 2 using traditional offline teaching modality) and cohort 2 (those who completed year 1 offline and year 2 online). The years 1 and 2 National Board of Medical Examiners (NBME) summative assessment scores of the students were used to determine which instructional modality generated better student performance outcomes for the two groups. Additionally, we evaluated the score variabilities between genders to determine if teaching modality had an impact on a specific group. All statistical comparisons were done using two-tailed t-tests. The study involves 213 students (112 in cohort 1, 101 in cohort 2). There was no significant difference in student performance between offline and online learners overall (74 ± 2.3vs. 73 ±1.3; p = 0.537) or with respect to gender (73 ±3.8 vs. 73 ±3.0; p= 0.709). In this comparative effectiveness study of traditional offline education versus online instructional modality, we observed no statistical difference in student performance evaluated with NBME summative assessment scores. Online classes were well-accepted by our students. These data show a significant and promising potential for the future of medical education using online teaching modalities. Remote online teaching could be used again in the future without detriment to student education if face-to-face (F2F) learning is not possible.

O.1.5 - Similarities between clinically matched and unmatched analogue patient raters: a mixed methods study: Presenter(s): Angelo D'Addario, National Board of Medical Examiners, PA, United States

O.1.5 - Similarities between clinically matched and unmatched analogue patient raters: a mixed methods study: Presenter(s): Angelo D'Addario, National Board of Medical Examiners, PA, United States

How Does ChatGPT Perform on the United States Medical Licensing Examination (USMLE)? The Implications of Large Language Models for Medical Education and Knowledge Assessment.

Chat Generative Pre-trained Transformer (ChatGPT) is a 175-billion-parameter natural language processing model that can generate conversation-style responses to user input. This study aimed to evaluate the performance of ChatGPT on questions within the scope of the United States Medical Licensing Examination (USMLE) Step 1 and Step 2 exams, as well as to analyze responses for user interpretability. We used 2 sets of multiple-choice questions to evaluate ChatGPT's performance, each with questions pertaining to Step 1 and Step 2. The first set was derived from AMBOSS, a commonly used question bank for medical students, which also provides statistics on question difficulty and the performance on an exam relative to the user base. The second set was the National Board of Medical Examiners (NBME) free 120 questions. ChatGPT's performance was compared to 2 other large language models, GPT-3 and InstructGPT. The text output of each ChatGPT response was evaluated across 3 qualitative metrics: logical justification of the answer selected, presence of information internal to the question, and presence of information external to the question. Of the 4 data sets, AMBOSS-Step1, AMBOSS-Step2, NBME-Free-Step1, and NBME-Free-Step2, ChatGPT achieved accuracies of 44% (44/100), 42% (42/100), 64.4% (56/87), and 57.8% (59/102), respectively. ChatGPT outperformed InstructGPT by 8.15% on average across all data sets, and GPT-3 performed similarly to random chance. The model demonstrated a significant decrease in performance as question difficulty increased (P=.01) within the AMBOSS-Step1 data set. We found that logical justification for ChatGPT's answer selection was present in 100% of outputs of the NBME data sets. Internal information to the question was present in 96.8% (183/189) of all questions. The presence of information external to the question was 44.5% and 27% lower for incorrect answers relative to correct answers on the NBME-Free-Step1 (P<.001) and NBME-Free-Step2 (P=.001) data sets, respectively. ChatGPT marks a significant improvement in natural language processing models on the tasks of medical question answering. By performing at a greater than 60% threshold on the NBME-Free-Step-1 data set, we show that the model achieves the equivalent of a passing score for a third-year medical student. Additionally, we highlight ChatGPT's capacity to provide logic and informational context across the majority of answers. These facts taken together make a compelling case for the potential applications of ChatGPT as an interactive medical education tool to support learning.

Perceptions of the Emergency Medicine Resident Selection Process by Program Directors Following the Transition to a Pass/Fail USMLE Step 1.

Beginning January 26th, 2022, the National Board of Medical Examiners transitioned scoring of the United States Medical Licensing Examination (USMLE) Step 1 from a 3-digit score to pass/fail. In the past, the Step 1 score has been weighted heavily by program directors (PDs) as one of the most important metrics when assessing medical student's competitiveness. The objective of this study was to evaluate the perceptions of emergency medicine (EM) PDs on the transition to a pass/fail USMLE Step 1 exam, and to elicit the opinions of EM PDs on the USMLE examinations' ability to predict resident performance. A survey consisting of ranking and multiple-choice questions was sent to EM PDs. The multiple-choice questions were asked to determine EM PDs level of confidence in the ability of Step 1 and Step 2 Clinical Knowledge (CK) to predict a student's ability to succeed in residency. The ranking questions focused on assessing each program's current resident selection practices in comparison to expected selection criteria changes following a transition to pass/fail Step 1. R studio and MATLAB were used for statistical analysis, and a P value <0.05 was considered significant. The survey was completed by 57 (20.21%) EM PDs. When asked if Step 1 and Step 2 CK are accurate predictors of a resident's ability to perform clinically within EM, only 10.5% of PDs answered 'yes' to Step 1 being predictive, compared to 31.6% for Step 2 CK. Regarding selection criteria, the top quartile of attributes (standardized letters of evaluation [1st], away rotations [2nd], clerkship grades [3rd] and Step 2 CK score [4th]) remained the same following the transition. Our results indicate that the top quartile of attributes might remain the same, despite most PDs agreeing that Step 2 CK is a better predictor of a resident's performance.

Is the Comprehensive Basic Science Examination Still Useful in the Oral and Maxillofacial Surgery Residency Application Process?

For any dental student interested in pursuing a career in oral and maxillofacial surgery (OMS), perhaps the biggest barrier to overcome is passing the Comprehensive Basic Science Examination (CBSE), a 200-question multiple choice test written by the National Board of Medical Examiners designed to prepare medical students for the United States Medical Licensing Examination (USMLE) Step 1.1 Prior to having the CBSE as a requirement for OMS residency applications, board scores from National Board Dental Examination (NBDE) were considered as a benchmark to compare applicants across all dental schools.

Time Does Matter: Reduced Internal Medicine Clerkship Clinical Experiences Due to COVID-19.

Introduction The coronavirus disease 2019 (COVID-19) pandemic has affected medical education in many ways.The Association of American Medical Colleges (AAMC) temporarily suspended clinical student rotations, calling for a transition to remote learning. Unfortunately, due to the heavy impact of COVID-19 in our South Florida community, medical students were not able to return to in-person activities for a significant time.During this period,students hadremote clerkship learning activities, didactic sessions, narrative projects, and small-group learning sessions, which were front-loaded using Zoom technology (Zoom Video Communications, Inc.,San Jose, California, United States) and web-based learning tools. Once in-person clinical experiences resumed, the duration of all third-year clerkships for the remainder of the year wasreduced to five weeks to allow for timely graduation.The Herbert Wertheim College of Medicine (HWCOM) Internal Medicine (IM) clerkshiphas traditionally beenan eight-week-long rotation.Other clerkships that varied from six to eight weeks were similarly reduced to five weeks.We hypothesized that the shortened duration of the IM clerkship would have negative impacts on National Board of Medical Examiners (NBME) exam performance and clerkship clinical experiences would likely be affected. Methods We compared the NBME subject exam results and end of clerkship evaluations from the Class of 2021 (CO2021) which had the traditional eight weeks of patient care, with the CO2022, which had only five weeks of in-person patient care.A T-test analysis was performed comparing performance on the NBME medicine clinical subject exam between students who completed the usual eight-week rotation versus those who completed a five-week rotation.We also evaluated the IM clerkship course evaluation and analyzed student responses and ratings to assess any areas that were statistically significant when comparing the traditional eight-week IM clerkship to the shortened five-week clerkship. Results There was no statistically significant difference (t=0.68,p<0.4951) in mean NBME subject exam performance between cohorts. Students who completed the shortened five-week IM clerkship indicated there waslimited volume and diversity of patients, which consequently affected their ability to complete all the required clinical experiences for the IM clerkship.These results indicated a statistically significant difference between the two cohorts (t =3.33, p<.001). Conclusion Students with shortened IM clerkship clinical care time (five weeks) were found to have no significant statistical differences in NBME subject exam performance compared to the traditional eight-week cohorts. However, students felt there was a decreased volume and diversity of patients, and they reported greater difficulties in completing the required clinical experiences, with diminished clinical confidence. Time does matter, and clinical time is very valuable for a student's undergraduate medical education.If another pandemic were to arise, the durationof differentclerkships should be carefully assessed and individualized, and methods to assess and reclaim lost clinical time during the advanced clinical and postgraduate years should be considered.

Medical student exam performance and perceptions of a COVID-19 pandemic-appropriate pre-clerkship medical physiology and pathophysiology curriculum

BackgroundMedical schools were compelled to abruptly transition pre-clerkship curricula to remote learning formats due to the emergence of the Coronavirus Disease 2019 (COVID-19) pandemic. We evaluated student perceptions of remote learning, exam performance, and utilization of third-party learning resources to assess the implementation of a newly developed pandemic-appropriate physiology curriculum.MethodsThis was an observational study based on a survey conducted in the Spring of 2021 at the University of California, Irvine, School of Medicine (UCISOM). This study aimed to assess first (MS1) and second year (MS2) medical students’ perceptions of satisfaction, support, academic performance, and connectedness before and during the COVID-19 pandemic. The MS1 class began medical school during the first year of the COVID-19 pandemic, whereas the MS2 class did so prior to the start of the pandemic. A survey instrument was developed and validated to identify the impact remote learning had on student self-perceptions of the Medical Physiology and Pathophysiology course. Surveys were distributed to all students and responses were collected on a voluntary basis. Exam scores on a customized National Board of Medical Examiners (NBME) physiology shelf exam were also compared to objectively identify how the remote curriculum during the pandemic impacted academic performance.ResultsOf 204 students enrolled, 74 responses were analyzed, with 42 MS1 (40% of MS1s) and 32 MS2 (31% of MS2s) responses. Overall, MS1s and MS2s were satisfied with the curriculum they received (95 and 97% respectively) and the school’s support of their concerns (86 and 100% respectively). Notably, only 50% of MS1s felt connected to their peers, compared to 94% of MS2s. Lecture attendance and self-perception of their academic performance were similar between both classes. Interestingly, the intra-pandemic class’s NBME exam average in 2020 (60.2% ± 8.9, n = 104) was significantly higher than the pre-pandemic class average in 2019 (56.8% ± 11.3, n = 100). Both classes primarily used course materials over third-party learning resources. An additional set of survey questions distributed only to the MS1 class found that the majority of MS1s reported minimal barriers with regards to accessibility, including internet connectivity, study-conducive environments, and balancing family commitments. Overall, pre-clerkship medical students had positive perceptions of the newly developed pandemic-appropriate physiology curriculum.ConclusionsChanges to the pre-clerkship physiology curriculum during the COVID-19 pandemic were met with overall satisfaction from the students and an increase in NBME scores. More attention to student connectedness is needed to improve how remote learning can be best optimized into future curricula development.