Written Board Examination Research Articles

Assessment of ChatGPT’s performance on neurology written board examination questions

Background and objectivesChatGPT has shown promise in healthcare. To assess the utility of this novel tool in healthcare education, we evaluated ChatGPT’s performance in answering neurology board exam questions.MethodsNeurology board-style...

AI did not pass Finnish plastic surgery written board examination

AI did not pass Finnish plastic surgery written board examination

Performance of ChatGPT and GPT-4 on Neurosurgery Written Board Examinations.

Interest surrounding generative large language models (LLMs) has rapidly grown. Although ChatGPT (GPT-3.5), a general LLM, has shown near-passing performance on medical student board examinations, the performance of ChatGPT or its successor GPT-4 on specialized examinations and the factors affecting accuracy remain unclear. This study aims to assess the performance of ChatGPT and GPT-4 on a 500-question mock neurosurgical written board examination. The Self-Assessment Neurosurgery Examinations (SANS) American Board of Neurological Surgery Self-Assessment Examination 1 was used to evaluate ChatGPT and GPT-4. Questions were in single best answer, multiple-choice format. χ 2 , Fisher exact, and univariable logistic regression tests were used to assess performance differences in relation to question characteristics. ChatGPT (GPT-3.5) and GPT-4 achieved scores of 73.4% (95% CI: 69.3%-77.2%) and 83.4% (95% CI: 79.8%-86.5%), respectively, relative to the user average of 72.8% (95% CI: 68.6%-76.6%). Both LLMs exceeded last year's passing threshold of 69%. Although scores between ChatGPT and question bank users were equivalent ( P = .963), GPT-4 outperformed both (both P < .001). GPT-4 answered every question answered correctly by ChatGPT and 37.6% (50/133) of remaining incorrect questions correctly. Among 12 question categories, GPT-4 significantly outperformed users in each but performed comparably with ChatGPT in 3 (functional, other general, and spine) and outperformed both users and ChatGPT for tumor questions. Increased word count (odds ratio = 0.89 of answering a question correctly per +10 words) and higher-order problem-solving (odds ratio = 0.40, P = .009) were associated with lower accuracy for ChatGPT, but not for GPT-4 (both P > .005). Multimodal input was not available at the time of this study; hence, on questions with image content, ChatGPT and GPT-4 answered 49.5% and 56.8% of questions correctly based on contextual context clues alone. LLMs achieved passing scores on a mock 500-question neurosurgical written board examination, with GPT-4 significantly outperforming ChatGPT.

Can Artificial Intelligence Pass the American Board of Orthopaedic Surgery Examination? Orthopaedic Residents Versus ChatGPT.

Advances in neural networks, deep learning, and artificial intelligence (AI) have progressed recently. Previous deep learning AI has been structured around domain-specific areas that are trained on dataset-specific areas of interest that yield high accuracy and precision. A new AI model using large language models (LLM) and nonspecific domain areas, ChatGPT (OpenAI), has gained attention. Although AI has demonstrated proficiency in managing vast amounts of data, implementation of that knowledge remains a challenge. (1) What percentage of Orthopaedic In-Training Examination questions can a generative, pretrained transformer chatbot (ChatGPT) answer correctly? (2) How does that percentage compare with results achieved by orthopaedic residents of different levels, and if scoring lower than the 10th percentile relative to 5th-year residents is likely to correspond to a failing American Board of Orthopaedic Surgery score, is this LLM likely to pass the orthopaedic surgery written boards? (3) Does increasing question taxonomy affect the LLM's ability to select the correct answer choices? This study randomly selected 400 of 3840 publicly available questions based on the Orthopaedic In-Training Examination and compared the mean score with that of residents who took the test over a 5-year period. Questions with figures, diagrams, or charts were excluded, including five questions the LLM could not provide an answer for, resulting in 207 questions administered with raw score recorded. The LLM's answer results were compared with the Orthopaedic In-Training Examination ranking of orthopaedic surgery residents. Based on the findings of an earlier study, a pass-fail cutoff was set at the 10th percentile. Questions answered were then categorized based on the Buckwalter taxonomy of recall, which deals with increasingly complex levels of interpretation and application of knowledge; comparison was made of the LLM's performance across taxonomic levels and was analyzed using a chi-square test. ChatGPT selected the correct answer 47% (97 of 207) of the time, and 53% (110 of 207) of the time it answered incorrectly. Based on prior Orthopaedic In-Training Examination testing, the LLM scored in the 40th percentile for postgraduate year (PGY) 1s, the eighth percentile for PGY2s, and the first percentile for PGY3s, PGY4s, and PGY5s; based on the latter finding (and using a predefined cutoff of the 10th percentile of PGY5s as the threshold for a passing score), it seems unlikely that the LLM would pass the written board examination. The LLM's performance decreased as question taxonomy level increased (it answered 54% [54 of 101] of Tax 1 questions correctly, 51% [18 of 35] of Tax 2 questions correctly, and 34% [24 of 71] of Tax 3 questions correctly; p = 0.034). Although this general-domain LLM has a low likelihood of passing the orthopaedic surgery board examination, testing performance and knowledge are comparable to that of a first-year orthopaedic surgery resident. The LLM's ability to provide accurate answers declines with increasing question taxonomy and complexity, indicating a deficiency in implementing knowledge. Current AI appears to perform better at knowledge and interpretation-based inquires, and based on this study and other areas of opportunity, it may become an additional tool for orthopaedic learning and education.

Teaching quality in neurosurgery: quantitating outcomes over time.

High-quality neurosurgery resident training is essential to developing competent neurosurgeons. Validated formative tools to assess faculty teaching performance exist, but are not used widely among Accreditation Council for Graduate Medical Education (ACGME) residency programs in the United States. Furthermore, their longer-term impact on teaching performance improvement and educational outcomes remains unclear. The goal of this study was to assess the impact of implementing an evaluation system to provide faculty with feedback on teaching performance in a neurosurgery residency training program over a 4-year period. The authors performed a prospective cohort study in which a modified version of the System for Evaluation of Teaching Qualities (SETQ) instrument was administered to neurosurgical trainees in their department regularly every 6 months. The authors analyzed subscale score dynamics to identify the strongest correlates of faculty teaching performance improvement. ACGME program survey results and trainee performance on written board examinations were compared for the 3 years before and after SETQ implementation. The overall response rate among trainees was 91.8%, with 1044 surveys completed for 41 faculty. Performance scores improved progressively from cycle 1 to cycle 6. The strongest correlate of overall performance was providing positive feedback to trainees. Compared to the 3 years prior, the 3 years following SETQ implementation saw significant increases in written board examination and ACGME resident survey scores compared to the national mean. Implementation of SETQ was associated with significant improvements in faculty teaching performance as judged by trainees over a 4-year period, and guided curricular changes in the authors' training program that resulted in improved educational outcomes.

Two-Screen Virtual Board Game Didactic for Teaching Wilderness and Environmental Medicine Topics to Emergency Medicine Residents.

This game is appropriate for medical students, interns, junior and senior residents. The COVID-19 pandemic has forced a transition from in-person to virtual learning, and educators must innovate and adapt to keep learners engaged. One way to achieve this is through gamification.1 The authors employed a novel approach to gamification of virtual learning which engaged not only learners' computers but also their mobile phones. Learners worked in teams communicating by text message to answer ABEM board-style questions and occupy sites on the virtual board. By the end of this didactic, the learner will:1. Describe the basics of the presentation of each topic listed above2. Recall the basics of management of each topic listed above3. Improve learners' preparedness for the Emergency Medicine Inservice Exam and Written Board Examination. Wilderness and environmental medicine topics were selected from the list of topics covered on the ABEM boards. Questions were then written by the authors teaching these concepts. Learners were surveyed immediately following the session using an evaluation tool containing both Likert-scale questions on quality and applicability as well as open-ended questions on strengths and areas for improvement. The response rate to this survey was 100%. Feedback was overwhelmingly positive, with 19/20 respondents rating the sessions 5/5 for effectiveness and value of teaching compared with other sessions, and 1/20 rating the session 4/5. Nineteen out of twenty respondents rated the content as "highly relevant"; 1/20 rated it as "mostly relevant." Nineteen out of twenty respondents rated the session 5/5 for being engaging and holding their attention; 1/20 rated it as somewhat engaging. Learners rated the session as highly relevant and engaging. We hypothesize that by engaging two screens and forcing learners to work together by text message, we were able to turn what would normally be a distraction (texting co-residents during resident conference) into a tool for learning. Electrical injury, lightning strike, thermal burns, inhalational injury, chemical burns, acute radiation syndrome, snake bites, scorpion envenomation, stingray envenomation, jellyfish stings, black widow spider bites, mammalian bites, rabies, murine typhus, bear encounters, decompression sickness, arterial gas embolism, drowning, dehydration, heat stroke, exercise-associated hyponatremia, frostbite, hypothermia, CO poisoning, hydrogen sulfide poisoning, giardia.

Correlation Between Neurosurgical Residency Written Board Scores and Case Logs

There are few objective measures for evaluating individual performance throughout surgical residency. Two commonly used objective measures are the case log numbers and written board examination scores. The objective of this study was to investigate possible correlations between these measures. We conducted a retrospective review of the American Board of Neurological Surgery (ABNS) written board scores and the Accreditation Council for Graduate Medical Education case logs of 27 recent alumni from neurologic surgery residency training programs at The Ohio State Wexner Medical Center and the University of Nebraska Medical Center. The number of spine cases logged was significantly correlated with the ABNS written examination performance in univariate linear regression (r2= 0.182, P= 0.0265). However, case numbers from all other neurosurgical subspecialties did not significantly correlate with ABNS written board performance (P > 0.1). Identifying which objective measures correlate most closely with resident education could help optimize the structure of residency training programs. We believe that early exposure to focused aspects of neurosurgery helps the young resident learn quickly and efficiently and ultimately score highly on standardized examinations. Therefore program directors may want to ensure focused exposure during the early years of residency, with particular attention to worthwhile rotations in spine neurosurgery.

Administering oral and written board examinations remotely during the COVID-19 pandemic.

Administering oral and written board examinations remotely during the COVID-19 pandemic.

Dental Topics for Plastic Surgeons, Part Four: Common Cysts and Tumors of the Jaw.

Conditions that affect dental and periodontal structures receive sparse coverage in the plastic surgery literature, yet a working knowledge of this subject matter is important in certain areas of clinical practice and a fundamental understanding is part of plastic surgery competency tested on the in-service and written board examinations. This 4-part series was written to provide plastic surgeons with a working knowledge of dental topics that may be relevant to their clinical practice. This section, Part IV, covers common dental cyst and tumors of the jaw.

Dental Topics for Plastic Surgeons, Part Three: Infection and Trauma of the Dentition.

Conditions that affect dental and periodontal structures receive sparse coverage in the plastic surgery literature, yet a working knowledge of this subject matter is important in certain areas of clinical practice and a fundamental understanding is part of plastic surgery competency tested on the in-service and written board examinations. This 4-part series written to provide plastic surgeons with a working knowledge of dental topics that may be relevant to their clinical practice. This section, Part III, covers inflammatory and infectious conditions of the dentition and related structures, as well as dentoalveolar trauma.

Dental Topics for Plastic Surgeons, Part Two: Abnormal Tooth Development, Conditions, and Treatment.

Conditions that affect dental and periodontal structures receive sparse coverage in the plastic surgery literature, yet a working knowledge of this subject matter is important in certain areas of clinical practice and a fundamental understanding is part of plastic surgery competency tested on the in-service and written board examinations. This four-part series written to provide plastic surgeons with a working knowledge of dental topics that may be relevant to their clinical practice. This section, Part II, covers abnormal tooth development and related conditions.

Dental Topics for Plastic Surgeons, Part One: Normal Anatomy, Growth and Development.

Normal and abnormal conditions affecting the dentition and the periodontal structures receive sparse coverage in the plastic surgery literature, textbooks, and training programs. Nevertheless, a working knowledge of this subject matter is important in certain areas of clinical practice, and a fundamental understanding is often part of plastic surgery competency tested for in the In-service and written board examinations. This four-part series is written to provide plastic surgeons with a working knowledge of relevant dental topics. Part 1 covers fundamental aspects of normal dental embryology, growth and anatomy.

Anesthesia Review: 1000 Questions and Answers to Blast the BASICS and Ace the ADVANCED, 1st ed

Anesthesia Review: 1000 Questions and Answers to Blast the BASICS and Ace the ADVANCED is a new edition to a crowded field of anesthesiology board review books. Although it is clearly intended for anesthesiology residents, certified registered nurse anesthetists, and attendings who are studying for written anesthesiology board examinations, it also can provide all practitioners with contemporary insight into current practice paradigms and refresh their knowledge base in the wide range of relevant academic topics. It is divided into the “basic” and “advanced” examinations subject matter categories that one is expected to be proficient to pass the written anesthesia board examinations. The book is the collective effort of 13 residents, 7 fellows, and 9 faculty members at Massachusetts General Hospital with Dr Edward Bittner, being the most senior of the 4 lead editors, all of whom have critical care backgrounds. There are 25 chapters, each with an average of 40 questions that are divided into sections targeted at the basic or advanced examination portions of the written board examination. The soft cover format is highly portable with ample margins to accommodate note-taking. The electronic version was easy to use and provides the user with an interface to select answer choices. Supportive illustrations, particularly in the critical care chapter, are helpful for solidifying key points and concepts. As is often the case with books designed to sharpen the “test-taking-acumen” of readers, the questions are replete with a craftiness that surely provided the residents and instructors who created them with scholastic delight. To that end, many questions are formatted in the inquisitive tone, “Which of the following is the LEAST effective method?” These types of questions are creative to write and often match the wits of the authors to the anxiety of the reader. However, it is important for the reader to understand that this is no longer the format used by the American Board of Anesthesiology written examination. The American Board of Anesthesiology positive format for questions such as the “most likely, or most correct” minimizes the ability to demonstrate test-taking abilities over mastery of concepts. Therefore, the reader should use this opportunity to review all correct and incorrect answers as a study opportunity for mastering key concepts. There are citations to journals and textbook chapters after the questions for further reading and clarification. To highlight key points, the authors incorporate deliberate distractions that are designed to address key examination points. For example, one can only smile at the following annotated question: “When evaluating a 4-month-old for elective circumcision, you are told that the child had breastmilk with formula powder 3.5 hours ago. According to ASA fasting guidelines, how many hours must pass between this feeding and induction of general anesthesia?” Choices are 2, 4, 6, 8, or 12 hours. The correct answer is 6 hours, so thumbs down to those who proudly remember the 4-hour algorithm for breast milk. In summary, the book covers a broad range of topics and fills the need for the reader who has exhausted other question books and plans to use one that is easy to read, affordable, and has a good electronic interface. Franklin Chiao, MD, MScDepartment of AnesthesiologyWestchester Medical CenterArdsley, New York[email protected] Miles Dinner, MDDepartment of AnesthesiologyNew York Presbyterian Medical Center/Weill Cornell MedicineNew York, New York

Anesthesia Review

Anesthesia Review: Blasting the Boards by Berg, Zhao, and Bittner aims at providing a broad-based, but concise, review book for anesthesiologists preparing to take the American Board of Anesthesiology’s (ABA) written board examination. The authors are or at one time were in the anesthesiology department at Massachusetts General Hospital in Boston, MA. The authors’ academic teaching roles are useful as they are able to distill complex topics in anesthesiology down to the most salient points relevant to the written board examination. The text of Anesthesia Review: Blasting the Boards is written in plain and simple language with short sentences, making it both quick to read and easy to maintain focus on the material presented. One such example of this is the chapter devoted to the anesthesia machine and breathing circuits. Normally, this important topic is not necessarily what candidates sitting for the examination look forward to reviewing. However, the authors have condensed this oft dreaded and onerous topic down to just a few short, easy-to-read pages devoid of excessive technical jargon. By communicating the information in this manner, even a topic such as this is easily digestible in this book. This represents a real strength of Anesthesia Review. The scope of topics covered is broad, including everything from the basics of anesthesia delivery systems to aspects of different subspecialty anesthetic practices such as cardiac and pediatrics. The content and outline of the book closely follow the American Board of Anesthesiology Part I Examination blueprint.1 Each chapter contains short practice questions with answers. These are written in the current ABA board examination format and are useful tools to help readers assess their understanding of the material. This book is a very efficient and high-yield review, because it does not require the devotion of excessive effort to overly lengthy explanations or controversial interventions in the information it provides. The information presented in the book appears to be correct and backed up by evidence when definitive claims are made. Like all strong and useful review books, this is an ideal template for an audience that is already well read on the fundamentals of anesthetic practice. However, this text will not substitute for a lack of previous effort for those who do not already possess a good basic fund of knowledge based on other sources (nor is it designed to). We would caution anyone who has not already spent a significant amount of time reading primary textbooks against using this book, or any purely review text, as the sole study tool for their ABA written board examination. Compared with other ABA written board examination review books currently available, this review book is far shorter in length and more concise. Many review books currently offered are several hundred pages long and contain significant amounts of redundant information. For example, in comparison with the popular Faust’s Anesthesiology Review,2 this book is approximately half the length and is specifically a board review book, whereas Faust markets itself as a more general review of anesthesiology.3 This book fills a gap in the current market for less arduous and daunting review books. The vast majority of the anesthesiologists taking the ABA written board examination have already spent significant amounts of time during residency reading various primary textbooks, and this book can be the perfect companion guide to brush up on and assess their knowledge base. After review of this text, we would not be at all surprised to see this book become a “go to” review book for study for the ABA written board examination. Jue T. Wang, MDCraig D. McClain, MD, MPHDepartment of Anesthesiology, Perioperativeand Pain MedicineBoston Children’s Hospital and Harvard Medical SchoolBoston, Massachusetts[email protected]

Relevancy of an In-Service Examination for Core Knowledge Training in a Surgical Subspecialty

To facilitate knowledge acquisition during plastic surgery residency, we analyzed the breast curriculum on the Plastic Surgery In-Service Training Exam (PSITE). Breast-related questions on 6 consecutive PSITEs were analyzed (2008-2013). Topics were categorized by the content outline for the American Board of Plastic Surgery written board examination. Question vignettes were classified by taxonomy and clinical setting. References for correct answer choices were categorized by source and publication lag. A total of 136 breast-related questions were analyzed (136/1174, 12%). Questions tended to appear more in the Breast and Cosmetic (75%) section than the Comprehensive (25%) section (p < 0.001). Most question vignettes were written in a clinical setting (64%, p < 0.001). Question taxonomy was evenly distributed among recall (34%), interpretation (28%), and decision-making (37%, p > 0.05). Only 6% of questions required photographic evaluation. Breast-related topics focused on esthetic problems (35%), traumatic deformities (22%), and tumors (21%). Answer references comprised 293 citations to 63 unique journals published a median of 6 years before PSITE administration. Plastic and Reconstructive Surgery (57%) was the most cited journal (p < 0.001) and Surgery of the Breast: Principles and Art by Spear was the most referenced textbook (22%). The PSITE affords a curriculum that reflects breast-related topics on the American Board of Plastic Surgery written board examination. These data may optimize knowledge acquisition in esthetic and reconstructive breast surgery.

Analysis of an In-Service Examination for Core Pediatric Craniofacial Surgery Knowledge

Little is known about designing an effective residency curriculum for pediatric craniofacial surgery. This study elucidates the pediatric craniofacial curriculum of the Plastic Surgery In-Service Training Examination (PSITE) to facilitate knowledge acquisition during residency. Approximately, 6 consecutive PSITEs were reviewed for pediatric craniofacial questions (2010-2015). Subjects were categorized according to topics on the American Board of Plastic Surgery written board examination. Questions were categorized using an educational taxonomy model. Answer references were categorized by source and publication lag. Of 1174 PSITE questions, 147 tested pediatric craniofacial topics (12.5%). Questions appeared predominately in the Craniomaxillofacial section (83.0%, p < 0.001). The annual representation was stable more than 6 years (range: 10.2%-14.4%, p = 0.842). Question taxonomy favored interpretation (45.6%) and decision-making (40.8%) over recall (13.6%, p < 0.001) skills, and 41 questions had an associated image (27.9%) and most were photographic (76.7%, p < 0.001). The most frequently tested categories on the American Board of Plastic Surgery written examination content outline were craniofacial anomalies (23.5%), benign and malignant tumors (17.6%), and cleft lip and palate (12.5%). Overall, 80 unique journals were cited 304 times with a mean publication lag of 9.4 ± 10.9 years. Plastic and Reconstructive Surgery (34.5%) was the most cited journal (p < 0.001). These data may assist in designating core knowledge competency in pediatric craniofacial surgery for plastic surgery residents. A further understanding of PSITE utility for core knowledge competency in pediatric craniofacial surgery would be the focus of future work.

Plastic Surgery Review: A Study Guide for the In-Service, Written Board, and Maintenance of Certification Exams

Gregory E. Lakin, ed. Plastic Surgery Review: A Study Guide for the In-Service, Written Board, and Maintenance of Certification Exams . Paperback: 235 pages. Publisher: Thieme (December 30, 2014). Language: English; ISBN-10: 1604068361, ISBN-13: 978-1604068368; Price: $69.99. ![Graphic][1] As its title clearly states, this text is intended to be a succinct review of testable content for the plastic surgery In-Service, Written Board, and Maintenance of Certification examinations. It is intended for surgical residents, freshly trained plastic surgeons preparing for their written board examinations, and those approaching a 10-year maintenance of certification examination interval. The content is subdivided into 4 sections based on the In-Service examination: (1) Comprehensive Plastic Surgery, (2) Craniomaxillofacial, (3) Breast and Cosmetic Surgery, and (4) Hand and Lower Extremity Surgery. Altogether, there are 50 chapters to cover the breadth of plastic surgery in a “point form” manner. Each chapter is reviewed by an expert in the field to confirm that it is up-to-date and accurate. Pertinent references included at the end of each chapter permit further in-depth review as required. This text delivers on its goal of providing “high-yield” content in a simultaneously concise and comprehensive manner. The author list is impressive and diverse with “giants” in their respective … Corresponding Author: Dr Terence M. Myckatyn, Washington University School of Medicine, 1040 N. Mason, Suite 124, Saint Louis, MO 63141, USA. E-mail: myckatyn{at}wudosis.wustl.edu [1]: /embed/inline-graphic-1.gif

323 Can In-Training Exam Review Quizzes Administered via Audience Response System Predict Scores on In-Training Exam?

In preparation for the American Board of Emergency Medicine (ABEM) written board examination (Qualifying Examination) residents take an annual in-training exam. All emergency medicine (EM) residencies strive for a 100% pass rate of this examination. However, each residency has its own training strengths and weaknesses (eg, strong educational sessions in toxicology or high clinical exposure to trauma). To improve in-training exam scores, many residency programs provide review sessions. These serve to improve performance. One method of review is board-style questions with an audience response system which has been shown to be an effective tool for knowledge acquisition and assessment in other settings. We investigate if scores on review quizzes delivered by an audience response system correlate with in-training exam scores. Prospective observational study of EM residents at 6 ACGME-accredited EM residency programs. Subjects included residents who had taken previous in-training examinations. Subjects participated in bimonthly review sessions using Rosh Review questions and an audience response system. Twelve review quizzes were administered each consisting of 10 multiple choice questions covering a major topic of the EM in-training exam: cardiology, gastroenterology, neurology, toxicology, signs and symptoms, procedures, trauma, respiratory, obstetrics and gynecology, infectious disease, head-ears-eyes-nose-throat, and environmental disorders. Total mean scores for the review quizzes were calculated for each participant. Scores from incomplete quizzes, defined as less than 7 out of 10 questions answered, were excluded from analysis. A mixed linear model was used to analyze the data accounting for prior 2012 in-training exam scores and nesting due to institution. A total of 135 residents participated and their data analyzed. Results from the mixed linear model indicate that the total mean score on the review quizzes was a significant [t(127) = 6.68; P<.00010] predictor of the 2013 in-training exam score after controlling for the 2012 in-training exam score. Our results suggest that scores from review quizzes delivered by an audience response system correlate with in-training exam scores. This information may be useful in identifying poor performers early so that targeted educational interventions can be made.

Enhancement of anesthesiology in-training exam performance with institution of an academic improvement policy

Anesthesiology resident physicians across the United States complete an annual in-training examination (ITE). The ITE evaluates resident knowledge and provides personalized feedback to guide future study in low scoring sections(1). Performance on the ITE correlates with outcomes on the American Board of Anesthesiology (ABA) written board examination(2). Over the last several years, declining ITE scores were observed at the University of North Carolina (UNC). In response to this decline, our department reprioritized the ITE by instituting an academic improvement policy (AIP). The AIP employed both reward for satisfactory achievement and consequence for under-performance to elevate the ITE as a "high stakes" examination. Our hypothesis was that implementation of this AIP would improve ITE scores. ITE scores were compiled from 150 residents in the Department of Anesthesiology at UNC for graduating classes from 2004-2015. Data is presented as the number of residents scoring below the 20th percentile when compared to the national distribution before and after the AIP. In addition, average USMLE Step 1 three-digit scores for each graduating class were compared to average ITE percentile scores of the corresponding graduating class (USMLE does not provide percentile scores). Between 2009 and 2013, the number of residents who scored below the 20th percentile on the ITE increased steadily to a peak of 10 in 2011. After implementation of the AIP in July 2011, there was an 80% decrease in those scoring below the 20th percentile, from 10 to 2 residents (p<0.05). Anesthesiology resident ITE scores improved after implementation of an academic improvement policy.

Focused Board Intervention (FBI): A Remediation Program for Written Board Preparation and the Medical Knowledge Core Competency

Residents deemed at risk for low performance on standardized examinations require focused attention and remediation. To determine whether a remediation program for residents identified as at risk for failure on the Emergency Medicine (EM) Written Board Examination is associated with improved outcomes. All residents in 8 classes of an EM 1-3 program were assessed using the In-Training Examination. Residents enrolled in the Focused Board Intervention (FBI) remediation program based on an absolute score on the EM 3 examination of <70% or a score more than 1 SD below the national mean on the EM 1 or 2 examination. Individualized education plans (IEPs) were created for residents in the FBI program, combining self-study audio review lectures with short-answer examinations. The association between first-time pass rate for the American Board of Emergency Medicine (ABEM) Written Qualifying Examination (WQE) and completion of all IEPs was examined using the χ(2) test. Of the 64 residents graduating and sitting for the ABEM examination between 2000 and 2008, 26 (41%) were eligible for the program. Of these, 10 (38%) residents were compliant and had a first-time pass rate of 100%. The control group (12 residents who matched criteria but graduated before the FBI program was in place and 4 who were enrolled but failed to complete the program) had a 44% pass rate (7 of 16), which was significantly lower (χ(2) = 8.6, P = .003). The probability of passing the ABEM WQE on the first attempt was improved through the completion of a structured IEP.