Medical Knowledge Self-Assessment Program Research Articles

Performance evaluation of ChatGPT 4.0 on cardiovascular questions from the medical knowledge self-assessment program

Abstract Background Medical trainees are increasingly using online chat-based artificial intelligence (AI) platforms as supplementary resources for board exam preparation and clinical decision support. Prior studies have evaluated the performance of AI chatbots like ChatGPT on various general standardized tests such as the United States Medical Licensing Examination (USMLE), but little is known about their performance on subspecialty-focused exam questions, particularly related to clinical management and treatment. Objective This study aims to evaluate the performance of ChatGPT version 4.0 on the cardiovascular questions from the Medical Knowledge Self-Assessment Program (MKSAP) 19, a widely used resource for board exam preparation in the United States. Methods We submitted all cardiovascular questions from MKSAP 19 to ChatGPT 4.0, covering a broad range of cardiology topics in a multiple-choice format. Performance was gauged against both the official MKSAP answer key and average trainee scores obtained from the MKSAP website. Out of 129 questions, 4 were invalidated due to post-publication data, and 18 were excluded due to reliance on visual aids, leaving 107 questions for analysis. Results ChatGPT 4.0 correctly answered 93 out of 107 questions, reflecting an 87% accuracy rate, compared to a 60% accuracy rate averaged among all human users for the same questions (p<0.0001). ChatGPT accuracy rates for each question category (e.g., heart failure, electrophysiology, etc.) are provided in the figure. On the 14 questions that ChatGPT answered incorrectly, human users averaged a 47% accuracy rate. All 14 incorrectly answered questions were related to clinical management and treatment (as opposed to diagnosis or epidemiology), and the majority (10/14) involved choosing treatment medications or choosing between medications versus interventions. Conclusion ChatGPT 4.0 surpassed the typical 70% accuracy threshold required for passing the internal medicine board certification but performed less well in answering questions about clinical management and treatment. This performance suggests that ChatGPT 4.0 could be an effective adjunct tool for cardiology education, but raises some concerns about its use in clinical decision support for trainees.

Evaluating Performance of ChatGPT on MKSAP Cardiology Board Review Questions

Chat Generative Pretrained Transformer (ChatGPT) is a natural language processing tool created by OpenAI. Much of the discussion regarding artificial intelligence (AI) in medicine is the ability of the language to enhance medical practice, improve efficiency and decrease errors. The objective of this study was to analyze the ability of ChatGPT to answer board-style cardiovascular medicine questions by using the Medical Knowledge Self-Assessment Program (MKSAP).The study evaluated the performance of ChatGPT (versions 3.5 and 4), alongside internal medicine residents and internal medicine and cardiology attendings, in answering 98 multiple-choice questions (MCQs) from the Cardiovascular Medicine Chapter of MKSAP. ChatGPT-4 demonstrated an accuracy of 74.5 %, comparable to internal medicine (IM) intern (63.3 %), senior resident (63.3 %), internal medicine attending physician (62.2 %), and ChatGPT-3.5 (64.3 %) but significantly lower than cardiology attending physician (85.7 %). Subcategory analysis revealed no statistical difference between ChatGPT and physicians, except in valvular heart disease where cardiology attending outperformed ChatGPT (p = 0.031) for version 3.5, and for heart failure (p = 0.046) where ChatGPT-4 outperformed senior resident. While ChatGPT shows promise in certain subcategories, in order to establish AI as a reliable educational tool for medical professionals, performance of ChatGPT will likely need to surpass the accuracy of instructors, ideally achieving the near-perfect score on posed questions.

Performance of Publicly Available Large Language Models on Internal Medicine Board-style Questions.

Ongoing research attempts to benchmark large language models (LLM) against physicians' fund of knowledge by assessing LLM performance on medical examinations. No prior study has assessed LLM performance on internal medicine (IM) board examination questions. Limited data exists on how knowledge supplied to the models, derived from medical texts improves LLM performance. The performance of GPT-3.5, GPT-4.0, LaMDA and Llama 2, with and without additional model input augmentation, was assessed on 240 randomly selected IM board-style questions. Questions were sourced from the Medical Knowledge Self-Assessment Program released by the American College of Physicians with each question serving as part of the LLM prompt. When available, LLMs were accessed both through their application programming interface (API) and their corresponding chatbot. Mode inputs were augmented with Harrison's Principles of Internal Medicine using the method of Retrieval Augmented Generation. LLM-generated explanations to 25 correctly answered questions were presented in a blinded fashion alongside the MKSAP explanation to an IM board-certified physician tasked with selecting the human generated response. GPT-4.0, accessed either through Bing Chat or its API, scored 77.5-80.7% outperforming GPT-3.5, human respondents, LaMDA and Llama 2 in that order. GPT-4.0 outperformed human MKSAP users on every tested IM subject with its highest and lowest percentile scores in Infectious Disease (80th) and Rheumatology (99.7th), respectively. There is a 3.2-5.3% decrease in performance of both GPT-3.5 and GPT-4.0 when accessing the LLM through its API instead of its online chatbot. There is 4.5-7.5% increase in performance of both GPT-3.5 and GPT-4.0 accessed through their APIs after additional input augmentation. The blinded reviewer correctly identified the human generated MKSAP response in 72% of the 25-question sample set. GPT-4.0 performed best on IM board-style questions outperforming human respondents. Augmenting with domain-specific information improved performance rendering Retrieval Augmented Generation a possible technique for improving accuracy in medical examination LLM responses.

Swiss general internal medicine board examination: quantitative effects of publicly available and unavailable questions on question difficulty and test performance.

Formerly, a substantial number of the 120 multiple-choice questions of the Swiss Society of General Internal Medicine (SSGIM) board examination were derived from publicly available MKSAP questions (Medical Knowledge Self-Assessment Program®). The possibility to memorise publicly available questions may unduly influence the candidates' examination performance. Therefore, the examination board raised concerns that the examination did not meet the objective of evaluating the application of knowledge. The society decided to develop new, "Helvetic" questions to improve the examination. The aim of the present study was to quantitatively assess the degree of difficulty of the Helvetic questions (HQ) compared with publicly available and unavailable MKSAP questions and to investigate whether the degree of difficulty of MKSAP questions changed over time as their status changed from publicly available to unavailable. The November 2019 examination consisted of 40 Helvetic questions, 40 publicly available questions from MKSAP edition 17 (MKSAP-17) and 40 questions from MKSAP-15/16, which were no longer publicly available at the time of the examination. An one factorial univariate analysis of variance (ANOVA) examined question difficulty (lower values mean higher difficulty) between these three question sets. A repeated ANOVA compared the difficulty of MKSAP-15/16 questions in the November 2019 examination with the difficulty of the exact same questions from former examinations, when these questions belonged to the publicly available MKSAP edition. The publicly available MKSAP-17 and the publicly unavailable Helvetic questions served as control. The analysis of the November 2019 exam showed a significant difference in average item difficulty between Helvetic and MKSAP-17 questions (71% vs 86%, p <0.001) and between MKSAP-15/16 and MKSAP-17 questions (70% vs 86%, p <0.001). There was no significant difference in item difficulty between Helvetic and MKSAP-15/16 questions (71% vs 70%, p = 0.993). The repeated measures ANOVA on question use and the three question categories showed a significant interaction (p <0.001, partial eta-squared = 0.422). The change in the availability of MKSAP-15/16 questions had a strong effect on difficulty. Questions became on average 21.9% more difficult when they were no longer publicly available. In contrast, the difficulty of the MKSAP-17 and Helvetic questions did not change significantly across administrations. This study provides the quantitative evidence that the public availability of questions has a decisive influence on question difficulty and thus on SSGIM board examination performance. Reducing the number of publicly available questions in the examination by introducing confidential, high-quality Helvetic questions contributes to the validity of the board examination by addressing higher order cognitive skills and making rote-learning strategies less effective.

Supplementing the Subinternship: Effect of E-Learning Modules on Subintern Knowledge and Confidence

Supplementing the Subinternship: Effect of E-Learning Modules on Subintern Knowledge and Confidence

Screening for Depression in People with Kidney Failure.

A large proportion of patients with CKD before and after initiation of maintenance dialysis suffer from major depressive disorder, such that about 20% of those with non-dialysis CKD and 25% of those with kidney failure are affected ([1][1]). More daunting is the actuality, validated in multiple

Don't Fall for That: A Residency Curricular Innovation about Fall Prevention.

Amid a growing geriatric population and rise in frailty-related morbidity, fall prevention represents an opportunity to improve patient outcomes and reduce health care costs. Traditional lectures on geriatric content have had limited impact on physician behaviors; however, use of multimodal teaching can be more effective in building knowledge and skills. To develop a novel, engaging fall prevention program to empower internal medicine residents to identify and manage patients at risk of falls and fall-related injuries. Two 20-minute multimodal workshops were created: 1) a classroom session with a video depicting a fall scenario, a team exercise ("Where's the Fall Risk?") and review of the American Geriatrics Society Beers Criteria; and 2) a small-group session reviewing a screening algorithm, case study, physical examination maneuvers, and patient resources. The first workshop included a 5-minute Kaiser Permanente video depicting an older couple whose travel plans are upended by a fall and how they modify their home and lifestyle, a competitive game in which trainees identify fall hazards, an overview of Beers Criteria, and Medical Knowledge Self-Assessment Program questions to apply knowledge to practice. The second workshop, held in small groups before clinic, included a discussion of the Centers for Disease Control and Prevention's fall prevention screening algorithm, review of a case, and education on how to properly perform the Timed Up and Go test. Fall prevention remains an important yet undertaught topic for trainees and practicing physicians. These brief multicomponent workshops can be easily implemented and adapted for all clinical learners.

Exploring examinee behaviours as validity evidence for multiple-choice question examinations.

Clinical-vignette multiple choice question (MCQ) examinations are used widely in medical education. Standardised MCQ examinations are used by licensure and certification bodies to award credentials that are meant to assure stakeholders as to the quality of physicians. Such uses are based on the interpretation of MCQ examination performance as giving meaningful information about the quality of clinical reasoning. There are several assumptions foundational to these interpretations and uses of standardised MCQ examinations. This study explores the implicit assumption that cognitive processes elicited by clinical-vignette MCQ items are like the processes thought to occur with 'real-world' clinical reasoning as theorised by dual-process theory. Fourteen participants (three medical students, five residents and six staff physicians) completed three sets of five timed MCQ items (total 15) from the Medical Knowledge Self-Assessment Program (MKSAP). Upon answering a set of MCQs, each participant completed a retrospective think aloud (TA) protocol. Using constant comparative analysis (CCA) methods sensitised by dual-process theory, we performed a qualitative thematic analysis. Examinee behaviours fell into three categories: clinical reasoning behaviours, test-taking behaviours and reactions to the MCQ. Consistent with dual-process theory, statements about clinical reasoning behaviours were divided into two sub-categories: analytical reasoning and non-analytical reasoning. Each of these categories included several themes. Our study provides some validity evidence that test-takers' descriptions of their cognitive processes during completion of high-quality clinical-vignette MCQs align with processes expected in real-world clinical reasoning. This supports one of the assumptions important for interpretations of MCQ examination scores as meaningful measures of clinical reasoning. Our observations also suggest that MCQs elicit other cognitive processes, including certain test-taking behaviours, that seem 'inauthentic' to real-world clinical reasoning. Further research is needed to explore if similar themes arise in other contexts (e.g. simulated patient encounters) and how observed behaviours relate to performance on MCQ-based assessments.

Editorial introductions

Editorial introductions

Dermatology curriculum for internal medicine residents: a randomized trial.

Physicians in specialties other than dermatology care for the majority of patients with skin diseases, yet most physicians receive little training in dermatology. The primary objective of this study was to determine whether there would be a sizable (20%) improvement in posttraining scores for internal medicine residents after completing 1 of 3 assigned curricula. A secondary objective was to determine whether there were significant differences in improvement among the 3 resident cohorts after completing their curriculum. Finally, we explored the residents' change in perceived clinical knowledge postcurriculum. Thirty-six postgraduate year 2 internal medicine residents were randomized to complete 1 of 3 one-month dermatology curricula (didactic, clinical, or combined). The main outcome measure was performance on different sets of Medical Knowledge Self-Assessment Program (MKSAP)-15 questions at study entry and completion. A secondary outcome was self-rated performance in 3 clinical domains. All participants completed the study. All curricula led to an improvement in MKSAP-15 scores, but only students who completed the didactic curriculum demonstrated a 20% improvement in posttraining scores. A larger number of residents completing the clinical and didactic curricula rated their clinical performance as improved compared to those who completed the combined curriculum. While all 3 curricula led to improvement, as measured by a standardized assessment, a didactic curriculum in dermatology resulted in the largest improvement in knowledge as measured by a multiple-choice test.

Trust in Residents and Board Examinations: When Sharing Crosses the Boundary

Recently, illegal reproduction and transmission of board certification examination questions has received public attention. In 2010, the American Board of Internal Medicine suspended or revoked the certification of 139 physicians found to be either disclosing or soliciting examination questions for a board review company that encouraged physicians to relay questions from memory. In 2012, CNN reported on two circumstances of residents preparing for board certification examinations using questions reproduced by previous examinees.1,2 Immediately after taking the American Board of Radiology (ABR) medical physics examination, radiology residents wrote down test questions to which they were assigned. These questions, known as “recalls,” were then shared among future trainees. Dermatology residents also reproduced questions (referred to as “airplane notes”) minutes after their certification examination, often in groups. In these two cases, 20% to 50% of test questions have appeared on previous examinations.1,2 Reproduction of the questions clearly violates examination copyright policies, breaching the agreement to which their signature binds them.3,4 In a relevant study, nearly all medical students and faculty members surveyed considered using a stolen copy of a board examination unethical.5 Moreover, the ABR considers using recalls cheating.1 A profession is distinguished from a craft guild on the basis of educational standards, coupled with an established code of ethics in which a service rather than profit orientation is enshrined to engender the public trust.6 For this reason, ethical breaches and unprofessional behavior, such as the morally misguided use of reproduced questions, undermine the integrity of our entire profession. Yet, some physicians interviewed by CNN rationalized recall use as simply a study guide, whereby test questions are used to confirm understanding of the underlying knowledge rather than rote memorization of answers. Cheating of this nature casts an overarching ethical pall over the moral formation and integrity of physicians, eroding public trust in professional practice, teaching, and scholarship. Cheating on examinations in medical school correlates with falsifying information in a patient’s medical record.7 Surprisingly, one third of medical students engaged or would consider engaging in such falsification.8 This behavior reflects poor moral reasoning and/or willingness to subordinate morality to the achievement of goals. Such morally unacceptable conduct violates the primacy of medicine’s beneficence-based responsibility and undermines the integrity of our profession. From the patient’s perspective, physicians should be certified on the basis of a valid assessment of the knowledge requisite to provide high-quality care. If patients cannot trust the knowledge that board certification is expected to measure, the medical profession loses the public’s trust and confidence. Moreover, patients will only benefit from certification if it indicates the ability to care for them, rather than efficient preparation for board examinations. The distinction between cheating and guided study is crucial. Historical test questions are routinely used throughout higher education,9 and recent examination experience is commonly used to create focused study materials. The American College of Physicians produces the Medical Knowledge Self-Assessment Program, which includes a summary of high-yield information likely to appear on the examination.10 The introduction states, “The content was turned over to 11 carefully selected chief residents and fellows who had recently passed the certification exam. These physicians strained the essential testable points.” The editor continues, “As a frequent lecturer on Board preparation, I rely on input from hundreds of post-examination residents. With their ideas in mind, I refined the outline of Board Basics to target important topics and eliminate nonessential information.”10 Used by medical students for over 2 decades, the First Aid series is updated annually based on examinee reports. Using a focused study guide and sample questions created with examinees’ input bears some similarity to using practice questions that may have been on a recent examination as a vehicle for targeted learning. The difference lies in the detail and specificity of the information conveyed to future examinees. Nevertheless, from the public’s perspective, both practices represent shortcuts for examination success. Surprisingly, only 43% of medical students surveyed thought that it was clearly wrong for a student completing a clinical examination to convey information regarding its content to another student taking the examination.7 There is clearly a commonality between this finding and the interviewed physicians’ rationalized view of recalls.

Deliberate Practice in Medicine: The Motivation to Engage in Work-Related Learning and Its Contribution to Expertise

This study examined physicians’ motivation to engage in work-related learning and its contribution to expertise development beyond work experience. Based on deliberate practice theory, semi-structured interviews were conducted with 17 residents and 28 experienced physicians in internal medicine, focusing on the activities they engaged in during work that might contribute to professional development and the goals underlying this behavior. Learning motivation was also measured using a goal orientation questionnaire. Expertise was measured by a case test derived from the Medical Knowledge Self Assessment Program (MKSAP). The interviews showed that participants’ learning was largely embedded in everyday work; most of their learning activities were inherent to the job rather than motivated by competence improvement goals. The problems encountered in patient care played a key role in prompting learning. Role, work experience and work situation affected the type of activities engaged in, as well as the intensity of practice. Deliberate engagement in work-related learning activities was related neither to goal orientations nor to case test performance, except activities by the experienced physicians to keep up-to-date. Work experience, in contrast, showed a clear positive relationship with the performance of residents. Two main conclusions can be drawn from this study: (1) patient care induced relevant learning activities aimed at performance improvement for the patient’s sake; (2) deliberate investments in learning can be enhanced to promote expertise development. Self-regulated learning by physicians needs to be strengthened and the organization for which they work should facilitate and encourage learning in daily practice.

Editorial introductions

Editorial introductions

The Internist's Literature on Chronic Rhinosinusitis: An Evaluation of the Amount and Quality of Published Information

Most patients with chronic rhinosinusitis (CRS) are treated by internists, yet the information available to internists about CRS is rarely assessed. The author undertook a study to evaluate the amount and quality of information about CRS that is available to internists. Four information sources were analyzed: (1) five journals routinely read by internists, published over a 10-year period and searched on PubMed for the "exploded" terms chronic and sinusitis; (2) seven representative general and outpatient medicine texts, searched on MD Consult and STAT!Ref; (3) the 14th edition of the Medical Knowledge Self-Assessment Program (MKSAP 14), a review of internal medicine; and (4) four commonly used Internet databases. The five journals contained a total of 37,250 cited articles of various types over the study period; CRS was mentioned in only 19 of them. The standard medical texts offered little information about CRS, and MKSAP 14 did not mention CRS at all in the text, although it included some information in the questions and answers. Finally, among the online sources, UpToDate, a proprietary Internet database, offered an excellent discourse on CRS; in contrast, the Physicians' Information and Education Resource, maintained by the American College of Physicians, contained no information about CRS in its 436 subject modules. The author concludes that, with few exceptions, the literature routinely available to internists offers little information about CRS.

Continuing Medical Education

PATIENTS EXPECT THEIR PHYSICIANS TO HAVE THE RIGHT answer every time. Does the testing process in medical school or the continuing medical education (CME) process increase the likelihood that patient expectations will be met? A medical student finishing Step 1 or Step 2 examinations receives a score. The score reflects answers that the student knew, answers chosen by process of elimination, or answers the student guessed correctly. The Medical Knowledge Self-Assessment Program is an authoritative manual designed to update internists on the practice of medicine over the previous 3 years. Each section of the document ends with a series of multiple-choice questions. The physician selects responses and submits the answers to the American College of Physicians, which awards the CME credits. How should the medical student or the practicing physician feel about answering 70% of the questions correctly? Is the testing system at the heart of why quality-of-care scores for individual practices are so low? The purpose of this Commentary is to suggest exploration of a culture shift in medicine to reinforce the notion of knowing the right answer every time. What if the testing process in medical school or CME were changed so that the medical student or physician were faced with a problem and had to decide what to do? This experience would teach a physician how to look up information, read articles, determine if the articles were relevant, and how to apply the literature and evidence-based medicine to individual patients. Such an approach raises concern if a physician needs to know immediately what to do. So what if a list of medical emergencies and related patient scenarios was developed, as well as a system for reminding all physicians, regardless of specialty, what to do in these situations? Certain types of chest pain, headaches, and other symptoms would fall in this category. In these cases, physicians should know the correct answer 100% of the time without looking it up. For the rest of medical practice, shouldn’t physicians be taught how to find the answer and be confident that it is correct? To accomplish the latter, evidence-based medicine must be translated into practice. Although the movement for evidence-based medicine began nearly 2 decades ago, little progress has been made in adopting the tools of evidencebased medicine into routine practice. Systematic reviews grade and synthesize evidence from diverse studies and many of these meta-analyses are used. But there is little documentation about how they have been used and whether they have affected practice. Beyond that, what about the simplest tools of quantitative sciences? Are the concepts of prior probability, posterior probability, transforming a prior probability into a posterior probability, or any other principles in decision analysis used in medicine? After reading the first half of the Medical Knowledge SelfAssessment Program curriculum for internists, I was fascinated by the absence of any quantitative decision tools. The field of decision analysis has produced numerous studies about using decision tools to improve the practice of medicine. Dean et al demonstrated that a decision tool could ensure that a patient received the correct antibiotic. Years ago Adams et al demonstrated how a simple decision tool could help to decide whether a patient who presents to an emergency department with abdominal pain needs surgery. Private firms are beginning to collect decision tools and make them available to physicians. However, in general, use of sensitivity, specificity, likelihood ratios, prior probabilities, and posterior probabilities in medical decision making has been largely ignored. An entire field of science is missing from the practice of medicine. At the same time, production of a slightly better drug or device prompts a fullscale advertising campaign, backed by a sales force. Despite almost 800 reports on the Framingham risk score this decade, few physicians enter on a patient’s record the probability that the patient will have a cardiac event in the next 10 years. Why are there no formal assessments of probabilities and utilities in making difficult clinical decisions? When a patient enters a physician’s office, why doesn’t the physician record a prior probability that the patient has condition x, y, or z; then, based on the history and physical, produce a posterior probability that serves as the basis for ordering tests? Thirty years ago, a physician had to know what laboratory tests to order. He or she might have been required to write the test legibly on a laboratory order slip. With today’s electronic medical record–supported process, the physician views

Ambulatory Morning Report: An Underutilized Educational Modality

Background: Many medicine residency training programs include a lecture-based preclinic conference series as part of the ambulatory educational curriculum when more effective teaching formats might be available. Our institution has replaced this lecture-based teaching format with an ambulatory morning report modeled after the inpatient paradigm. This study compares the efficacy of these 2 teaching models and defines the desired characteristics of this new teaching strategy. Description: We first conducted a background study by obtaining permission to use test questions from the Medical Knowledge Self-Assessment Program to develop pre- and postambulatory rotation tests to validate our ambulatory curriculum. Forty-three of 44 interns completed both pre- and postambulatory block testing. The mean score on these tests improved from 67% to 81%. Although this overall improvement achieved statistical significance, test question subgroup analysis clearly indicated that the improved knowledge in test items relating to our preclinic conference topics contributed negligibly to the overall statistical improvement. As such, we ultimately replaced our preclinic conference with a morning report format. In this study, 82 residents were enrolled in a 2-year prospective observational study. This study group completed an ambulatory curriculum in which the 1st year was completely lecture based, and the 2nd year included the morning report format. We were thus able to survey residents' opinions regarding the effectiveness of the 2 very different teaching formats used in consecutive years. Evaluation: The survey results from those residents exposed to both teaching formats over 2 years revealed a high degree of satisfaction with the ambulatory morning report format. When comparing the long-term educational value, 94% of the residents found the morning report format more effective than the lecture-based preclinic conference. In addition, many desirable characteristics of the morning report also were defined. Conclusions: Lecture-based preclinic conferences might not be the most effective way of conveying information over the long term. Residents seem to prefer the more interactive morning report format. When organizing such a teaching format, attention should be directed toward the characteristics that were felt to be desirable by those we are trying to teach.

The American college of physicians and the medical knowledge self-assessment program paradigm

The American college of physicians and the medical knowledge self-assessment program paradigm

Pulmonary Disease: An Annotated Bibliography of Recent Literature

Pulmonary Disease: An Annotated Bibliography of Recent Literature

Allergy and immunology: an annotated bibliography of recent literature. Allergy and Immunology Committee, American College of Physicians.

Allergy and immunology: an annotated bibliography of recent literature. Allergy and Immunology Committee, American College of Physicians.

Clinical Pharmacology: An Annotated Bibliography of Recent Literature

Clinical Pharmacology: An Annotated Bibliography of Recent Literature