Putting the Secure Examination to the Test

Abstract

Listen

Translate article

RECENT CONTROVERSY ABOUT THE AMERICAN Board of Internal Medicine (ABIM) Maintenance of Certification (MOC) process has centered on the role of the secure examination. Some physicians object to taking any examination, whereas others assert that the internal medicine and subspecialty examinations are not sufficiently focused to reflect physician practices. Others have questioned why the examination is not “open-book” to mirror a real-world practice environment. Even though all 24 certifying boards of the American Board of Medical Specialties require secure examinations for MOC, and many have required recertification for decades, the questions raised regarding ABIM examinations have relevance to all medical specialties. This Commentary intends to clarify the science and philosophy behind examination development and the current psychometric principles that guide the process. Certifying boards strive to provide the public with useful information about a physician’s competence while meeting expectations both of physicians and the public for a relevant, valid, and reliable assessment process. The ABIM’s MOC is a multidimensional program that meets these goals through tools designed to evaluate several competencies. Physicians choose assessments of knowledge, practice improvement, and communication skills based on personal evaluation of their learning needs and their practice profile. The secure examination component of MOC assesses the broad-based knowledge that the public expects of a physician who identifies himself or herself as a general internist or subspecialist. Its purpose is to define a discipline as well as test an individual’s clinical knowledge and judgment. The ABIM follows the Standards for Educational and Psychological Testing established through a collaboration of the American Psychological Association, the American Educational Research Association, and the National Council of Measurement and Evaluation. These standards represent a wellestablished professional consensus concerning appropriate and fair test use and are based on psychometrics, the science of testing that governs examination administration, production, content, scoring, generalization, and extrapolation to real-world practice. Developing an examination that meets these standards is a labor-intensive undertaking. Clinical content and case scenarios in the ABIM’s MOC examinations are developed by carefully selected committee members to ensure adequate content coverage and to include the perspectives of physicians both in practice and in academia. Topics in which physician intervention has significant effects on patient health outcomes are identified. Each topic proposed for the test is rated by practicing physicians, program directors, and trainees for importance and frequency in practice. Low-rated topics are typically excluded from the MOC examination. Question stems provide mini case simulations or patient vignettes that present a realistic description of the patient, symptoms, and laboratory values, requiring the integration of information, prioritization of alternatives, and use of clinical judgment in reaching an appropriate conclusion and deciding on a course of action. Vignette questions are more relevant to real-world patient care than nonvignettes. Through committee review and pretesting, questions are evaluated to ensure that they are evidence-based, technically sound, and fair. The performance of each question is reviewed for content, difficulty level, and how well it discriminates between physicians who clearly demonstrate competency in the domains expected of internists and those who do not. If a question is too difficult or does not discriminate well among physicians, it may be revised or discarded. The ABIM MOC examinations as currently constructed maintain high reproducibility of test scores (a Cronbach 0.90 [1.0 is the maximum]). To date, case-based multiple-choice questions portrayed as patient vignettes are the most efficient, cost-effective, and feasible method for assessing a broad domain area and extrapolating to real-world patient care. The ABIM is currently studying ways to incorporate computer clinical simulation into the examination to better assess clinical skills and increase realism. To set a credible pass-fail standard, the ABIM relies on a committee of knowledgeable internists from private practice and academic medicine to set an absolute stan-