Validity Evidence for the General Pediatrics Board Certification Examinations: A Practice Analysis

Abstract

The American Board of Pediatrics (ABP) is responsible for certifying that general pediatricians and pediatric subspecialists have met and continue to meet standards of excellence that lead to the high-quality health care of infants, children, and adolescents. Board certification involves 2 primary components: successfully completing an accredited training program and taking and passing a written examination. The ABP uses a process known as practice analysis to ensure that its exams measure the knowledge required for safe and effective clinical practice. Measurement professionals and industry standards, including the Standards for Educational and Psychological Testing1American Educational Research Association, American Psychological Association, and National Council on Measurement in Education Standards for educational and psychological testing. American Educational Research Association, Washington (DC)2014Google Scholar and the National Commission for Certifying Agencies' Standards for the Accreditation of Certification Programs,2National Commission for Certifying Agencies Standards for the accreditation of certification programs. Institute for Credentialing Excellence, Washington (DC)2014Google Scholar recognize practice analysis as the primary methodology for establishing a certification examination's validity and relevance.3Henderson J.P. Smith D. Job/practice analysis.in: Knapp J. Anderson L. Wild C. Certification: the ICE handbook. Institute for Credentialing Excellence, Washington (DC)2009: 123-148Google Scholar, 4Raymond M.R. Job analysis, practice analysis, and the content of credentialing examinations.in: Lane S. Raymond M.R. Haladyna T.M. Handbook of test development. Routledge/Taylor, New York2016: 144-164Google Scholar The purpose of the 2016 general pediatrics practice analysis was to identify the knowledge required for clinical practice as a general pediatrician and to use that information to update the ABP's General Pediatrics Content Outline,5Althouse L.A. Du Y. Ham H.P. Confirming the validity of the General Pediatrics certification examinations: a practice analysis.J Pediatr. 2009; 155 (e1): 155-156Abstract Full Text Full Text PDF PubMed Scopus (4) Google Scholar which specifies the knowledge areas measured by the ABP's general pediatrics exams. The previous practice analysis for general pediatrics was conducted in 2007.6American Board of Pediatrics General Pediatrics content outline.https://www.abp.org/sites/abp/files/pdf/gp_contentoutline_2017.pdfDate accessed: December 1, 2017Google Scholar Brief descriptions of each of the ABP's general pediatrics exams are provided in the next section. The ABP develops 3 different examinations that are administered to general pediatricians at various points in their careers. The In-Training Examination (ITE) is offered annually to pediatric residents. Residents receive feedback regarding their ITE performance to help them identify areas of strength and weakness. In addition, program directors receive information regarding their residents' performance, including national comparison data, to help them evaluate their programs and identify individuals who might benefit from additional assistance. Residents who successfully graduate from an accredited residency program and meet the ABP's eligibility requirements can apply to take the Certifying Examination. Those who pass the Certifying Examination become Board-certified and are subsequently referred to as “diplomates.” Diplomates who wish to maintain their certification must meet 4 separate requirements (commonly referred to as parts 1-4) within a 5-year cycle.7American Board of Pediatrics Maintenance of certification (MOC).https://www.abp.org/content/maintenance-certification-mocDate accessed: January 23, 2018Google Scholar One of those requirements is the examination requirement (ie, part 3), in which diplomates must demonstrate that they have kept their knowledge up to date by taking and passing the Maintenance of Certification (MOC) examination once every 10 years. The ABP is currently pilot-testing an alternative MOC assessment model (MOCA-Peds) that consists of shorter, more frequent assessments designed to function as both a measure of general pediatric medical knowledge and a learning opportunity for the test taker.8American Board of Pediatrics Maintenance of Certification Assessment for Pediatrics (MOCA-Peds).https://www.abp.org/mocapedsDate accessed: December 1, 2017Google Scholar If successful, MOCA-Peds will provide another option for meeting the part 3 requirement. The practice analysis consisted of 3 major phases. In phase 1, a diverse panel of practicing general pediatricians identified the knowledge required of a general pediatrician and used that information to produce a draft content outline. In phase 2, all Board-certified general pediatricians were invited to participate in an online survey designed to collect feedback regarding the draft outline. The results of the survey informed phase 3, which involved making final revisions to the new content outline and establishing the exam weights. The phases are described in more detail in the following sections. In January 2016, a diverse panel of 13 practicing general pediatricians met face-to-face for 3 days to develop a draft content outline (Table I; available at www.jpeds.com). Panelists were selected to obtain representation across many important factors, including practice setting, patient population, race/ethnicity, geographic location, and years of clinical experience, from a database of Board-certified pediatricians who had expressed interest in serving in a volunteer role for the ABP. The panel engaged in several activities facilitated by a trained psychometrician, including delineating the tasks performed by general pediatricians, identifying the knowledge required to safely and effectively perform those tasks, and organizing that knowledge into content categories (ie, content domains and subdomains) that ultimately resulted in the draft content outline. During the process of identifying tasks, the panel compared its list with the list of 17 entrustable professional activities (EPAs) for general pediatrics9American Board of Pediatrics Entrustable professional activities for general pediatrics.https://www.abp.org/entrustable-professional-activities-epasDate accessed: December 1, 2017Google Scholar, 10Carraccio C. Englander R. Gilhooly J. Mink R. Hofkosh D. Barone M.A. et al.Building a framework of entrustable professional activities, supported by competencies and milestones, to bridge the educational continuum.Acad Med. 2017; 92: 324-330Crossref PubMed Scopus (131) Google Scholar to help ensure that the panel had accurately captured the critical roles and responsibilities of the general pediatrician. During the development of the draft content outline (ie, identifying the knowledge required to safely and effectively perform those tasks), the panel also referenced the previous version of the outline to help ensure that no relevant knowledge categories were missed. The panel's final draft outline included a total of 25 major content domains (Table II; available at www.jpeds.com). Each domain included several subdomains, as described on the ABP's website.5Althouse L.A. Du Y. Ham H.P. Confirming the validity of the General Pediatrics certification examinations: a practice analysis.J Pediatr. 2009; 155 (e1): 155-156Abstract Full Text Full Text PDF PubMed Scopus (4) Google Scholar One of the most notable changes to the new outline that emerged during phase 1 was a second dimension for classifying test questions referred to as “universal tasks” (Table III; available at www.jpeds.com). Under the new outline, each exam question is classified into a content domain/subdomain and a universal task. For example, consider a test question designed to assess a pediatrician's ability to diagnose a child with depression. That question would likely be classified into subdomain 5.B.2.b (ie, Mental and Behavioral Health → Psychology/psychiatric disorders → Identified disorders → Depression) and universal task 3 (ie, Diagnosis). The addition of universal tasks was intended primarily to improve the clinical relevance of the exams. First, questions that do not align with both a content domain/subdomain and a universal task are removed from the pool of questions. Second, establishing universal tasks as a second dimension allows weighting of the overall exam along both dimensions in a manner that reflects clinical practice as closely as possible. For example, if general pediatricians spend most of their time making management or treatment decisions (ie, universal task 4), then the exam weights (ie, the percentage of test questions assigned to each category) can be specified accordingly. In the second phase, all Board-certified general pediatricians (n = 69 117) were invited to participate in an online survey to validate the work of the panel. Of the 69 177 diplomates invited, 10 214 (14.8%) participated in the survey, which, although low, is within the range of typical response rates for lengthy practice analysis surveys.11Wyse A. Eckerly C. Babcock B. Anderson D. A study of potential methods to increase response rates in task inventory surveys.CLEAR Exam Rev. 2016; 16: 15-22Google Scholar The first section of the survey collected demographic and practice setting information to evaluate the representativeness of the sample (Table IV; available at www.jpeds.com). The second section asked respondents to rate the content domains and subdomains using 2 rating scales: frequency and criticality. The frequency scale indicated how frequently each knowledge area was required in practice, and the criticality scale indicated the extent to which harm (eg, physical, psychological, financial) would result from a lack of knowledge in each content area. The average frequency rating for each content domain was multiplied by its average criticality rating to produce a relative importance score that provides a rough guide as to which domains general pediatricians consider the most important. The final 2 sections of the survey asked respondents to specify the percentage of exam questions that should be associated with each content domain (across all universal tasks) and each universal task (across all content domains), respectively. Respondents also had opportunities via open-ended comment fields to identify knowledge areas that were missing or to identify knowledge areas that were listed but were not deemed critical to the practice of general pediatrics and thus should be excluded. In the third phase, the panel reviewed the survey results, including open-ended comments, and used that feedback to guide minor modifications to the content outline. No content subdomains were removed owing to low frequency or criticality ratings or based on open-ended comments. A small number of content subdomains were added, however, and in a few existing subdomains, the wording was modified slightly to enhance clarity based on respondents' comments. Survey results were also used to establish preliminary exam weights for both content domains and universal tasks. Relative importance scores (ie, the product of frequency and criticality ratings) were used to compute preliminary weights for the content domains by dividing the relative importance score for a domain by the sum of the relative importance scores for all domains. Expressed as an equation, preliminary domain weights were calculated asDWi=RISi∑iRISi,where DWi is the preliminary domain weight for domain i and RISi is the relative importance score for domain i. Preliminary weights for the universal tasks were obtained directly from the survey question that asked diplomates to specify the percentage of exam questions that should fall into each universal task. Ultimately, the panel reached consensus regarding the final draft of the content outline. The panel's recommended weights for both the content domains and universal tasks reflected only minor deviations from the preliminary weights. Average frequency ratings, average criticality ratings, average relative importance scores, and final exam weights for all 25 content domains are presented in Table II. In the table, the 3 highest-rated domains with respect to frequency, criticality, and relative importance are noted in bold. The universal tasks and their final exam weights are presented in Table III. The draft General Pediatrics Content Outline was presented to the ABP's Board of Directors and approved in October 2016. The new outline was published on the ABP's website in December 2016 and went into effect on September 1, 2017, for all the ABP general pediatrics examinations (In Training, Certifying, and MOC exams, including the MOCA-Peds pilot). Compared with results from the previous practice analysis study,6American Board of Pediatrics General Pediatrics content outline.https://www.abp.org/sites/abp/files/pdf/gp_contentoutline_2017.pdfDate accessed: December 1, 2017Google Scholar this study indicates that the domains of knowledge required for the clinical practice of general pediatrics have remained relatively stable in recent years. Although some domains in the previous version of the outline (36 total domains) have been consolidated in the new outline (25 total domains), there is a significant amount of overlap in the overall content reflected in both versions. One specific trend that was recognized by the practice analysis panel and validated by the survey results is the increasing importance of knowledge pertaining to mental and behavioral health. Only 2 other content domains—Preventive Pediatrics/Well Child Care and Infectious Diseases—had higher relative importance scores and received higher exam weights than the Mental and Behavioral Health domain (Table II). The most notable changes to the structure of the new outline are the addition of the universal tasks and the reduced level of detail within the content domains/subdomains. Regarding the latter, broader categories of general pediatric knowledge remain relatively stable over time, whereas specific knowledge elements within each category may change quickly as the field of medicine evolves. The transition to a less granular outline helps avoid the development and administration of exam questions that may no longer be current. It is worth noting that the primary purpose of the General Pediatrics Content Outline is to serve as a blueprint for the general pediatric exams (and not as a study or curriculum development guide). Decisions regarding the structure of the outline, including the level of detail provided, were designed to improve the validity of the exams for the purposes of making certification decisions. The ABP welcomes additional feedback regarding the content outline. We thank Laurel Leslie, MD, and Gail McGuinness, MD, who reviewed the paper and provided feedback that greatly improved the clarity and relevance to the readership of this journal. Table IPractice analysis panelist characteristicsPanelistLocationPractice descriptionYears certifiedInpatient/hospital*Patient care performed in an inpatient or hospital setting.Outpatient†Patient care performed in an outpatient or private practice setting.Academic‡Education, training, and/or mentorship of pediatric residents.1Raleigh, NC++++182Cincinnati, OH++++133Durham, NC+++304Mount Airy, NC++++65Crownpoint, NM+++166Mercedes, TX+++117Kittanning, PA+++248Herndon, VA+++219Denver, CO+++2110Chapel Hill, NC++++1611Greenfield, MA+++412High Point, NC++++++1813Flint, MI++++7+Total percentage of work hours <25%.++Total percentage of work hours between 25% and 75%.+++Total percentage of work hours >75%.* Patient care performed in an inpatient or hospital setting.† Patient care performed in an outpatient or private practice setting.‡ Education, training, and/or mentorship of pediatric residents. Open table in a new tab Table IIContent domains: survey ratings and final exam weightsContent domainAverage frequency*Frequency rating scale: How frequently is knowledge in this content domain required in your practice? (1 = never, 2 = yearly, 3 = monthly, 4 = weekly, 5 = daily).Average criticality†Criticality rating scale: To what extent would a lack of knowledge in this content domain result in harm? (1 = no harm, 2 = little harm, 3 = moderate harm, 4 = severe harm).Relative importance‡Relative importance = average frequency × average criticality.Exam weight, %1 Preventive pediatrics/well child care4.583.2114.7082 Fetal and neonatal care3.843.2712.5653 Adolescent care4.273.0212.9054 Genetics, dysmorphology, and metabolic disorders3.072.898.8735 Mental and behavioral health4.153.2213.3656 Child abuse and neglect2.863.5410.1247 Emergency and critical care3.193.5111.2048 Infectious diseases4.473.3915.1579 Oncology2.363.127.36210 Hematology3.152.959.29411 Allergy and immunology3.822.7510.51412 Endocrinology3.262.979.68413 Orthopedics and sports medicine3.562.699.58414 Rheumatology2.532.586.53215 Neurology3.413.0210.30516 Ear, eye, nose, and throat4.192.8411.90417 Cardiology3.433.2811.25418 Pulmonology4.003.1912.76519 Gastroenterology3.952.8811.38420 Nephrology, fluids, and electrolytes3.313.1210.33421 Urology and genital disorders2.992.647.89322 Skin/dermatology4.132.5110.37423 Psychosocial issues4.193.0512.78224 Ethics3.312.849.40225 Research methods, patient safety, and quality improvement3.122.387.432The 3 highest-rated domains with respect to frequency, criticality, and relative importance are in bold type.* Frequency rating scale: How frequently is knowledge in this content domain required in your practice? (1 = never, 2 = yearly, 3 = monthly, 4 = weekly, 5 = daily).† Criticality rating scale: To what extent would a lack of knowledge in this content domain result in harm? (1 = no harm, 2 = little harm, 3 = moderate harm, 4 = severe harm).‡ Relative importance = average frequency × average criticality. Open table in a new tab Table IIIUniversal tasks: descriptions and final exam weightsUniversal taskDescriptionExam weight, %Basic science and pathophysiologyUnderstanding best practices, clinical guidelines, and foundational pediatric knowledge, including normal and abnormal function of the body and mind in an age-specific development context20Epidemiology and risk assessmentRecognizing patterns of health and disease and understanding the variables that influence those patterns10DiagnosisUsing available information (eg, patient history, physical exam) to formulate differential diagnoses, choose appropriate tests, and interpret test results to reach a likely diagnosis35Management and treatmentFormulating a comprehensive management and/or treatment plan, including reevaluation and long-term follow-up, taking into account multiple options for care35 Open table in a new tab Table IVSurvey respondent characteristics: practice setting and years certified (n = 10 214)Characteristicsn%Current clinical practice status Working primarily in the field of general pediatrics641463.1 Working primarily in 1 or more pediatric subspecialties268526.4 Working in both general pediatrics and in 1 or more pediatric subspecialties5315.2 Not working in clinical practice as a pediatrician5335.2Clinical practice setting Self-employed or privately owned medical group414342.5 Academic health center/medical school309631.7 Community or private hospital160516.5 Government (including military/VA)2973.0 Other6136.3Size of practice (number of pediatricians) 1 (solo practice)9969.9 2-5341834.0 6-10242924.1 11-25158615.8 >25163016.2Practice location community size <10 0002792.9 10 000-99 999238524.5 100 000-249 999214422.0 250 000-999 999244225.1 1 million or more247725.5Primary patient population Mostly publicly insured/underserved population360037.0 Mostly privately insured population271427.9 Approximately equal publicly and privately insured populations328533.8 Military population1251.3Current work hours Full time (≥35 h/wk)785177.3 Part time (<35 h/wk)190318.7 On temporary leave of absence1501.5 Retired or no longer working in the field of pediatrics2512.5Years Board-certified 1-10374536.7 11-20309930.3 21-30256425.1 31-406836.7 41-501141.1 >5090.1 Open table in a new tab +Total percentage of work hours <25%. ++Total percentage of work hours between 25% and 75%. +++Total percentage of work hours >75%. The 3 highest-rated domains with respect to frequency, criticality, and relative importance are in bold type.