American Board Of Family Medicine Research Articles

Assessment of ChatGPT-4 in Family Medicine Board Examinations Using Advanced AI Learning and Analytical Methods: Observational Study.

This research explores the capabilities of ChatGPT-4 in passing the American Board of Family Medicine (ABFM) Certification Examination. Addressing a gap in existing literature, where earlier artificial intelligence (AI) models showed limitations in medical board examinations, this study evaluates the enhanced features and potential of ChatGPT-4, especially in document analysis and information synthesis. The primary goal is to assess whether ChatGPT-4, when provided with extensive preparation resources and when using sophisticated data analysis, can achieve a score equal to or above the passing threshold for the Family Medicine Board Examinations. In this study, ChatGPT-4 was embedded in a specialized subenvironment, "AI Family Medicine Board Exam Taker," designed to closely mimic the conditions of the ABFM Certification Examination. This subenvironment enabled the AI to access and analyze a range of relevant study materials, including a primary medical textbook and supplementary web-based resources. The AI was presented with a series of ABFM-type examination questions, reflecting the breadth and complexity typical of the examination. Emphasis was placed on assessing the AI's ability to interpret and respond to these questions accurately, leveraging its advanced data processing and analysis capabilities within this controlled subenvironment. In our study, ChatGPT-4's performance was quantitatively assessed on 300 practice ABFM examination questions. The AI achieved a correct response rate of 88.67% (95% CI 85.08%-92.25%) for the Custom Robot version and 87.33% (95% CI 83.57%-91.10%) for the Regular version. Statistical analysis, including the McNemar test (P=.45), indicated no significant difference in accuracy between the 2 versions. In addition, the chi-square test for error-type distribution (P=.32) revealed no significant variation in the pattern of errors across versions. These results highlight ChatGPT-4's capacity for high-level performance and consistency in responding to complex medical examination questions under controlled conditions. The study demonstrates that ChatGPT-4, particularly when equipped with specialized preparation and when operating in a tailored subenvironment, shows promising potential in handling the intricacies of medical board examinations. While its performance is comparable with the expected standards for passing the ABFM Certification Examination, further enhancements in AI technology and tailored training methods could push these capabilities to new heights. This exploration opens avenues for integrating AI tools such as ChatGPT-4 in medical education and assessment, emphasizing the importance of continuous advancement and specialized training in medical applications of AI.

How Early Career Family Medicine Women Physicians Negotiate Their First Job After Residency.

Nested within a growing body of evidence of a gender pay gap in medicine are more alarming recent findings from family medicine: a gender pay gap of 16% can be detected at a very early career stage. This article explores qualitative evidence of women's experiences negotiating for their first job out of residency to ascertain women's engagement with and approach to the negotiation process. We recruited family physicians who graduated residency in 2019 and responded to the American Board of Family Medicine 2022 graduate survey. We developed a semistructured interview guide following a modified life history approach to uncover women's experiences through the transitory stages from residency to workforce. A qualitative researcher used Zoom to interview 19 geographically and racially diverse early career women physicians. Interviews were transcribed verbatim and analyzed using NVivo software following an Inductive Content Analysis approach. Three main themes emerged from the data. First, salary was found to be nonnegotiable, exemplified by participants' inability to change initial salary offers. Second, the role of peer support throughout residency and early career was crucial to uncovering and rectifying salary inequity. Third, a pay expectation gap was identified among women from minority and low-income households. To rectify the gender pay gap in medicine, a systems-level approach is required. This can be achieved through various levels of interventions: societally expanding the use of and removing the stigma around parental leave, recognizing the importance of contributions not currently valued by productivity-based payment models, examining assumptions about leadership; and institutionally moving away from fee-for-service systems, encouraging flexible schedules, increasing salary transparency, and improving advancement transparency.

Performance Evaluation of the Generative Pre-trained Transformer (GPT-4) on the Family Medicine In-Training Examination.

In this study, we sought to comprehensively evaluate GPT-4 (Generative Pre-trained Transformer)'s performance on the 2022 American Board of Family Medicine's (ABFM) In-Training Examination (ITE), compared with its predecessor, GPT-3.5, and the national family residents' performance on the same examination. We utilized both quantitative and qualitative analyses. First, a quantitative analysis was employed to evaluate the model's performance metrics using zero-shot prompt (where only examination questions were provided without any additional information). After this, qualitative analysis was executed to understand the nature of the model's responses, the depth of its medical knowledge, and its ability to comprehend contextual or new information through chain-of-thoughts prompts (interactive conversation) with the model. This study demonstrated that GPT-4 made significant improvement in accuracy compared with GPT-3.5 over a 4-month interval between their respective release dates. The correct percentage with zero-shot prompt increased from 56% to 84%, which translates to a scaled score growth from 280 to 690, a 410-point increase. Most notably, further chain-of-thought investigation revealed GPT-4's ability to integrate new information and make self-correction when needed. In this study, GPT-4 has demonstrated notably high accuracy, as well as rapid reading and learning capabilities. These results are consistent with previous research indicating GPT-4's significant potential to assist in clinical decision making. Furthermore, the study highlights the essential role of physicians' critical thinking and lifelong learning skills, particularly evident through the analysis of GPT-4's incorrect responses. This emphasizes the indispensable human element in effectively implementing and using AI technologies in medical settings.

Performance of Language Models on the Family Medicine In-Training Exam.

Artificial intelligence (AI), such as ChatGPT and Bard, has gained popularity as a tool in medical education. The use of AI in family medicine has not yet been assessed. The objective of this study is to compare the performance of three large language models (LLMs; ChatGPT 3.5, ChatGPT 4.0, and Google Bard) on the family medicine in-training exam (ITE). The 193 multiple-choice questions of the 2022 ITE, written by the American Board of Family Medicine, were inputted in ChatGPT 3.5, ChatGPT 4.0, and Bard. The LLMs' performance was then scored and scaled. ChatGPT 4.0 scored 167/193 (86.5%) with a scaled score of 730 out of 800. According to the Bayesian score predictor, ChatGPT 4.0 has a 100% chance of passing the family medicine board exam. ChatGPT 3.5 scored 66.3%, translating to a scaled score of 400 and an 88% chance of passing the family medicine board exam. Bard scored 64.2%, with a scaled score of 380 and an 85% chance of passing the boards. Compared to the national average of postgraduate year 3 residents, only ChatGPT 4.0 surpassed the residents' mean of 68.4%. ChatGPT 4.0 was the only LLM that outperformed the family medicine postgraduate year 3 residents' national averages on the 2022 ITE, providing robust explanations and demonstrating its potential use in delivering background information on common medical concepts that appear on board exams.

Sexual Misconduct by Board Certified Family Physicians.

Sexual misconduct by physicians is a consequential violation of patient trust. The purpose of this study was to determine the frequency and patterns of sexual misconduct by physicians certified by the American Board of Family Medicine (ABFM). We described a cohort of current or formerly ABFM certified physicians ("Diplomates") disciplined for sexual misconduct in 2016 to 2022. Ninety-four physicians, representing only 0.1% of ABFM Diplomates, were identified as having received disciplinary action(s) for reported sexual misconduct. These constituted 8.9% of the 1122 cases that resulted in a physician losing board certification or eligibility for any cause in 2016 to 2022. Ninety-three of the 94 physicians identified as male, with an average age of 56 (range 22 to 88 years). Eighty-nine percent of victims were female, and 90% were patients of the physician. Unwanted sexual behavior/assault occurred in more than half of the cases, whereas one third described an ongoing sexual relationship between patient and physician. Nearly 1 in 5 cases also included controlled substance prescribing. Seven cases involved minors. Noncontact ("grooming") behaviors were described in 34 cases, 28 of which included subsequent physical sexual behavior. A clinical setting was the site of misconduct in 84% of cases. Reports of sexual misconduct among board-certified family physicians are infrequent. However, any sexual misconduct by physicians is harmful to patients and the profession. The specialty should work to enhance education and change professional culture to mitigate this important problem.

Only One Quarter of Family Physicians Are Very Satisfied with Their Electronic Health Records Platform.

Two decades into the era of Electronic Health Records (EHRs), the promise of streamlining clinical care, reducing burden, and improving patient outcomes has yet to be realized. A cross-sectional family physician census conducted by the American Board of Family Medicine in 2022 and 2023 included self-reported physician EHR satisfaction. Of the nearly 10,000 responding family physicians, only one-in-four (26.2%) report being very satisfied and one-in-three (33.8%) were not satisfied. These low levels of satisfaction point to the need for greater transparency in the marketplace and pressure to increase user-centric EHR design.

Impact of response bias in three surveys on primary care providers' experiences with electronic health records.

To identify impacts of different survey methodologies assessing primary care physicians' (PCPs') experiences with electronic health records (EHRs), we compared three surveys: the 2022 Continuous Certification Questionnaire (CCQ) from the American Board of Family Medicine, the 2022 University of California San Francisco (UCSF) Physician Health IT Survey, and the 2021 National Electronic Health Records Survey (NEHRS). We evaluated differences between survey pairs using Rao-Scott corrected chi-square tests, which account for weighting. CCQ received 3991 responses from PCPs (100% response rate), UCSF received 1375 (3.6% response rate), and NEHRS received 858 (18.2% response rate). Substantial, statistically significant differences in demographics were detected across the surveys. CCQ respondents were younger and more likely to work in a health system; NEHRS respondents were more likely to work in private practice; and UCSF respondents disproportionately practiced in larger academic settings. Many EHR experience indicators were similar between CCQ and NEHRS, but CCQ respondents reported higher documentation burden. The UCSF approach is unlikely to supply reliable data. Significant demographic differences between CCQ and NEHRS raise response bias concerns, and while there were similarities in some reported EHR experiences, there were important, significant differences. Federal EHR policy monitoring and maintenance require reliable data. This test of existing and alternative sources suggest that diversified data sources are necessary to understand physicians' experiences with EHRs and interoperability. Comprehensive surveys administered by specialty boards have the potential to contribute to these efforts, since they are likely to be free of response bias.

Peer Reviewers for theJournal of the American Board of Family Medicinein 2023

Peer Reviewers for the<i>Journal of the American Board of Family Medicine</i>in 2023

The Gender Wage Gap Among Early-Career Family Physicians.

Numerous studies have documented salary differences between male and female physicians. For many specialties, this wage gap has been explored by controlling for measurable factors that influence pay such as productivity, work-life balance, and practice patterns. In family medicine where practice activities differ widely between physicians, it is important to understand what measurable factors may be contributing to the gender wage gap, so that employers and policymakers and can address unjust disparities. We used data from the 2017 to 2020 American Board of Family Medicine (ABFM) National Graduate Survey (NGS) which is administered to family physicians 3 years after residency (n = 8608; response rate = 63.9%, 56.2% female). The survey collects clinical income and practice patterns. Multiple linear regression analysis was performed, which included variables on hours worked, degree type, principal professional activity, rural/urban, and region. Although early-career family physician incomes averaged $225,278, female respondents reported incomes that were $43,566 (17%) lower than those of male respondents (P = .001). Generally, female respondents tended toward lower-earning principal professional activities and US regions; worked fewer hours (2.9 per week); and tended to work more frequently in urban settings. However, in adjusted models, this gap in income only fell to $31,804 (13% lower than male respondents, P = .001). Even after controlling for measurable factors such as hours worked, degree type, principal professional activity, population density, and region, a significant wage gap persists. Interventions should be taken to eliminate gender bias in wage determinations for family physicians.

Machine Learning to Identify Clusters in Family Medicine Diplomate Motivations and Their Relationship to Continuing Certification Exam Outcomes: Findings and Potential Future Implications.

The potential for machine learning (ML) to enhance the efficiency of medical specialty boards has not been explored. We applied unsupervised ML to identify archetypes among American Board of Family Medicine (ABFM) Diplomates regarding their practice characteristics and motivations for participating in continuing certification, then examined associations between motivation patterns and key recertification outcomes. Diplomates responding to the 2017 to 2021 ABFM Family Medicine continuing certification examination surveys selected motivations for choosing to continue certification. We used Chi-squared tests to examine difference proportions of Diplomates failing their first recertification examination attempt who endorsed different motivations for maintaining certification. Unsupervised ML techniques were applied to generate clusters of physicians with similar practice characteristics and motivations for recertifying. Controlling for physician demographic variables, we used logistic regression to examine the effect of motivation clusters on recertification examination success and validated the ML clusters by comparison with a previously created classification schema developed by experts. ML clusters largely recapitulated the intrinsic/extrinsic framework devised by experts previously. However, the identified clusters achieved a more equal partitioning of Diplomates into homogenous groups. In both ML and human clusters, physicians with mainly extrinsic or mixed motivations had lower rates of examination failure than those who were intrinsically motivated. This study demonstrates the feasibility of using ML to supplement and enhance human interpretation of board certification data. We discuss implications of this demonstration study for the interaction between specialty boards and physician Diplomates.

Variation in Practice Patterns of Early- and Later-Career Family Physicians.

Understanding how physicians' practice patterns change over a career is important for workforce and medical education planning. This study examined trends in self-reported practice activity among early- and later-career stage family physicians (FPs). Data on early career FPs came from the American Board of Family Medicine's National Graduate Survey (NGS) and on later career FPs from its Continuous Certification Questionnaire (CCQ). Both cohorts could complete the Practice Demographic Survey (PDS) 3 years later. Longitudinal cohorts were from 2016 to 2019 and 2017 to 2020, respectively. All surveys included identical items on scope of practice, practice type, organization, and location. We characterized physicians as outpatient continuity only, outpatient and inpatient care (mixed practice), and no outpatient continuity (for example, hospitalist). We conducted repeated cross-sectional and longitudinal analysis of practice type. Our sample included 8,492 NGS and 30,491 CCQ FPs. In both groups, the vast majority provided outpatient continuity of care (77% to 81%). Approximately 25% of NGS had a mixed practice compared with approximately 16% of the CCQ group. The percent of FPs who had a mixed practice declined in both groups (34.21% to 27.10% and 23.88% to 19.33%). In both groups, physicians with higher odds of leaving mixed practice were in metropolitan counties or changed practice types. Although early-career FPs more frequently reported providing both inpatient and outpatient care and serving as hospitalists compared with later-career FPs, both groups had a decline in frequency of providing mixed practice. This change after only 3 years in practice has significant implications for patient care and medical education.

Impact of Service-Based Student Loan Repayment Program on the Primary Care Workforce.

As the average level of medical education indebtedness rises, physicians look to programs such as Public Service Loan Forgiveness (PSLF) and National Health Service Corps (NHSC) to manage debt burden. Both represent service-dependent loan repayment programs, but the requirements and program outcomes diverge, and assessing the relative uptake of each program may help to inform health workforce policy decisions. We sought to describe variation in the composition of repayment program participant groups and measure relative impact on patient access to care. In this bivariate analysis, we analyzed data from 10,677 respondents to the American Board of Family Medicine's National Graduate Survey to study differences in loan repayment program uptake as well as the unique participant demographics, scope of practice, and likelihood of practicing with a medically underserved or rural population in each program cohort. The rate of PSLF uptake tripled between 2016 and 2020, from 7% to 22% of early career family physicians, while NHSC uptake remained static at 4% to 5%. Family physicians reporting NHSC assistance were more likely than those reporting PSLF assistance to come from underrepresented groups, demonstrated a broader scope of practice, and were more likely to practice in rural areas (23.3% vs 10.8%) or whole-county Health Professional Shortage Areas (12.5% vs 3.7%) and with medically underserved populations (82.2% vs 24.2%). Although PSLF supports family physicians intending to work in public service, their peers who choose NHSC are much more likely to work in underserved settings. Our findings may prompt a review of the goals of service loan forgiveness programs with potential to better serve health workforce needs.

Maternity Access in Rural America: The Role of Family Physicians in Providing Access to Cesarean Sections.

As an increasing number of rural hospitals close their maternity care units, many of the approximately 28 million reproductive-age women living in rural America do not have local access to obstetric services. We sought to describe the characteristics and distribution of cesarean section-providing family physicians who may provide critical services in maintaining obstetric access in rural hospitals. Using a cross-sectional study design, we linked data from the 2017 to 2022 American Board of Family Medicine's Continuting Certification Questionnaire on provision of cesarean sections as primary surgeon and practice characteristics to geographic data. Logistic regression determined associations with provision of cesarean sections. Of 28,526 family physicians, 589 (2.1%) provided cesarean sections as primary surgeon. Those who provided cesarean sections were more likely to be male (odds ratio (OR) = 1.573, 95% confidence limits (CL) 1.246-1.986), and work in rural health clinics (OR = 2.157, CL 1.397-3.330), small rural counties (OR = 4.038, CL 1.887-8.642), and in counties without obstetrician/gynecologists (OR = 2.163, CL 1.440-3.250). Although few in number, family physicians who provide cesarean sections as primary surgeon disproportionately serve rural communities and counties without obstetrician/gynecologists, suggesting that they provide access to obstetric services in these communities. Policies that support family physician training in cesarean sections and facilitate credentialing of trained family physicians could reverse the trend of closing obstetric units in rural communities and reduce disparities in maternal and infant health outcomes.

Peer Reviewers for theJournal of the American Board of Family Medicinein 2022

Peer Reviewers for the<i>Journal of the American Board of Family Medicine</i>in 2022

Claiming APA and Psychiatry’s Role in the Future of Mental Health

Claiming APA and Psychiatry’s Role in the Future of Mental Health

Variation in Family Physicians' Experiences Across Different Electronic Health Record Platforms: a Descriptive Study.

Electronic health records (EHRs) have been connected to excessive workload and physician burnout. Little is known about variation in physician experience with different EHRs, however. To analyze variation in reported usability and satisfaction across EHRs. Internet-based survey available between December 2021 and October 2022 integrated into American Board of Family Medicine (ABFM) certification process. ABFM-certified family physicians who use an EHR with at least 50 total responding physicians. Self-reported experience of EHR usability and satisfaction. We analyzed the responses of 3358 physicians who used one of nine EHRs. Epic, athenahealth, and Practice Fusion were rated significantly higher across six measures of usability. Overall, between 10 and 30% reported being very satisfied with their EHR, and another 32 to 40% report being somewhat satisfied. Physicians who use athenahealth or Epic were most likely to be very satisfied, while physicians using Allscripts, Cerner, or Greenway were the least likely to be very satisfied. EHR-specific factors were the greatest overall influence on variation in satisfaction: they explained 48% of variation in the probability of being very satisfied with Epic, 46% with eClinical Works, 14% with athenahealth, and 49% with Cerner. Meaningful differences exist in physician-reported usability and overall satisfaction with EHRs, largely explained by EHR-specific factors. User-centric design and implementation, and robust ongoing evaluation are needed to reduce physician burden and ensure excellent experience with EHRs.

Analyzing trends in treatment of acne vulgaris in pregnancy: a retrospective study.

Analyzing trends in treatment of acne vulgaris in pregnancy: a retrospective study.

Differences in Physician Performance and Self-rated Confidence on High- and Low-Stakes Knowledge Assessments in Board Certification.

Evidence links assessment to optimal learning, affirming that physicians are more likely to study, learn, and practice skills when some form of consequence ("stakes") may result from an assessment. We lack evidence, however, on how physicians' confidence in their knowledge relates to performance on assessments, and whether this varies based on the stakes of the assessment. Our retrospective repeated-measures design compared differences in patterns of physician answer accuracy and answer confidence among physicians participating in both a high-stakes and a low-stakes longitudinal assessment of the American Board of Family Medicine. After 1 and 2 years, participants were more often correct but less confident in their accuracy on a higher-stakes longitudinal knowledge assessment compared with a lower-stakes assessment. There were no differences in question difficulty between the two platforms. Variation existed between platforms in time spent answering questions, use of resources to answer questions, and perceived question relevance to practice. This novel study of physician certification suggests that the accuracy of physician performance increases with higher stakes, even as self-reported confidence in their knowledge declines. It suggests that physicians may be more engaged in higher-stakes compared with lower-stakes assessments. With medical knowledge growing exponentially, these analyses provide an example of the complementary roles of higher- and lower-stakes knowledge assessment in supporting physician learning during continuing specialty board certification.

Academic Achievement and Competency in Rural and Urban Family Medicine Residents.

The quality of training in rural family medicine (FM) residencies has been questioned. Our objective was to assess differences in academic performance between rural and urban FM residencies. We used American Board of Family Medicine (ABFM) data from 2016-2018 residency graduates. Medical knowledge was measured by the ABFM in-training examination (ITE) and Family Medicine Certification Examination (FMCE). The milestones included 22 items across six core competencies. We measured whether residents met expectations on each milestone at each assessment. Multilevel regression models determined associations between resident and residency characteristics milestones met at graduation, FMCE score, and failure. Our final sample was 11,790 graduates. First-year ITE scores were similar between rural and urban residents. Rural residents passed their initial FMCE at a lower rate than urban residents (96.2% vs 98.9%) with the gap closing upon later attempts (98.8% vs 99.8%). Being in a rural program was not associated with a difference in FMCE score but was associated with higher odds of failure. Interactions between program type and year were not significant, indicating equal growth in knowledge. The proportions of rural vs urban residents who met all milestones and each of six core competencies were similar early in residency but diverged over time with fewer rural residents meeting all expectations. We found small, but persistent differences in measures of academic performance between rural- and urban-trained FM residents. The implications of these findings in judging the quality of rural programs are much less clear and warrant further study, including their impact on rural patient outcomes and community health.

The Association Between Length of Training and Family Medicine Residents’ Clinical Knowledge: A Report From the Length of Training Pilot Study

The associations between training length and clinical knowledge are unknown. We compared family medicine in-training examination (ITE) scores among residents who trained in 3- versus 4-year programs and to national averages over time. In this prospective case-control study, we compared the ITE scores of 318 consenting residents in 3-year programs to 243 who completed 4 years of training between 2013 through 2019. We obtained scores from the American Board of Family Medicine. The primary analyses involved comparing scores within each academic year according to length of training. We used multivariable linear mixed effects regression models adjusted for covariates. We performed simulation models to predict ITE scores after 4 years of training among residents who underwent only 3 years of training. At baseline postgraduate year-1 (PGY1), the estimated mean ITE scores were 408.5 for 4-year programs and 386.5 for 3-year programs, a 21.9 point difference (95% CI=10.1-33.8). At PGY2 and PGY3, 4-year programs scored 15.0 points higher and 15.6 points higher, respectively. When extrapolating an estimated mean ITE score for 3-year programs, 4-year programs would still score 29.4 points higher (95% CI=15.0-43.8). Our trend analysis revealed those in 4-year programs had a slightly lesser slope increase compared to 3-year programs in the first 2 years. Their drop-off in ITE scores is less steep in later years, though these differences were not statistically significant. While we found significantly higher absolute ITE scores in 4 versus 3-year programs, these increases in PGY2, PGY3 and PGY4 may be due to initial differences in PGY1 scores. Additional research is needed to support a decision to change the length of family medicine training.