Evaluating the Accuracy and Utility of Large Language Models in Answering Common Contraception Questions [ID 2683633

Abstract

Listen

Translate article

INTRODUCTION: Traditional web resources that provide information on contraception produce varied results demanding user synthesis. However, direct answers provided by large language models (LLMs) can offer accessible contraception counseling. This study examines the efficacy of four LLMs, OpenAI's ChatGPT 4.0, OpenAI's ChatGPT 3.5, Google Bard, and Microsoft Bing, in answering common contraception questions. METHODS: Contraception questions and responses in a recent American Academy of Family Physicians (AAFP) review were used as reference text. We presented GPT-4.0, GPT-3.5, Bard, and Bing with six contraception questions from the AAFP text, ensuring no prior chat bias. Response formatting was standardized for analysis. Two ob-gyn physician reviewers (S.J. and A.A.), blinded to the source LLM, assessed responses via a 5-point Likert scale on the following criteria: response specificity, evidence basis, completeness, referrals to outside resources, and avoiding speaking in absolutes. Responses were analyzed using the Kruskal–Wallis test (SAS 9.4). RESULTS: The AAFP review scored highest in response specificity, evidence basis, and completeness. Large language models differed significantly on whether responses were evidence based (P<.0001), made referrals to outside resources (P<.001), or spoke in absolutes (P<.05). Notably, GPT-3.5 and Bard consistently provided referrals across all six questions, outperforming the AAFP text on this criteria. No significant differences were found across LLMs in response specificity or response completeness. CONCLUSION: Despite promising LLM-generated responses, the AAFP review outperformed LLMs in critical areas including evidence basis and completeness. Nonetheless, LLMs, especially GPT-3.5 and Bard, demonstrated strong potential as supplementary resources for reproductive health education.