Accuracy of Spanish and English-generated ChatGPT responses to commonly asked patient questions about labor epidurals: A survey-based study among bilingual obstetric anesthesia experts

Abstract

BackgroundLarge language models (LLMs), of which ChatGPT is the most wellknown, are now available to patients to seek medical advice in various languages. However, the accuracy of the information utilized to train these models remains unknown. MethodsTen commonly asked questions regarding labor epidurals were translated from English to Spanish, and all twenty questions were entered into ChatGPT version 3.5. The answers were transcribed. A survey was then sent to ten fellowship-trained obstetric anesthesiologists to assess the accuracy of these answers utilizing a 5-point Likert scale. ResultsOverall, the accuracy scores for the ChatGPT-generated answersin Spanish were lower than for the English answerswith a median score (IQR) of 34 (33–36.5) versus40.5 (39–44.3), respectively (P value = 0.02). Answers to twoquestions were sored significantly lower: “Do epidurals prolong labor?” (2 (IQR 2–2.5) versus 4 (IQR 4–4.5), P value 0.03) and “Do epidurals increase the risk of needing cesarean delivery?” (3(IQR 2–4) versus 4 (IQR 4–5); P value 0.03). There was a strong agreement that answers to the question “Do epidurals cause autism” were accurate in both Spanish and English. ConclusionChatGPT-generated answers in Spanish to ten questions about labor epidurals scored lowerfor accuracythananswers generated in English, particularly regarding the effect of labor epidurals on labor course and mode of delivery. This disparity in ChatGPT-generated information may extend already-known health inequities among non-English-speaking patients and perpetuate misinformation.