AUA Guideline Committee Members Determine Quality of Artificial Intelligence‒Generated Responses for Female Stress Urinary Incontinence.

Abstract

Stress urinary incontinence (SUI) affects countless women worldwide. Given ChatGPT's rising ubiquity, patients may turn to the platform for SUI advice. Our objective was to evaluate the quality of clinical information about SUI from the ChatGPT platform. The most-asked patient questions regarding SUI were derived from patient materials from societal websites and forums, and queried using ChatGPT 3.5. The responses from ChatGPT were compiled into a survey and disseminated to 3 AUA guideline committee members who developed the Surgical Management of Female SUI guidelines. They were asked to grade responses on reliability, understandability, quality, and actionability using DISCERN and Patient Education Materials Assessment Tool standardized questionnaires. Accuracy was assessed with a 4-point Likert scale and readability using Flesch Reading Ease score. The overall material was rated as moderate to moderately high quality (DISCERN = 3.73/5) with potentially important but no serious shortcomings. Reliability and quality were reported to be 63% and 75%. Understandability was 89%, actionability 18%, and accuracy 88%. All question domains were rated at moderate or better. Actionability was poor in all domains. Every response was "hard to read" translating to a college graduate reading level. The urologic community should critically evaluate this platform's output if patients are to use it for adjunctive medical guidance. AUA committee members, who are experts in the field, rate ChatGPT-produced responses on SUI as moderate to moderately high quality, moderate reliability, excellent understandability, and poor actionability utilizing standardized questionnaires. The reading level of the material was advanced, which is an area of potential improvement to make generated responses more comprehensible.