(376) High Capability of ChatGPT to Answer Common Patient Questions on Vasectomy Reviewed by Sexual Medicine Specialists

Abstract

Abstract Introduction Novel technologies utilizing artificial intelligence (AI) have shown their potential to reduce provider workload and improve task efficiency in certain clinical settings. ChatGPT is an AI-powered chatbot which utilizes natural language processing (NLP) technology to provide human-like responses to open-ended questions posed by users. Recent studies have shown ChatGPT’s ability to provide diabetes counseling in lieu of trained educators, and effectively communicate technical research papers in layman’s terms to patients. Vasectomy is a common urologic clinic procedure that can be a source of anxiety for patients, many of whom desire the opportunity for further pre-operative counseling. Objective The objective of this preliminary study was to evaluate the ability of ChatGPT to respond to common patient questions on vasectomy encountered by sexual medicine specialists. Methods We composed a list of 20 questions commonly asked by patients pertaining to vasectomy. The questions were submitted to ChatGPT Plus with the prompt: “Answer the following questions in 50 words or less”. 3 fellowship-trained sexual medicine specialists then rated the accuracy of ChatGPT’s responses on a scale from 1-5 (1 representing a completely inaccurate response and 5 representing a response in line with what they would recommend to patients). Scores were tabulated and summarized (Table 1). Descriptive statistics were performed. Weighted Cohen’s kappa was calculated to assess for interrater reliability. Results Median score across all 20 responses from 3 raters was 4.1 (IQR 1.3). The most common rating was 5, indicating a high level of agreement between ChatGPT’s responses and those of sexual medicine specialists. The lowest score assigned was 2 (indicating poor accuracy), and was received by 6 responses. Weighted Cohen’s kappa was 0.49, implying moderate agreement amongst raters. Overall, ChatGPT was able to provide relatively accurate and concise responses to these common questions. Conclusions Our results demonstrate the capability of using ChatGPT as a potentially reliable source of information for patients desiring vasectomy counseling. Given the high clinical volume faced by many vasectomy providers, ChatGPT could possibly improve patient satisfaction and reduce provider burden. Further collection of real-world data is needed to better assess the accuracy and reliability of ChatGPT’s responses. Sexual medicine specialists should be aware of the strengths and limitations of emerging AI technologies as a medium for providing accurate health information to patients. Disclosure Any of the authors act as a consultant, employee or shareholder of an industry for: Fellow, Posterity Health, Coloplast, Hims, Genentech.