Can ChatGPT help patients understand their andrological diseases?

Abstract

We aimed to assess the reliability of Chat Generative Pre-training Transformer (ChatGPT)'s andrology information and its suitability for informing patients and medical students accurately about andrology topics. We presented a series of systematically organized frequently asked questions on andrology topics and sentences containing strong recommendations from the European Association of Urology (EAU) Guideline to ChatGPT-3.5 and 4.0 as questions. These questions encompassed Male Hypogonadism, Erectile Dysfunction and Sexual Desire Disorder, Disorders of Ejaculation, Penile Curvature and Penile Size Abnormalities, Priapism, and Male Infertility. Two expert urologists independently evaluated and assigned scores ranging from 1 to 4 to each response based on its accuracy, with the following ratings: (1) Completely true, (2) Accurate but insufficient, (3) A mixture of accurate and misleading information, and (4) Completely false. A total of 120 questions were included in the study. Among these questions, 50.0% received a grade of 1 (completely correct) (55.4% for 4.0 version). The combined rate of correct answers (grades 1 and 2) was 85.2% for frequently asked questions (88.8% for 4.0 version) and 81.5% for questions obtained from the guideline. The rate of completely incorrect answers (grade 4) was 1.8% for frequently asked questions (0% for 4.0 version) and 5.2% for questions based on strong recommendations. The response rate of version 4.0 to questions created from sentences containing strong recommendations from the EAU guideline was the same as version 3.5. ChatGPT provided satisfactory answers to the questions asked, although some responses lacked completeness. It may be beneficial to utilize ChatGPT under the guidance of a urologist to enhance patients' comprehension of their andrology issues.