Abstract
To assess chat generative pre-trained transformer's (ChatGPT) performance onthe AmericanUrological Association Self-Assessment Study Program (AUA SASP) and stratify performance by question stem complexity. Questions from the 2021-2022 AUA SASP programwere administered to ChatGPT version 3 (ChatGPT-3). Questions were administered to the model utilizing a standardized prompt. The answer choice selected by ChatGPT was then used to answer the question stem in the AUA SASP program. ChatGPT was then prompted to assign a question stem order (first, second, third) to each question. The percentage of correctly answered questions was determined for each order level. All responses provided by ChatGPT were qualitatively assessed for appropriate rationale. A total of 268 questions were administered to ChatGPT. ChatGPT performed better on 2021 compared to the2022 AUA SASP question set, answering 42.3% versus 30.0% of questions correctly (P<.05). Hundred percentof answer explanations provided appropriate, relevant rationale regardless of whether the answer was correct. Further stratification included assessment by question order level. ChatGPT performed progressively better on the 2021 question set with decreasing order levels, with first-order questions reaching 53.8% (n=14). However, differences in proportions did not reach statistical significance (P>.05). ChatGPT answered many high-level questions correctly and provided a reasonable rationale for each answer choice. While ChatGPT was unable to answer numerous first-order questions, future language processing modellearning may lead to the optimization of its fund of knowledge. This may lead to the utilization of artificial intelligence like ChatGPT as an educational tool for urology trainees and professors.
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Disclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.