ChatGPT-3.5 and -4.0 Do Not Reliably Create Readable Patient Education Materials for Common Orthopaedic Upper- and Lower-Extremity Conditions

Abstract

Listen

Translate article

PurposeTo investigate if ChatGPT 3.5 and 4.0 can serve as a viable tool to create readable patient education materials (PEMs) for patients with common orthopaedic upper and lower extremity conditions. MethodsUtilizing ChatGPT versions 3.5 and 4.0, the authors asked the AI program a series of two questions pertaining to patient education for 50 common orthopaedic upper extremity and 50 common orthopaedic lower extremity pathologies. Two templated questions were created and used for all conditions. Readability scores were calculated using the Python library TextStat. Multiple readability test scores were generated, and a consensus reading level was created taking into account the results of 8 reading tests. ResultsChatGPT 3.5 produced only 2% and 4% of responses at the appropriate reading level for upper and lower extremity conditions, respectively, compared to 54% produced by ChatGPT 4.0 for both upper and lower extremity conditions (both p<0.0001). Following a priming phase, ChatGPT 3.5 did not produce any viable responses for either the upper or lower extremity conditions, compared to 64% for both upper and lower extremity conditions by ChatGPT 4.0 (both p<0.0001). Additionally, ChatGPT 4.0 was more successful than ChatGPT 3.5 in producing viable responses both pre- and post-priming phase based on all available metrics for reading level (all p<0.001), including Automated Readability scores, Coleman Liau Index, Dale Chall Score, Flesch Kincaid Grade, Flesch Reading Ease, Gunning Fog, Linsear Write Formula Score, and Simple Measure of Gobbledygook (SMOG) Index. ConclusionsOur results indicate that ChatGPT unreliably created readable PEMs for common orthopaedic upper and lower extremity conditions at the time of the study. Clinical RelevanceThe findings of this study suggest that ChatGPT, while constantly improving as evidenced by the advances from 3.5 to 4.0, should not be substituted for traditional methods of patient education at this time and in its current state may be utilized as a supplemental resource at the discretion of providers.