Frequently Asked Questions Research Articles

FAQ-Gen: An automated system to generate domain-specific FAQs to aid content comprehension

Frequently Asked Questions (FAQs) refer to the most common inquiries about specific content. They serve as content comprehension aids by simplifying topics and enhancing understanding through succinct presentation of information. In this paper, we address FAQ generation as a well-defined Natural Language Processing task through the development of an end-to-end system leveraging text-to-text transformation models. We present a literature review covering traditional question-answering systems, highlighting their limitations when applied directly to the FAQ generation task. We propose a system capable of building FAQs from textual content tailored to specific domains, enhancing their accuracy and relevance. We utilise self-curated algorithms to obtain an optimal representation of information to be provided as input and also to rank the question-answer pairs to maximise human comprehension. Qualitative human evaluation showcases the generated FAQs as well-constructed and readable while also utilising domain-specific constructs to highlight domain-based nuances and jargon in the original content.

Is the information provided by large language models valid in educating patients about adolescent idiopathic scoliosis? An evaluation of content, clarity, and empathy : The perspective of the European Spine Study Group.

Large language models (LLM) have the potential to bridge knowledge gaps in patient education and enrich patient-surgeon interactions. This study evaluated three chatbots for delivering empathetic and precise adolescent idiopathic scoliosis (AIS) related information and management advice. Specifically, we assessed the accuracy, clarity, and relevance of the information provided, aiming to determine the effectiveness of LLMs in addressing common patient queries and enhancing their understanding of AIS. We sourced 20 webpages for the top frequently asked questions (FAQs) about AIS and formulated 10 critical questions based on them. Three advanced LLMs-ChatGPT 3.5, ChatGPT 4.0, and Google Bard-were selected to answer these questions, with responses limited to 200 words. The LLMs' responses were evaluated by a blinded group of experienced deformity surgeons(members of the European Spine Study Group) from seven Europeanspine centers. A pre-established 4-level rating system from excellent to unsatisfactory was used with a further rating for clarity, comprehensiveness, and empathy on the 5-point Likert scale. If not rated 'excellent', the raters were asked to report the reasons for their decision for each question. Lastly, raters were asked for their opinion towards AI in healthcare in general in six questions. The responses among all LLMs were 'excellent' in 26% of responses, with ChatGPT-4.0 leading (39%), followed by Bard (17%). ChatGPT-4.0 was rated superior to Bard and ChatGPT 3.5 (p = 0.003). Discrepancies among raters were significant (p < 0.0001), questioning inter-rater reliability. No substantial differences were noted in answer distribution by question (p = 0.43). The answers on diagnosis (Q2) and causes (Q4) of AIS were top-rated. The most dissatisfaction was seen in the answers regarding definitions (Q1) and long-term results (Q7). Exhaustiveness, clarity, empathy, and length of the answers were positively rated (> 3.0 on 5.0) and did not demonstrate any differences among LLMs. However, GPT-3.5 struggled with language suitability and empathy, while Bard's responses were overly detailed and less empathetic. Overall, raters found that 9% of answers were off-topic and 22% contained clear mistakes. Our study offers crucial insights into the strengths and weaknesses of current LLMs in AIS patient and parent education, highlighting the promise of advancements like ChatGPT-4.oand Gemini alongside the need for continuous improvement in empathy, contextual understanding, and language appropriateness.

Evaluation of validity and reliability of AI Chatbots as public sources of information on dental trauma.

This study aimed to assess the validity and reliability of AI chatbots, including Bing, ChatGPT 3.5, Google Gemini, and Claude AI, in addressing frequently asked questions (FAQs) related to dental trauma. A set of 30 FAQs was initially formulated by collecting responses from four AI chatbots. A panel comprising expert endodontists and maxillofacial surgeons then refined these to a final selection of 20 questions. Each question was entered into each chatbot three times, generating a total of 240 responses. These responses were evaluated using the Global Quality Score (GQS) on a 5-point Likert scale (5: strongly agree; 4: agree; 3: neutral; 2: disagree; 1: strongly disagree). Any disagreements in scoring were resolved through evidence-based discussions. The validity of the responses was determined by categorizing them as valid or invalid based on two thresholds: a low threshold (scores of ≥ 4 for all three responses) and a high threshold (scores of 5 for all three responses). A chi-squared test was used to compare the validity of the responses between the chatbots. Cronbach's alpha was calculated to assess the reliability by evaluating the consistency of repeated responses from each chatbot. The results indicate that the Claude AI chatbot demonstrated superior validity and reliability compared to ChatGPT and Google Gemini, whereas Bing was found to be less reliable. These findings underscore the need for authorities to establish strict guidelines to ensure the accuracy of medical information provided by AI chatbots.

The utility of ChatGPT in gender-affirming mastectomy education

IntroductionThe integration of AI such as ChatGPT in medicine has been showing promise in enhancing patient education. Gender-affirming mastectomy (GAM) is a surgical procedure designed to help individuals transition to their self-identified gender, playing a crucial role in mitigating psychological distress for many transmasculine and non-binary (TNB) patients. With increased demand and attention towards GAM, plastic and reconstructive surgeons may rely on AI-driven chatbots as an accessible, accurate, and patient-driven model for relevant details on this procedure. Specific Aim(s)This study aimed to assess the quality and readability of information provided by ChatGPT in response to frequently asked questions (FAQs) related to GAM. MethodsInspired by online forums and physician websites, 10 frequently asked questions (FAQs) about pre- and postoperative topics were submitted to ChatGPT and assessed using validated readability score measures and expert interpretation. ResultsThe average readability score was 16.0 ± 1, indicating a college or graduate reading level. Mean accuracy, comprehensiveness, and danger scores were 8.8 ± 0.5, 7.8 ± 0.7, and 2.2 ± 0.4, respectively. Although physicians appreciated ChatGPT's tone, patient autonomy, and advice to seek professional medical and mental help, they also cited instance of generic information, misinformation, support of debated techniques, and pathologization of gender dysphoria. ConclusionEven with its promise in providing accurate and comprehensive information on GAM, ChatGPT’s current limitations suggest caution as a supplementary tool to physician consultation.

Assessing the Clinical Appropriateness and Practical Utility of ChatGPT as an Educational Resource for Patients Considering Minimally Invasive Spine Surgery.

Introduction Minimally invasive spine surgery (MISS) has evolved over the last three decades as a less invasive alternative to traditional spine surgery, offering benefits such as smaller incisions, faster recovery, and lower complication rates. With patients frequently seeking information about MISS online, the comprehensibility and accuracy of this information are crucial. Recent studies have shown that much of the online material regarding spine surgery exceeds the recommended readability levels, making it difficult for patients to understand. This study explores the clinical appropriateness and readability of responses generated by Chat Generative Pre-Trained Transformer (ChatGPT) to frequently asked questions (FAQs) about MISS. Methods A set of 15 FAQs was formulated based on clinical expertise and existing literature on MISS. Each question was independently inputted into ChatGPT five times, and the generated responses were evaluated by three neurosurgery attendings for clinical appropriateness. Appropriateness was judged based on accuracy, readability, and patient accessibility. Readability was assessed using seven standardized readability tests, including the Flesch-Kincaid Grade Level and Flesch Reading Ease (FRE) scores. Statistical analysis was performed to compare readability scores across preoperative, postoperative, and intraoperative/technical question categories. Results The mean readability scores for preoperative, postoperative, and intraoperative/technical questions were 15±2.8, 16±3, and 15.7±3.2, respectively, significantly exceeding the recommended sixth- to eighth-grade reading level for patient education (p=0.017). Differences in readability across individual questions were also statistically significant (p<0.001). All responses required a reading level above 11th grade, with a majority indicating college-level comprehension. Although preoperative and postoperative questions generally elicited clinically appropriate responses, 50% of intraoperative/technical questions yielded either "inappropriate" or "unreliable" responses, particularly for inquiries about radiation exposure and the use of lasers in MISS. Conclusions While ChatGPT is proficient in providing clinically appropriate responses to certain FAQs about MISS, it frequently produces responses that exceed the recommended readability level for patient education. This limitation suggests that its utility may be confined to highly educated patients, potentially exacerbating existing disparities in patient comprehension. Future AI-based patient education tools must prioritize clear and accessible communication, with oversight from medical professionals to ensure accuracy and appropriateness. Further research comparing ChatGPT's performance with other AI models could enhance its application in patient education across medical specialties.

Does Introducing FAQs Boost Self-Service Government?—A Study of Local Government Websites

ABSTRACT Government websites remain a crucial means of communication with citizens, even in times of artificial intelligence. However, if not designed with usability in mind, users will call or e-mail to get answers, thereby wasting administrative resources. Little research exists on this issue of government efficiency. Drawing on self-efficacy theory, we address this gap and introduce Frequently Asked Questions (FAQs) to the websites of five Swiss local governments to see if self-service government increases. We find improvements only if administrations tell users about the FAQs on other channels. Thus, our study advances the debate on government efficiency in the digital space.

ChatGPT for Addressing Patient-centered Frequently Asked Questions in Glaucoma Clinical Practice

Large language models such as ChatGPT-3.5 are often used by the public to answer questions related to daily life, including health advice. This study evaluated the responses of ChatGPT-3.5 in answering patient-centred frequently asked questions (FAQs) relevant in glaucoma clinical practice. Prospective cross-sectional survey. Twelve experts across a range of clinical, education and research practices in optometry and ophthalmology. Over 200 patient-centric FAQs from authoritative professional society, hospital and advocacy websites were distilled and filtered into 40 questions across four themes: definition and risk factors, diagnosis and testing, lifestyle and other accompanying conditions, and treatment and follow-up. The questions were individually input into ChatGPT-3.5 to generate responses. The responses were graded by the twelve experts individually. A 5-point Likert scale (1 = strongly disagree; 5 = strongly agree) was used to grade ChatGPT-3.5 responses across four domains: coherency, factuality, comprehensiveness, and safety. Across all themes and domains, median scores were all 4 ("agree"). Comprehensiveness had the lowest scores across domains (mean 3.7±0.9), followed by factuality (mean 3.9±0.9), and coherency and safety (mean 4.1±0.8 for both). Examination of the individual 40 questions showed that 8 (20%), 17 (42.5%), 24 (60%) and 8 (20%) of the questions had average scores below 4 (i.e. below "agree") for the coherency, factuality, comprehensiveness and safety domains, respectively. Free-text comments by the experts highlighted omissions of facts and comprehensiveness (e.g. secondary glaucoma) and remarked on the vagueness of some responses (i.e. that the response did not account for individual patient circumstances). ChatGPT-3.5 responses to FAQs in glaucoma were generally agreeable in terms of coherency, factuality, comprehensiveness, and safety. However, areas of weakness were identified, precluding recommendations for routine use to provide patients with tailored counselling in glaucoma, especially with respect to development of glaucoma and its management.

Can ChatGPT answer patient questions regarding reverse shoulder arthroplasty?

IntroductionIn recent years, artificial intelligence (AI) has seen substantial progress in its utilization, with Chat Generated Pre-Trained Transformer (ChatGPT) is emerging as a popular language model. The purpose of this study was to test the accuracy and reliability of ChatGPT's responses to frequently asked questions (FAQ) pertaining to reverse shoulder arthroplasty (RSA). MethodsThe ten most common FAQs were queried from institution patient education websites. These ten questions were then input into the chatbot during a single session without additional contextual information. The responses were then critically analyzed by two orthopedic surgeons for clarity, accuracy, and the quality of evidence-based information using The Journal of the American Medical Association (JAMA) Benchmark criteria and the DISCERN score. The readability of the responses was analyzed using the Flesch-Kincaid Grade Level. ResultsIn response to the ten questions, the average DISCERN score was 44 (range 38–51). Seven responses were classified as fair and three were poor. The JAMA Benchmark criteria score was 0 for all responses. Furthermore, the average Flesch-Kincaid Grade Level was 14.35, which correlates to a college graduate reading level. ConclusionOverall, ChatGPT was able to provide fair responses to common patient questions. However, the responses were all written at a college graduate reading level and lacked reliable citations. The readability greatly limits its utility. Thus, adequate patient education should be done by orthopedic surgeons. This study underscores the need for patient education resources that are reliable, accessible, and comprehensible. Level of evidenceIV.

A Multi-Lingual Conversational AI Chatbot for Effective Educational Consultations: A Study of ACE-DS, University of Rwanda

The demand for real-time consultation services from organizations is increasing, leading to prolonged waiting times, primarily due to limited opportunities for face-to-face interactions and language barriers. This study addresses this challenge by leveraging Artificial Intelligence (AI), Natural Language Processing (NLP), and linguistic technologies to develop a multilingual conversational AI Chabot for managing educational consultation services, using the African Center of Excellence in Data Science (ACE-DS), University of Rwanda, as a case study. Information and frequently asked questions (FAQs) about ACE-DS were used to train a Deep Learning Gated Recurrent Units (GRUs) algorithm to power the Chabot. Language detection and translation APIs were integrated to facilitate seamless multilingual conversations. The result of user survey conducted revealed that over 60% of respondents expressed high satisfaction with the Chabot’s performance including grammar, efficiency, language preferences, and response quality. This study showcases the potential of AI particularly NLP in enhancing educational consultation services, providing a framework for efficient information acquisition.

Practical Answers to Frequently Asked Questions in Anterior Cervical Spine Surgery for Degenerative Conditions.

Surgical counseling enables shared decision making and optimal outcomes by improving patients' understanding about their pathologies, surgical options, and expected outcomes. Here, we aimed to provide practical answers to frequently asked questions (FAQs) from patients undergoing an anterior cervical diskectomy and fusion (ACDF) or cervical disk replacement (CDR) for the treatment of degenerative conditions. Patients who underwent primary one-level or two-level ACDF or CDR for the treatment of degenerative conditions with a minimum of 1-year follow-up were included. Data were used to answer 10 FAQs that were generated from author's experience of commonly asked questions in clinic before ACDF or CDR. A total of 395 patients (181 ACDF, 214 CDR) were included. (1, 2, and 3) Will my neck/arm pain and physical function improve? Patients report notable improvement in all patient-reported outcome measures. (4) Is there a chance I will get worse? 13% (ACDF) and 5% (CDR) reported worsening. (5) Will I receive a significant amount of radiation? Patients on average received a 3.7 (ACDF) and 5.5 mGy (CDR) dose during. (6) How long will I stay in the hospital? Most patients get discharged on postoperative day one. (7) What is the likelihood that I will have a complication? 13% (8% minor and 5% major) experienced in-hospital complications (ACDF) and 5% (all minor) did (CDR). (8) Will I need another surgery? 2.2% (ACDF) and 2.3% (CDR) of patients required a revision surgery. (9 & 10) When will I be able to return to work/driving? Most patients return to working (median of 16 [ACDF] and 14 days [CDR]) and driving (median of 16 [ACDF] and 12 days [CDR]). The answers to the FAQs can assist surgeons in evidence-based patient counseling.

Analyzing the performance of ChatGPT in answering inquiries about cervical cancer.

To analyze the knowledge of ChatGPT about cervical cancer (CC). Official websites of professional health institutes, and websites created by patients and charities underwent strict screening. Using CC-related keywords, common inquiries by the public and comments about CC were searched in social media applications with these data, a list of frequently asked questions (FAQs) was prepared. When preparing question about CC, the European Society of Gynecological Oncology (ESGO), European Society for Radiotherapy and Oncology (ESTRO), and European Society of Pathology (ESP) guidelines were used. The answers given by ChatGPT were scored according to the Global Quality Score (GQS). When all ChatGPT answers to FAQs about CC were evaluated with regard to GQS, 68 ChatGPT answers were classified as score 5, and none of ChatGPT answers for FAQs were scored as 2 or 1. Moreover, ChatGPT answered 33 of 53 (62.3%) CC-related questions based on ESGO, ESTRO, and ESP guidelines with completely accurate and satisfactory responses (GQS 5). In addition, eight answers (15.1%), seven answers (13.2%), four answers (7.5%), and one answer (1.9%) were categorized as GQS 4, GQS 3, GQS 2, and GQS 1, respectively. The reproducibility rate of ChatGPT answers about CC-related FAQs and responses about those guideline-based questions was 93.2% and 88.7%, respectively. ChatGPT had an accurate and satisfactory response rate for FAQs about CC with regards to GQS. However, the accuracy and quality of ChatGPT answers significantly decreased for questions based on guidelines.

Answers to frequently asked questions about the pulsar timing array Hellings and Downs curve

Abstract We answer frequently asked questions (FAQs) about the Hellings and Downs correlation curve—the ‘smoking-gun’ signature that pulsar timing arrays (PTAs) have detected gravitational waves (GWs). Many of these questions arise from inadvertently applying intuition about the effects of GWs on LIGO-like detectors to the case of pulsar timing, where not all of it applies. This is because Earth-based detectors, like LIGO and Virgo, have arms that are short (km scale) compared to the wavelengths of the GWs that they detect ( ≈ 10 2 –104 km). In contrast, PTAs respond to GWs whose wavelengths (tens of light-years) are much shorter than their arms (a typical PTA pulsar is hundreds to thousands of light-years from Earth). To demonstrate this, we calculate the time delay induced by a passing GW along an Earth-pulsar baseline (a ‘one-arm, one-way’ detector) and compare it in the ‘short-arm’ (LIGO-like) and ‘long-arm’ (PTA) limits. This provides qualitative and quantitative answers to many questions about the Hellings and Downs curve. The resulting FAQ sheet should help in understanding the ‘evidence for GWs’ recently announced by several PTA collaborations.

Frequently asked questions to the 2023 Obesity Medicine Association Position Statement on Compounded Peptides: A call for action

BackgroundIn 2023, the Obesity Medicine Association (OMA) published “Compounded peptides: An Obesity Medicine Association Position Statement.” Since this publication, the use of compounded peptides for the treatment of obesity has continued to evolve, leading to additional confusion and questions from obesity medicine clinicians and their patients. MethodsThis frequently asked questions (FAQ) document and “Call for Action” commentary is based upon the existing evidence and practical clinical experiences of the authors. ResultsThis FAQ is intended to provide insights beyond the original 2023 OMA Position Statement regarding the use of “compounded peptides” for treating obesity. Three obstacles impair patient access to highly effective peptide anti-obesity medications: insufficient production especially during times of high demand, high costs, and lack of clarity surrounding the role of compounded peptides. Solutions to enhance patient access to these medications lie within the existing legal and regulatory framework and Food and Drug Administration policies. Implementing these solutions necessitates dispelling misinformation and providing clear guidance on the appropriate prescribing and administration of compounded peptides, particularly during times of acknowledged shortage. ConclusionAmong stakeholders with aligned priorities, challenges can often be overcome by collaboration and communication. Towards the goal of providing patient-centered care, the OMA calls on applicable stakeholders (e.g., pharmaceutical companies, compounding pharmacy organizations, health insurance companies, and the Food and Drug Administration) to work collaboratively to achieve a consensus that improves patient access to safe anti-obesity medications. The purpose of this “Call to Action” is to ask stakeholders to provide clinicians and their patients clarity regarding the role of compounded peptide anti-obesity medications during times of FDA-acknowledged shortages. Finally, this FAQ review provides clinicians with a simple and practical checklist respective to the potential use of compounded peptides.

Is ChatGPT a Reliable Source of Patient Information on Asthma?

ChatGPT (OpenAI, San Francisco, CA, USA) is a novel artificial intelligence (AI) application that is used by millions of people, and the numbers are growing by the day. Because it has the potential to be a source of patient information, the study aimed to evaluate the ability of ChatGPT to answer frequently asked questions (FAQs) about asthma with consistent reliability, acceptability, and easy readability. We collected 30 FAQs about asthma from the Global Initiative for Asthma website. ChatGPT was asked each question twice, by two different users, to assess for consistency. The responses were evaluated by five board-certified internal medicine physicians for reliability and acceptability. The consistency of responses was determined by the differences in evaluation between the two answers to the same question. The readability of all responses was measured using the Flesch Reading Ease Scale (FRES), the Flesch-Kincaid Grade Level (FKGL), and the Simple Measure of Gobbledygook (SMOG). Sixty responses were collected for evaluation. Fifty-six (93.33%) of the responses were of good reliability. The average rating of the responses was 3.65 out of 4 total points. 78.3% (n=47) of the responses were found acceptable by the evaluators to be the only answer for an asthmatic patient. Only two (6.67%) of the 30 questions had inconsistent answers. The average readability of all responses was determined to be 33.50±14.37 on the FRES, 12.79±2.89 on the FKGL, and 13.47±2.38 on the SMOG. Compared to online websites, we found that ChatGPT can be a reliable and acceptable source of information for asthma patients in terms of information quality. However, all responses were of difficult readability, and none followed the recommended readability levels. Therefore, the readability of this AI application requires improvement to be more suitable for patients.

Assessing ChatGPT Ability to Answer Frequently Asked Questions About Essential Tremor.

Large-language models (LLMs) driven by artificial intelligence allow people to engage in direct conversations about their health. The accuracy and readability of the answers provided by ChatGPT, the most famous LLM, about Essential Tremor (ET), one of the commonest movement disorders, have not yet been evaluated. Answers given by ChatGPT to 10 questions about ET were evaluated by 5 professionals and 15 laypeople with a score ranging from 1 (poor) to 5 (excellent) in terms of clarity, relevance, accuracy (only for professionals), comprehensiveness, and overall value of the response. We further calculated the readability of the answers. ChatGPT answers received relatively positive evaluations, with median scores ranging between 4 and 5, by both groups and independently from the type of question. However, there was only moderate agreement between raters, especially in the group of professionals. Moreover, readability levels were poor for all examined answers. ChatGPT provided relatively accurate and relevant answers, with some variability as judged by the group of professionals suggesting that the degree of literacy about ET has influenced the ratings and, indirectly, that the quality of information provided in clinical practice is also variable. Moreover, the readability of the answer provided by ChatGPT was found to be poor. LLMs will likely play a significant role in the future; therefore, health-related content generated by these tools should be monitored.

Improving access to buprenorphine for rural veterans in a learning health care system.

To describe a learning health care system research process designed to increase buprenorphine prescribing for the treatment of opioid use disorder (OUD) in rural primary care settings within U.S. Department of Veterans Affairs (VA) treatment facilities. Using national administrative data from the VA Corporate Data Warehouse, we identified six rural VA health care systems that had improved their rate of buprenorphine prescribing within primary care from 2015 to 2020 (positive deviants). We conducted qualitative interviews with leaders, clinicians, and staff involved in buprenorphine prescribing within primary care from these sites to inform the design of an implementation strategy. Qualitative interviews to inform implementation strategy development. Interviews were audio-recorded, transcribed verbatim, and coded by a primary coder and secondary reviewer. Analysis utilized a mixed inductive/deductive approach. To develop an implementation strategy, we matched clinical needs identified within interviews with resources and strategies participants had utilized to address these needs in their own sites. Interview participants (n = 30) identified key clinical needs and strategies for implementing buprenorphine in rural, primary care settings. Common suggestions included the need for clinical mentorship or a consult service, buprenorphine training, and educational resources. Building upon interview findings and in partnership with a clinical team, we developed an implementation strategy composed of an engaging case-based training, an audit and feedback process, and educational resources (e.g., Buprenorphine Frequently Asked Questions, Rural Care Model Infographic). We describe a learning health care system research process that leveraged national administrative data, health care provider interviews, and clinical partnership to develop an implementation strategy to encourage buprenorphine prescribing in rural primary care settings.

Understanding How ChatGPT May Become a Clinical Administrative Tool Through an Investigation on the Ability to Answer Common Patient Questions Concerning Ulnar Collateral Ligament Injuries.

The consumer availability and automated response functions of chat generator pretrained transformer (ChatGPT-4), a large language model, poise this application to be utilized for patient health queries and may have a role in serving as an adjunct to minimize administrative and clinical burden. To evaluate the ability of ChatGPT-4 to respond to patient inquiries concerning ulnar collateral ligament (UCL) injuries and compare these results with the performance of Google. Cross-sectional study. Google Web Search was used as a benchmark, as it is the most widely used search engine worldwide and the only search engine that generates frequently asked questions (FAQs) when prompted with a query, allowing comparisons through a systematic approach. The query "ulnar collateral ligament reconstruction" was entered into Google, and the top 10 FAQs, answers, and their sources were recorded. ChatGPT-4 was prompted to perform a Google search of FAQs with the same query and to record the sources of answers for comparison. This process was again replicated to obtain 10 new questions requiring numeric instead of open-ended responses. Finally, responses were graded independently for clinical accuracy (grade 0 = inaccurate, grade 1 = somewhat accurate, grade 2 = accurate) by 2 fellowship-trained sports medicine surgeons (D.W.A, J.S.D.) blinded to the search engine and answer source. ChatGPT-4 used a greater proportion of academic sources than Google to provide answers to the top 10 FAQs, although this was not statistically significant (90% vs 50%; P = .14). In terms of question overlap, 40% of the most common questions on Google and ChatGPT-4 were the same. When comparing FAQs with numeric responses, 20% of answers were completely overlapping, 30% demonstrated partial overlap, and the remaining 50% did not demonstrate any overlap. All sources used by ChatGPT-4 to answer these FAQs were academic, while only 20% of sources used by Google were academic (P = .0007). The remaining Google sources included social media (40%), medical practices (20%), single-surgeon websites (10%), and commercial websites (10%). The mean (± standard deviation) accuracy for answers given by ChatGPT-4 was significantly greater compared with Google for the top 10 FAQs (1.9 ± 0.2 vs 1.2 ± 0.6; P = .001) and top 10 questions with numeric answers (1.8 ± 0.4 vs 1 ± 0.8; P = .013). ChatGPT-4 is capable of providing responses with clinically relevant content concerning UCL injuries and reconstruction. ChatGPT-4 utilized a greater proportion of academic websites to provide responses to FAQs representative of patient inquiries compared with Google Web Search and provided significantly more accurate answers. Moving forward, ChatGPT has the potential to be used as a clinical adjunct when answering queries about UCL injuries and reconstruction, but further validation is warranted before integrated or autonomous use in clinical settings.

Poster 372: ChatGPT is a Useful Tool for Patients with Acute Achilles Tendon Ruptures

Objectives: The use of web-based technology in healthcare delivery is ubiquitous. ChatGPT, a readily available artificial intelligence (AI) chatbot, has the potential to augment current practice. ChatGPT has shown rudimentary promise as a source of information for orthopaedic patients. The use of ChatGPT in foot and ankle surgery, particularly for acute Achilles tendon ruptures (AATR), has not been described. The purpose of this study was to assess the quality of information provided by ChatGPT in response to frequently asked questions about AATR. Methods: ChatGPT was posed 10 frequently asked questions about AATR in a conversational thread. Responses were recorded and no follow-up queries or clarification was obtained. The responses were subsequently graded “A,” “B,” “C,” or “F,” corresponding with “excellent response,” “adequate response needing mild clarification,” “inadequate response needing moderate clarification,” and “poor response needing severe clarification.” Results: Of the 10 responses, 2 were grade “A,” 6 were grade “B,” 1 was grade “C,” and 0 were grade “F.” Overall, ChatGPT provided good quality responses to the posed questions. Conclusions: ChatGPT responded well to questions regarding AATR, using simple language to provide responses representative of the current literature surrounding AATR. Overall, ChatGPT performed well providing good-to-great quality responses to frequently-asked questions about AATR. This study emphasized the potential role of ChatGPT as a source of information for patients who suffer AATR.

ChatGPT and Google Provide Mostly Excellent or Satisfactory Responses to the Most Frequently Asked Patient Questions Related to Rotator Cuff Repair

PurposeThe purpose of this study was to assess the differences in frequently asked questions (FAQs) and responses related to rotator cuff surgery between Google and ChatGPT. MethodsBoth Google and ChatGPT (version 3.5) were queried for the top 10 FAQs using the search term “rotator cuff repair.” Questions were categorized according to Rothwell’s Classification. In addition to questions and answers for each website, the source that the answer was pulled from was noted and assigned a category (academic, medical practice, etc.). Responses were also graded as “excellent response not requiring clarification,” (1) “satisfactory requiring minimal clarification,” (2) “satisfactory requiring moderate clarification,” (3) or “unsatisfactory requiring substantial clarification” (4) ResultsOverall, 30% of questions were similar between what Google and ChatGPT deemed to be the most FAQs. For questions from Google web search, the majority of answers came from medical practices (40%). For ChatGPT, the majority of answers were provided by academic sources (90%). For numerical questions, ChatGPT and Google provided similar responses for 30% of questions. For the majority of questions, both Google and ChatGPT responses were either “excellent” or “satisfactory requiring minimal clarification”. Google had one response rated as satisfactory requiring moderate clarification while ChatGPT had two responses rated as unsatisfactory. ConclusionBoth Google and ChatGPT offer mostly excellent or satisfactory responses to the most FAQs regarding rotator cuff repair. However, ChatGPT may provide inaccurate or even fabricated answers and associated citations.

Reliability of artificial intelligence chatbot responses to frequently asked questions in breast surgical oncology.

Artificial intelligence (AI)-driven chatbots, capable of simulating human-like conversations, are becoming more prevalent in healthcare. While this technology offers potential benefits in patient engagement and information accessibility, it raises concerns about potential misuse, misinformation, inaccuracies, and ethical challenges. This study evaluated a publicly available AI chatbot, ChatGPT, in its responses to nine questions related to breast cancer surgery selected from the American Society of Breast Surgeons' frequently asked questions (FAQ) patient education website. Four breast surgical oncologists assessed the responses for accuracy and reliability using a five-point Likert scale and the Patient Education Materials Assessment (PEMAT) Tool. The average reliability score for ChatGPT in answering breast cancer surgery questions was 3.98 out of 5.00. Surgeons unanimously found the responses understandable and actionable per the PEMAT criteria. The consensus found ChatGPT's overall performance was appropriate, with minor or no inaccuracies. ChatGPT demonstrates good reliability in responding to breast cancer surgery queries, with minor, nonharmful inaccuracies. Its answers are accurate, clear, and easy to comprehend. Notably, ChatGPT acknowledged its informational role and did not attempt to replace medical advice or discourage users from seeking input from a healthcare professional.