Question Classification Research Articles

Advancing AI in oncology: A performance comparison of ChatGPT-4o and ChatGPT-o1 in neuroendocrine tumor clinical decision making.

e13730 Background: Artificial intelligence (AI) is increasingly integrated into clinical decision-making, offering potential support in complex oncologic management. While ChatGPT (developed by OpenAI, San Francisco, CA) has demonstrated utility in various medical applications, its ability to accurately navigate guideline-based recommendations for neuroendocrine tumors (NETs) remains unexplored. Objective: This study aimed to assess the comparative performance of GPT-4o and GPT-o1 in providing guideline-based recommendations for NET management, utilizing the National Comprehensive Cancer Network (NCCN) guidelines as a reference standard. Methods: A total of 43 clinical questions were systematically derived from the NCCN guidelines, covering key decision-making domains: treatment, surveillance, and diagnostics. Both GPT-4o and GPT-o1 were queried, with responses independently evaluated by two physicians using a Likert scale (1-5), where: 5 = Fully correct, 4 = Mostly correct (minor omissions), 3 = Partially correct (lacking completion), 2 = Partially incorrect, 1 = Completely incorrect. Results: To explore the interpretative difference of ChatGPT-4o and ChatGPT-o1 on neuroendocrine tumor management, a Mann-Whitney U test was conducted. The analysis was statistically borderline (p = .050). The Mean Rank of ChatGPT-o1 was greater than that of ChatGPT-4o, thus GPT-o1 can be considered superior to GPT-4o. (Mann-Whitney U-test; Mean Rank = 47.16 vs 39.84) When stratified by question category, no statistically significant differences were observed in diagnostics, treatment, or surveillance. Conclusions: ChatGPT-o1 exhibited marginal improvements in the decision making for NET management while maintaining comparable performance to ChatGPT-4o in other domains. These findings suggest that iterative advancements in AI models may enhance their ability to support evidence-based clinical decision-making. Further studies are warranted to validate these findings in real-world oncologic practice.

Comparison of ChatGPT plus (version 4.0) and pretrained AI model (Orthopod) on orthopaedic in-training exam (OITE).

Comparison of ChatGPT plus (version 4.0) and pretrained AI model (Orthopod) on orthopaedic in-training exam (OITE).

Leveraging label semantics and meta-label refinement for multi-label question classification

Leveraging label semantics and meta-label refinement for multi-label question classification

In dogs with gastric dilatation volvulus (GDV) undergoing gastropexy, what is the rate of recurrence of GDV?

Question In dogs with a gastric dilatation volvulus (GDV) treated surgically and undergoing a gastropexy, what is the rate of recurrence of GDV? Clinical bottom line The category of research question was: Risk. The number and type of study designs that were critically appraised were: Sixteen studies were critically appraised. Two were prospective randomised controlled trials, two were prospective cohort studies, one was a case-control study, and eleven were case series. Critical appraisal of the selected papers meeting the inclusion criteria collectively provide zero/weak/moderate/strong evidence in terms of their experimental design and implementation: Weak. The outcomes reported are summarised as follows… A low rate of recurrence of GDV after gastropexy was consistently reported, although methodological limitations were recognised in all studies. In view of the strength of evidence and the outcomes from the studies the following conclusion is made… Overall, the studies provide weak evidence that the rate of recurrent GDV is low after gastropexy.

Can American Board of Surgery in Training Examinations be passed by Large Language Models? Comparative assessment of Gemini, Copilot, and ChatGPT.

ObjectiveThis study aimed to evaluate the performance of large language models (LLMs) in answering questions from the American Board of Surgery In-Training Examination (ABSITE).MethodsMultiple choice ABSITE Quiz was entered into the most popular LLMs as prompts. ChatGPT-4 (OpenAI), Copilot (Microsoft), and Gemini (Google) were used in the study. The research comprised 170 questions from 2017 to 2022, which were divided into four subgroups: Definitions, Biochemistry/Pharmaceutical, Case Scenario, and Treatment & Surgical Procedures. All questions were queried in LLMs, between October 1, 2024, and October 5, 2024. Correct answer rates of LLMs were evaluated.ResultsThe correct response rates for all questions were 79.4% for ChatGPT, 77.6% for Copilot, and 52.9% for Gemini, with Gemini significantly lower than both LLMs (P < 0.001). In the definition category, the correct response rates were 93.5% for ChatGPT, 90.3% for Copilot, and 64.5% for Gemini, with Gemini significantly lower (P = 0.005 and P = 0.015, respectively). In the Biochemistry/Pharmaceutical question category, the correct response rates were equal in all three groups (83.3%). In the Case Scenario category, the correct response rates were 76.3% in ChatGPT, 72.8% for Copilot, and 46.5% for Gemini, with Gemini significantly lower (P < 0.001). In the Treatment & Surgical Procedures category, the correct response rates were 69.2% for ChatGPT, 84.6% for Copilot, and 53.8% for Gemini. Although Gemini had the lowest accuracy, there was no statistically significant difference (P = 0.236).ConclusionIn the ABSITE Quiz, ChatGPT and Copilot had similar success, whereas Gemini was significantly behind.

Design of an inventory for the evaluation of attitudes and teaching practices on sustainable development in physical education

This pilot study examines the content validity and reliability of an inventory designed to assess physical education teachers’ attitudes, perceptions, and teaching practices related to sustainable development. The inventory was developed by adapting two validated instruments: the Attitudes and Perceptions on Education for Sustainable Development scale and the Assessment Tool for Physical Education for Sustainable Development. These instruments underwent content and face validity evaluation before piloting. The study aimed to assess the inventory’s internal consistency using Cronbach’s α coefficients and its stability through a test-retest procedure. Thirty-two Greek physical education teachers (8 males, 24 females; M = 48.85 years, SD = 6.18) completed the inventory twice, with a 20-day interval. Reliability analysis indicated that Cronbach’s α coefficients for individual factors ranged from 0.70 to 0.87, demonstrating adequate internal consistency. Test-retest results confirmed satisfactory stability in most question categories. Findings suggest that the inventory is a valid and reliable tool for assessing sustainable development-related attitudes and practices in physical education, with minor refinements recommended for specific items before broader application. This study highlights the need for robust assessment tools to evaluate education for sustainable development, particularly in physical education, where fostering sustainability-oriented attitudes is essential for promoting corresponding teaching practices.

Assessment of Learning: Student Teams Achievement Division Technique for Empowering Students in Problem-Solving Courses

Nowadays, there is a growing interest in group activities as interdisciplinary and teamwork raises effective learning among the students than the conventional teaching methodology. It has become a huge challenge to foster learning interest among students and engage them during long hours of problem-solving courses. This study explores the effects of Student Teams Achievement Divisions (STAD) as a cooperative learning technique in classroom teaching, on student learning experiences focusing on engineering students' learning experiences within team dynamics. Through an online survey question, (n=89) responses were collected that included four main categorical questions. The questions were divided into the following four input attributes such as ‘Student – Professor Interaction’, ‘Student – Team Interaction’, ‘Learning Tool’ and ‘Assessment Methodology’ and each of these inputs were further divided into sub attributes. Regression analysis and correlation was examined to analyze the association between the inputs and the Student Learning Experience as the output for the purpose of this research. The findings indicate that STAD promoted positive correlation with (R2 = 0.71) on student learning experience for the input attributes like ‘Student – Team Interaction’ and (R2 = 0.77) for ‘Assessment’ respectively where the students exhibited problem solving skills particularly in the context of the control systems course, suggesting that STAD methodology fosters a conducive environment for active learning and motivation within engineering education.

In sheep undergoing general anaesthesia does inclusion of medetomidine result in hypoxaemia?

Question In healthy adult female non-pregnant sheep undergoing general anaesthesia for research studies does the inclusion of intravenous medetomidine as part of the anaesthetic protocol cause hypoxaemia? Clinical bottom line The category of research question was: Treatment. The number and type of study designs that were critically appraised were: Four papers were available for critical appraisal. The quality of the evidence is weak. There were four experimental studies; three of which were cross-over studies and one study which was run in parallel with primary orthopaedic research. None of the studies were specifically focussed on the potential hypoxaemic effects of medetomidine. Critical appraisal of the selected papers meeting the inclusion criteria collectively provide zero/weak/moderate/strong evidence in terms of their experimental design and implementation: Weak. The outcomes reported are summarised as follows… Sheep across all four studies developed hypoxaemia (indicated by arterial oxygen tension; either PaO2 &lt; 80 mmHg/10 kPa when breathing room air, or a statistically significant fall in PaO2 compared with baseline, when breathing oxygen enriched gases), when medetomidine was administered intravenously and in combination with various drugs (i.e. midazolam, propofol, ketamine, halothane, and isoflurane). However, as the sheep were receiving various doses of medetomidine at various timepoints, different quantities of supplemental oxygen (if any), varying ventilatory management (two studies used mechanical ventilation and two studies allowed sheep to breathe spontaneously), and different agents were used to maintain anaesthesia, the clinical significance of the PaO2 values was difficult to assess. In view of the strength of evidence and the outcomes from the studies the following conclusion is made… In clinically healthy (judged by clinical examination) adult female non-pregnant sheep undergoing general anaesthesia (characterised by placement of an endotracheal tube and/or the use of anaesthetic induction agents i.e. barbiturates, ketamine, propofol), the weak evidence presented here suggests that use of intravenous medetomidine can be expected to cause hypoxaemia. However, hypoxaemia is variable and its clinical effects can be lessened with anaesthetic techniques such as the provision of supplemental oxygen.

Is ChatGPT a more academic source than google searches for patient questions about hip arthroscopy? An analysis of the most frequently asked questions.

Is ChatGPT a more academic source than google searches for patient questions about hip arthroscopy? An analysis of the most frequently asked questions.

In cats with cranial cruciate rupture are better long-term outcomes achieved by surgical or conservative management?

Question In cats diagnosed with isolated rupture of the cranial cruciate ligament, does surgical intervention result in improved long-term function, when compared with conservative management? Clinical bottom line The category of research question was: Treatment. The number and type of study designs that were critically appraised were: One retrospective cohort study comparing surgical management and conservative management. Critical appraisal of the selected papers meeting the inclusion criteria collectively provide zero/weak/moderate/strong evidence in terms of their experimental design and implementation: Weak. The outcomes reported are summarised as follows… This study evaluated and compared the long-term Feline Musculoskeletal Pain Index (FMPI) of patients with a cranial cruciate injury (39–87 months post injury). Patients that were managed conservatively showed statistically significant (P = 0.017) lower FMPI scores long term. In view of the strength of evidence and the outcomes from the studies the following conclusion is made… Conservative management can be considered in cats presenting with isolated cranial cruciate rupture; however, further research is required to establish the optimum treatment in these cases. Given the limitations of this study and the current lack of evidence, clinicians must rely on their clinical judgement at this point when managing this condition until further research is available.

Medical text classification model integrating medical entity label semantics

Automatic classification of medical questions is of great significance in improving the quality and efficiency of online medical services, and belongs to the task of intent recognition. Joint entity recognition and intent recognition perform better than single task models. Currently, most publicly available medical text intent recognition datasets lack entity annotation, and manual annotation of these entities requires a lot of time and manpower. To solve this problem, this paper proposes a medical text classification model, bidirectional encoder representation based on transformer-recurrent convolutional neural network-entity-label-semantics (BRELS), which integrates medical entity label semantics. This model firstly utilizes an adaptive fusion mechanism to absorb prior knowledge of medical entity labels, achieving local feature enhancement. Then in global feature extraction, a lightweight recurrent convolutional neural network (LRCNN) is used to suppress parameter growth while preserving the original semantics of the text. The ablation and comparison experiments are conducted on three public medical text intent recognition datasets to validate the performance of the model. The results show that F1 score reaches 87.34%, 81.71%, and 77.74% on each dataset, respectively. The results show that the BRELS model can effectively identify and understand medical terminology, thereby effectively identifying users' intentions, which can improve the quality and efficiency of online medical services.

Pre- Trained Language Models for Mental Health: An Empirical Study on Arabic Q&A Classification.

Background: Pre-Trained Language Models hold significant promise for revolutionizing mental health care by delivering accessible and culturally sensitive resources. Despite this potential, their efficacy in mental health applications, particularly in the Arabic language, remains largely unexplored. To the best of our knowledge, comprehensive studies specifically evaluating the performance of PLMs on diverse Arabic mental health tasks are still scarce. This study aims to bridge this gap by evaluating the performance of pre-trained language models in classifying questions and answers within the mental health care domain. Methods: We used the MentalQA dataset, which comprises Arabic Questions and Answers interactions related to mental health. Our experiments involved four distinct learning strategies: traditional feature extraction, using PLMs as feature extractors, fine-tuning PLMs, and employing prompt-based techniques with models, such as GPT-3.5 and GPT-4 in zero-shot and few-shot learning scenarios. Arabic-specific PLMs, including AraBERT, CAMelBERT, and MARBERT, were evaluated. Results: Traditional feature-extraction methods paired with Support Vector Machines (SVM) showed competitive performance, but PLMs outperformed them due to their superior ability to capture semantic nuances. In particular, MARBERT achieved the highest performance, with Jaccard scores of 0.80 for the question classification and 0.86 for the answer classification. Further analysis revealed that fine-tuning PLMs enhances their performance, and the size of the training dataset plays a critical role in model effectiveness. Prompt-based techniques, particularly few-shot learning with GPT-3.5, demonstrated significant improvements, increasing the accuracy of question classification by 12% and the accuracy of answer classification by 45%. Conclusions: The study demonstrates the potential of PLMs and prompt-based approaches to provide mental health support to Arabic-speaking populations, providing valuable tools for individuals seeking assistance in this field. This research advances the understanding of PLMs in mental health care and emphasizes their potential to improve accessibility and effectiveness in Arabic-speaking contexts.

Can ChatGPT-3.5 Be Used to Answer Questions Related to Pediatric Dentistry?

Aim: This study aimed to evaluate and compare the quality of chatGPT-3.5’s responses to frequently asked and undergraduate education-related questions provided by chatGPT-3.5 in the field of pediatric dentistry. Materials and Methods: Frequently asked questions by patients about procedures in pediatric dentistry and questions for educational purposes were generated to be asked on ChatGPT-3.5. The questions were divided into two categories consisting of 14 Patient Questions and 18 Educational Questions. Specific topics covered: Oral hygiene and preventive dentistry, patient management, dental trauma, restorative dentistry and pulp therapy, orthodontics, dental pathology and dental radiology. ChatGPT-3.5 answered these questions on April 15, 2024. Responses were scored on a 1 to 5 scale by pediatric dentists using a modified global quality scale. All statistical calculations were performed using IBM SPSS Statistics version 25.0. Results: Responses were scored by 25 pediatric dentists. The mean score for responses to Patient Questions was 4.21±0.53, while the mean score for responses to Education Questions was 3.44±0.71. The difference in scores between the two question categories and the subgroups of the categories was found to be statistically significant (p

Analysis of High-Dimensional Coordination in Human Movement Using Variance Spectrum Scaling and Intrinsic Dimensionality

Full-body movement involving multi-segmental coordination has been essential to our evolution as a species, but its study has been focused mostly on the analysis of one-dimensional data. The field is poised for a change by the availability of high-density recording and data sharing. New ideas are needed to revive classical theoretical questions such as the organization of the highly redundant biomechanical degrees of freedom and the optimal distribution of variability for efficiency and adaptiveness. In movement science, there are popular methods that up-dimensionalize: they start with one or a few recorded dimensions and make inferences about the properties of a higher-dimensional system. The opposite problem, dimensionality reduction, arises when making inferences about the properties of a low-dimensional manifold embedded inside a large number of kinematic degrees of freedom. We present an approach to quantify the smoothness and degree to which the kinematic manifold of full-body movement is distributed among embedding dimensions. The principal components of embedding dimensions are rank-ordered by variance. The power law scaling exponent of this variance spectrum is a function of the smoothness and dimensionality of the embedded manifold. It defines a threshold value below which the manifold becomes non-differentiable. We verified this approach by showing that the Kuramoto model obeys the threshold when approaching global synchronization. Next, we tested whether the scaling exponent was sensitive to participants’ gait impairment in a full-body motion capture dataset containing short gait trials. Variance scaling was highest in healthy individuals, followed by osteoarthritis patients after hip replacement, and lastly, the same patients before surgery. Interestingly, in the same order of groups, the intrinsic dimensionality increased but the fractal dimension decreased, suggesting a more compact but complex manifold in the healthy group. Thinking about manifold dimensionality and smoothness could inform classic problems in movement science and the exploration of the biomechanics of full-body action.

Verbal Information From Parents About Stillbirth: The VIPS Study (Phase One).

In many countries, a baby's cause of death (COD) following stillbirth is informed only by case notes and pathology investigations. However, parents' understanding of their baby's COD may inform or even change the COD diagnosis. We aimed to produce a standardised co-designed interview schedule to enable parents to contribute information to improve overall understanding of the causes of stillbirth. Consensus for the interview schedule was sought via a two-round modified Delphi study. We recruited internationally for panel membership comprising bereaved parents, clinicians, and researchers. In Round 1, each panellist provided up to five questions to ask bereaved parents. After collation into categories, Round 2 asked panellists to rate the importance of each question category on a four-point scale. Panellists (n = 126 Round 1, n = 75 Round 2) were mainly bereaved parents. In Round 1, 553 potential interview questions were generated. These were grouped into categories which were pregnancy experience, antenatal care, fetal wellbeing, maternal wellbeing, days Prior to stillbirth and perceived COD. These proposed questions and question categories were then put to panel members in Round 2. All categories achieved consensus for inclusion in the final interview schedule with positive consensus percentage scores ranging from 83% to 98%. Panel membership comprising mainly bereaved parents provided a clear mandate for questions parents want to be asked. The interview schedule will soon be trialled with recently bereaved parents at a tertiary-referral maternity health service. Findings from the study will inform future research on how to include parents' voices in COD determination.

Reinforcement Learning-Based Intent Classification of Chinese Questions About Respiratory Diseases

The intent classification of Chinese questions about respiratory diseases (CQRD) can not only promote the development of smart medical care, but also strengthen epidemic surveillance. The major core of the intent classification of CQRD is text representation. This paper studies how to utilize keywords to construct CQRD representation. Based on the characteristics of CQRD, we propose a keywords-based reinforcement learning model. In the reinforcement learning model based on keywords, we crafted a word frequency reward function to aid in generating the reward function and determining keyword categories. Simultaneously, to generate CQRD representations using keywords, we developed two models: keyword-driven LSTM (KD-LSTM) and keyword-driven GCN (KD-GCN). The KD-LSTM incorporates two methods: one based on word weights and the other based on category vectors. The KD-GCN employs keywords to construct a weight matrix for training. The method based on word weight achieves the best results on the CQRD_28000 dataset, which is 0.72% higher than the Bi-LSTM model. The method based on category vector outperforms the Bi-LSTM model in the CQRD_8000 dataset by 2.41%. The KD-GCN, although not attaining the optimal outcome, exhibited a superior performance of 3.12% compared to the GCN model. Both methods have significantly improved the classification results of minority classes.

Development and Assessment of a Patient-Reported Outcome Instrument for Gender-Affirming Care

There is an urgent need for a validated gender-affirming care-specific patient-reported outcome measure (PROM). To field test the GENDER-Q, a new PROM for gender-affirming care, in a large, international sample of transgender and gender diverse (TGD) adults and evaluate its psychometric properties. This international cross-sectional study was conducted among TGD adults aged 18 years and older who were seeking or had received gender-affirming care within the past 5 years at 21 clinical sites across Canada, the United States, the Netherlands, and Spain; participants were also recruited through community groups (eg, crowdsourcing platform, social media). The study was conducted between February 2022 and March 2024. Participants had to be capable of completing the instrument in English, Danish, Dutch, or French-Canadian. Eligible participants accessed an online REDCap survey to complete sociodemographic questions and questions about gender-affirming care they had received or sought (ie, to look, function, or feel masculine, feminine, gender fluid, or another way). Branching logic was used to assign relevant instrument scales. Rasch measurement theory (RMT) analysis was used to examine the fit of the observed data to the Rasch model for each scale. Test-retest reliability and hypothesis-based construct validity of instrument scales were examined. The hypothesis was that instrument scale scores would increase with better outcomes on corresponding categorical questions. A total of 5497 participants (mean [SD] age, 32.8 [12.3] years; 1837 [33.4%] men; 1307 [23.8%] nonbinary individuals; and 2036 [37.0%] women) completed the field test survey. Participants sought or had the following types of gender-affirming care: 2674 (48.6%) masculinizing, 2271 (41.3%) femininizing, and 552 (10.0%) other. RMT analysis led to the development of 54 unidimensional scales and 2 checklists covering domains of health-related quality of life, sexual, urination, gender practices, voice, hair, face and neck, body, breasts, genital feminization, chest, genital masculinization, and experience of care. Test-retest reliability of the scales (intraclass correlation coefficient [average] >0.70) was demonstrated. Only 1 item (phalloplasty donor flap) had an ICC less than 0.70. As hypothesized, scores increased incrementally with better associated self-reported categorical responses. For example, among 661 participants who reported poor psychological well-being, the mean (SD) scale score was 45 (18) points; for those who reported excellent psychological well-being, the mean (SD) scale score was 85 (16) points (P < .001). In this cross-sectional study of 5497 TGD adults, the instrument demonstrated reliability and validity. The instrument was validated in an international sample and is designed to collect and compare evidence-based outcome data for gender-affirming care from the patients' perspective.

AI-driven patient support: Evaluating the effectiveness of ChatGPT-4 in addressing queries about ovarian cancer compared with healthcare professionals in gynecologic oncology.

Artificial intelligence (AI) chatbots, such as ChatGPT-4, allow a user to ask questions on an interactive level. This study evaluated the correctness and completeness of responses to questions about ovarian cancer from a GPT-4 chatbot, LilyBot, compared with responses from healthcare professionals in gynecologic cancer care. Fifteen categories of questions about ovarian cancer were collected from an online patient Chatgroup forum. Ten healthcare professionals in gynecologic oncology generated 150 questions and responses relative to these topics. Responses from LilyBot and the healthcare professionals were scored for correctness and completeness by eight independent healthcare professionals with similar backgrounds blinded to the identity of the responders. Differences between groups were analyzed with Mann-Whitney U and Kruskal-Wallis tests, followed by Tukey's post hoc comparisons. Mean scores for overall performance for all 150 questions were significantly higher for LilyBot compared with the healthcare professionals for correctness (5.31 ± 0.98 vs. 5.07 ± 1.00, p = 0.017; range = 1-6) and completeness (2.66 ± 0.55 vs. 2.36 ± 0.55, p < 0.001; range = 1-3). LilyBot had significantly higher scores for immunotherapy compared with the healthcare professionals for correctness (6.00 ± 0.00 vs. 4.70 ± 0.48, p = 0.020) and completeness (3.00 ± 0.00 vs. 2.00 ± 0.00, p < 0.010); and gene therapy for completeness (3.00 ± 0.00 vs. 2.20 ± 0.42, p = 0.023). The significantly better performance by LilyBot compared with healthcare professionals highlights the potential of ChatGPT-4-based dialogue systems to provide patients with clinical information about ovarian cancer.

TWTFPOS-IDF: Thematic Term Weighting Scheme for Enhanced Question Classification Using Bloom's Taxonomy

Creating question text using a cognitive approach based on Bloom’s Taxonomy (BT) is essential for maintaining question quality in learning assessment. Various studies have explored term weighting schemes to improve BT-based question classification accuracy. However, achieving higher accuracy in classifying cognitive levels requires more than just analyzing verbs—it must also incorporate thematic terms relevant to BT. Existing approaches primarily assign weights to verbs and supporting verbs, often neglecting thematic terms that provide crucial context for classification. This study introduces a novel thematic term weighting scheme, TWTFPOS-IDF, which assigns the highest weight to thematic terms compared to verbs and other supporting words. Thematic terms are identified using the BT word database, with feature extraction, selection, and model tuning optimized to enhance classification accuracy. To ensure robustness, the model is evaluated using a newly constructed, larger dataset that includes a diverse set of educational questions across multiple domains. Machine Learning (ML) and Deep Neural Networks (DNN) are employed for classification, with performance assessed using standard metrics and ANOVA statistical testing. The experimental results demonstrate that the proposed model significantly outperforms previous schemes, achieving an average accuracy of 0.905 and a k-fold value of 0.886. The highest-performing ML algorithm recorded an accuracy of 0.977 and a k-fold value of 0.970. The use of a larger dataset ensures greater generalizability and stability of the model across different question structures. The ANOVA test confirms that model optimization and the expanded dataset significantly improve classification accuracy compared to prior research. This research addresses key challenges in automated question classification, enhancing the precision of cognitive level identification in educational assessment. Future studies will focus on automating weight identification and leveraging deep learning techniques to further refine classification performance and scalability.

Evaluating the Accuracy and Readability of ChatGPT-4o’s Responses to Patient-Based Questions about Keratoconus

ABSTRACT Purpose This study aimed to evaluate the accuracy and readability of responses generated by ChatGPT-4o, an advanced large language model, to frequently asked patient-centered questions about keratoconus. Methods A cross-sectional, observational study was conducted using ChatGPT-4o to answer 30 potential questions that could be asked by patients with keratoconus. The accuracy of the responses was evaluated by two board-certified ophthalmologists and scored on a scale of 1 to 5. Readability was assessed using the Simple Measure of Gobbledygook (SMOG), Flesch-Kincaid Grade Level (FKGL), and Flesch Reading Ease (FRE) scores. Descriptive, treatment-related, and follow-up-related questions were analyzed, and statistical comparisons between these categories were performed. Results The mean accuracy score for the responses was 4.48 ± 0.57 on a 5-point Likert scale. The interrater reliability, with an intraclass correlation coefficient of 0.769, indicated a strong level of agreement. Readability scores revealed a SMOG score of 15.49 ± 1.74, an FKGL score of 14.95 ± 1.95, and an FRE score of 27.41 ± 9.71, indicating that a high level of education is required to comprehend the responses. There was no significant difference in accuracy among the different question categories (p = 0.161), but readability varied significantly, with treatment-related questions being the easiest to understand. Conclusion ChatGPT-4o provides highly accurate responses to patient-centered questions about keratoconus, though the complexity of its language may limit accessibility for the general population. Further development is needed to enhance the readability of AI-generated medical content.