Artificial Intelligence Chatbot Research Articles

Assessing ChatGPT vs. Standard Medical Resources for Endoscopic Sleeve Gastroplasty Education: A Medical Professional Evaluation Study.

The Chat Generative Pre-Trained Transformer (ChatGPT) represents a significant advancement in artificial intelligence (AI) chatbot technology. While ChatGPT offers promising capabilities, concerns remain about its reliability and accuracy. This study aims to evaluate ChatGPT's responses to patients' frequently asked questions about Endoscopic Sleeve Gastroplasty (ESG). Expert Gastroenterologists and Bariatric Surgeons, with experience in ESG, were invited to evaluate ChatGPT-generated answers to eight ESG-related questions, and answers sourced from hospital websites. The evaluation criteria included ease of understanding, scientific accuracy, and overall answer satisfaction. They were also tasked with discerning whether each response was AI generated or not. Twelve medical professionals with expertise in ESG participated, 83.3% of whom had experience performing the procedure independently. The entire cohort possessed substantial knowledge about ESG. ChatGPT's utility among participants, rated on a scale of one to five, averaged 2.75. The raters demonstrated a 54% accuracy rate in distinguishing AI-generated responses, with a sensitivity of 39% and specificity of 60%, resulting in an average of 17.6 correct identifications out of a possible 31. Overall, there were no significant differences between AI-generated and non-AI responses in terms of scientific accuracy, understandability, and satisfaction, with one notable exception. For the question defining ESG, the AI-generated definition scored higher in scientific accuracy (4.33 vs. 3.61, p = 0.007) and satisfaction (4.33 vs. 3.58, p = 0.009) compared to the non-AI versions. This study underscores ChatGPT's efficacy in providing medical information on ESG, demonstrating its comparability to traditional sources in scientific accuracy.

Physician and Artificial Intelligence Chatbot Responses to Cancer Questions From Social Media.

Artificial intelligence (AI) chatbots pose the opportunity to draft template responses to patient questions. However, the ability of chatbots to generate responses based on domain-specific knowledge of cancer remains to be tested. To evaluate the competency of AI chatbots (GPT-3.5 [chatbot 1], GPT-4 [chatbot 2], and Claude AI [chatbot 3]) to generate high-quality, empathetic, and readable responses to patient questions about cancer. This equivalence study compared the AI chatbot responses and responses by 6 verified oncologists to 200 patient questions about cancer from a public online forum. Data were collected on May 31, 2023. Random sample of 200 patient questions related to cancer from a public online forum (Reddit r/AskDocs) spanning from January 1, 2018, to May 31, 2023, was posed to 3 AI chatbots. The primary outcomes were pilot ratings of the quality, empathy, and readability on a Likert scale from 1 (very poor) to 5 (very good). Two teams of attending oncology specialists evaluated each response based on pilot measures of quality, empathy, and readability in triplicate. The secondary outcome was readability assessed using Flesch-Kincaid Grade Level. Responses to 200 questions generated by chatbot 3, the best-performing AI chatbot, were rated consistently higher in overall measures of quality (mean, 3.56 [95% CI, 3.48-3.63] vs 3.00 [95% CI, 2.91-3.09]; P < .001), empathy (mean, 3.62 [95% CI, 3.53-3.70] vs 2.43 [95% CI, 2.32-2.53]; P < .001), and readability (mean, 3.79 [95% CI, 3.72-3.87] vs 3.07 [95% CI, 3.00-3.15]; P < .001) compared with physician responses. The mean Flesch-Kincaid Grade Level of physician responses (mean, 10.11 [95% CI, 9.21-11.03]) was not significantly different from chatbot 3 responses (mean, 10.31 [95% CI, 9.89-10.72]; P > .99) but was lower than those from chatbot 1 (mean, 12.33 [95% CI, 11.84-12.83]; P < .001) and chatbot 2 (mean, 11.32 [95% CI, 11.05-11.79]; P = .01). The findings of this study suggest that chatbots can generate quality, empathetic, and readable responses to patient questions comparable to physician responses sourced from an online forum. Further research is required to assess the scope, process integration, and patient and physician outcomes of chatbot-facilitated interactions.

Mental health stigma: a conundrum for healthcare practitioners in conservative communities

This paper presents perspectives on the stigma and shame around mental health in conservative communities, and some of the issues faced by health systems in those communities. The various causes of stigma are explored, and how these are often more pronounced in culturally reserved, conservative communities. While health systems are supposed to provide support for mental health sufferers, this stigma sometimes even extends to healthcare workers, which can discourage patients from asking for assistance. Solutions and reforms are needed, for example education programs; addressing gender norms, and the consideration of culture and religion, to form effective solutions. It is also suggested that alternative therapies and support mechanisms, including digital solutions such as artificial intelligence chatbots, may be useful to provide much needed support to individuals with poor mental health. Along with integrating options such as CBT (cognitive behavioral therapy), it may be useful to draw on indigenous psychologies, such as Islamic psychology, as a way of decolonizing approaches. Therefore, when considering solutions, cultural and religious norms must be considered to ensure their efficacy and acceptance.

Enhancing Readability of Online Patient-Facing Content: The Role of AI Chatbots in Improving Cancer Information Accessibility.

Internet-based health education is increasingly vital in patient care. However, the readability of online information often exceeds the average reading level of the US population, limiting accessibility and comprehension. This study investigates the use of chatbot artificial intelligence to improve the readability of cancer-related patient-facing content. We used ChatGPT 4.0 to rewrite content about breast, colon, lung, prostate, and pancreas cancer across 34 websites associated with NCCN Member Institutions. Readability was analyzed using Fry Readability Score, Flesch-Kincaid Grade Level, Gunning Fog Index, and Simple Measure of Gobbledygook. The primary outcome was the mean readability score for the original and artificial intelligence (AI)-generated content. As secondary outcomes, we assessed the accuracy, similarity, and quality using F1 scores, cosine similarity scores, and section 2 of the DISCERN instrument, respectively. The mean readability level across the 34 websites was equivalent to a university freshman level (grade 13±1.5). However, after ChatGPT's intervention, the AI-generated outputs had a mean readability score equivalent to a high school freshman education level (grade 9±0.8). The overall F1 score for the rewritten content was 0.87, the precision score was 0.934, and the recall score was 0.814. Compared with their original counterparts, the AI-rewritten content had a cosine similarity score of 0.915 (95% CI, 0.908-0.922). The improved readability was attributed to simpler words and shorter sentences. The mean DISCERN score of the random sample of AI-generated content was equivalent to "good" (28.5±5), with no significant differences compared with their original counterparts. Our study demonstrates the potential of AI chatbots to improve the readability of patient-facing content while maintaining content quality. The decrease in requisite literacy after AI revision emphasizes the potential of this technology to reduce health care disparities caused by a mismatch between educational resources available to a patient and their health literacy.

Performance of artificial intelligence chatbots in interpreting clinical images of pressure injuries.

To evaluate the accuracy of AI chatbots in staging pressure injuries according to the National Pressure Injury Advisory Panel (NPIAP) Staging through clinical image interpretation, a cross-sectional design was conducted to assess five leading publicly available AI chatbots. As a result, three chatbots were unable to interpret the clinical images, whereas GPT-4 Turbo achieved a high accuracy rate (83.0%) in staging pressure injuries, notably outperforming BingAI Creative mode (24.0%) with statistical significance (p < 0.001). GPT-4 Turbo accurately identified Stages 1 (p < 0.001), 3 (p = 0.001), and 4 (p < 0.001) pressure injuries, and suspected deep tissue injuries (p < 0.001), while BingAI demonstrated significantly lower accuracy across all stages. The findings highlight the potential of AI chatbots, especially GPT-4 Turbo, in accurately diagnosing images and aiding the subsequent management of pressure injuries.

Evaluating ChatGPT's Ability to Answer Common Patient Questions Regarding Hip Fracture.

ChatGPT is an artificial intelligence chatbot software programmed for conversational applications using reinforcement learning techniques. With its growing popularity and overall versatility, it is likely that ChatGPT's applications will expand into health care especially because it relates to patients researching their injuries. The purpose of this study was to investigate ChatGPT's ability to accurately answer frequently asked questions regarding hip fractures. Eleven frequently asked questions regarding hip fractures were posed to ChatGPT, and the responses were recorded in full. Five of these questions were determined to be high-yield based on the likelihood that a patient would ask the question to a chatbot software. The chatbot's responses were analyzed by five fellowship-trained orthopaedic trauma surgeons for their quality and accuracy using an evidence-based approach. The chatbot's answers were rated as "Excellent response requiring no clarification", "Satisfactory response requiring minimal clarification", "Satisfactory response requiring moderate clarification", or "Unsatisfactory response requiring significant clarification." Of the five high-yield questions posed to the chatbot, no question was determined to be unsatisfactory requiring significant clarification by the authors. The remaining responses were either satisfactory requiring minimal clarification (n = 3) or satisfactory requiring moderate clarification (n = 2). The chatbot was generally found to provide unbiased and evidence-based answers that would be clearly understood by most orthopaedic patients. These findings suggest that ChatGPT has the potential to be an effective patient education tool especially because it continues to grow and improve as a chatbot application. Level IV study.

Subscription intentions for ChatGPT plus: a look at user satisfaction and self-efficacy

PurposeThis study examines the key determinants of subscription intentions for ChatGPT Plus (paid version) in business settings, focusing on tasks such as system quality, information support, service quality, perceived intelligence, goal-congruent outcome and self-efficacy.Design/methodology/approachThe study utilized a survey of office workers, analyzed through structural equation modeling, to explore these determinants.FindingsThe results demonstrate that system quality, service quality and perceived intelligence significantly influence satisfaction, while service quality and perceived intelligence also impact goal-congruent outcomes. Contrary to traditional models, satisfaction does not significantly correlate with usage. Instead, a significant relationship is observed between goal-congruent outcomes and usage. Self-efficacy emerges as a crucial predictor of subscription intentions, further underlined by the significant impact of usage on subscription intention.Research limitations/implicationsThe study’s focus on office workers and a single artificial intelligence (AI) chatbot type may limit generalizability. Its findings illuminate several avenues for future research, particularly in diversifying the context and demographics studied.Practical implicationsThis research offers actionable insights for businesses and practitioners in the implementation of AI chatbots. It highlights the importance of enhancing system quality, personalization and user confidence to boost subscription intentions, thereby guiding strategies for user engagement and technology adoption.Originality/valueThis study pioneers in investigating subscription intentions towards AI chatbots, particularly ChatGPT, providing a novel framework that expands upon traditional user behavior theories.

Use of large language model-based chatbots in managing the rehabilitation concerns and education needs of outpatient stroke survivors and caregivers

BackgroundThe utility of large language model-based (LLM) artificial intelligence (AI) chatbots in many aspects of healthcare is becoming apparent though their ability to address patient concerns remains unknown. We sought to evaluate the performance of two well-known, freely-accessible chatbots, ChatGPT and Google Bard, in responding to common questions about stroke rehabilitation posed by patients and their caregivers.MethodsWe collected questions from outpatients and their caregivers through a survey, categorised them by theme, and created representative questions to be posed to both chatbots. We then evaluated the chatbots' responses based on accuracy, safety, relevance, and readability. Interrater agreement was also tracked.ResultsAlthough both chatbots achieved similar overall scores, Google Bard performed slightly better in relevance and safety. Both provided readable responses with some general accuracy, but struggled with hallucinated responses, were often not specific, and lacked awareness of the possibility for emotional situations with the potential to turn dangerous. Additionally, interrater agreement was low, highlighting the variability in physician acceptance of their responses.ConclusionsAI chatbots show potential in patient-facing support roles, but issues remain regarding safety, accuracy, and relevance. Future chatbots should address these problems to ensure that they can reliably and independently manage the concerns and questions of stroke patients and their caregivers.

Comparing and assessing four AI chatbots' competence in economics.

Artificial Intelligence (AI) chatbots have emerged as powerful tools in modern academic endeavors, presenting both opportunities and challenges in the learning landscape. They can provide content information and analysis across most academic disciplines, but significant differences exist in terms of response accuracy for conclusions and explanations, as well as word counts. This study explores four distinct AI chatbots, GPT-3.5, GPT-4, Bard, and LLaMA 2, for accuracy of conclusions and quality of explanations in the context of university-level economics. Leveraging Bloom's taxonomy of cognitive learning complexity as a guiding framework, the study confronts the four AI chatbots with a standard test for university-level understanding of economics, as well as more advanced economics problems. The null hypothesis that all AI chatbots perform equally well on prompts that explore understanding of economics is rejected. The results are that significant differences are observed across the four AI chatbots, and these differences are exacerbated as the complexity of the economics-related prompts increased. These findings are relevant to both students and educators; students can choose the most appropriate chatbots to better understand economics concepts and thought processes, while educators can design their instruction and assessment while recognizing the support and resources students have access to through AI chatbot platforms.

How artificial intelligence chatbots becomes author’s true friend in medical writing without risking ethical violations

[No abstract available]

Unlocking Health Literacy: The Ultimate Guide to Hypertension Education From ChatGPT Versus Google Gemini.

Background Google Gemini (Google, Mountain View, CA) represents the latest advances in the realm of artificial intelligence (AI) and has garnered attention due to its capabilities similar to the increasingly popular ChatGPT (OpenAI, San Francisco, CA). Accurate dissemination of information on common conditions such as hypertension is critical for patient comprehension and management. Despite the ubiquity of AI, comparisons between ChatGPT and Gemini remain unexplored. Methods ChatGPT and Gemini were asked 52 questions derived from the American College of Cardiology's (ACC) frequently asked questions on hypertension, following a specified prompt. Prompts included: no prompting (Form 1), patient-friendly prompting (Form 2), physician-level prompting (Form 3), and prompting for statistics/references (Form 4). Responses were scored as incorrect, partially correct, or correct. Flesch-Kincaid (FK) grade level and word count were recorded. Results Across all forms, scoring frequencies were as follows: 23 (5.5%) incorrect, 162 (38.9%) partially correct, and 231 (55.5%) correct. ChatGPT showed higher rates of partially correct answers than Gemini (p = 0.0346). Physician-level prompts resulted in a higher word count across both platforms (p < 0.001). ChatGPT showed a higher FK grade level (p = 0.033) in physician-friendly prompting. Gemini exhibited a significantly higher mean word count (p < 0.001); however, ChatGPT had a higher FK grade level across all forms (p > 0.001). Conclusion To our knowledge, this study is the first to compare cardiology-related responses from ChatGPT and Gemini, two of the most popular AI chatbots. The grade level for most responses was collegiate level, which was above average for the National Institutes of Health (NIH) recommendations, but on par with most online medical information. Both chatbots responded with a high degree of accuracy, with inaccuracies being rare. Therefore, it is reasonable that cardiologists suggest either chatbot as a source of supplementary education.

Examining the Influence of AI Chatbots on Semantic Web-Based Global Information Management in Various Industries

This article presents a comprehensive analysis of the application and effect of ChatGPT, an advanced AI chatbot model, on global information management across various industries such as healthcare, industry, education, and more. Leveraging a dataset obtained from the Scopus database encompassing research papers from 2022 to 2023, this study investigates the influence of ChatGPT by examining publisher impact, authorship patterns based on Lotka's Law, country-specific scientific production, and keyword distribution. The analysis sheds light on prominent publishers, prolific authors, geographic distribution of research contributions, and prevailing research themes. By understanding the impact of ChatGPT in these sectors, this research contributes to the advancement of knowledge and facilitates informed decision-making regarding the responsible and effective utilization of artificial intelligence (AI) chatbot technologies in global information management.

Study of adoption of artificial intelligence technology-driven natural large language model-based chatbots by firms for customer service interaction

Purpose The purpose of this study was to comprehend the adoption of artificial intelligence (AI) technology-driven natural large language model (LLM)-based chatbots by customers. Design/methodology/approach A qualitative research study method was conducted. This was to explore managerial perspectives towards consumer centric technology adoption of AI plus LLM-based chatbots. This was specifically for AI-driven natural LLM-based chatbots services. The author conducted conducted in-depth personal interviews with 32 experts of digital content AI + LLM chatbot services. Thematic content analysis was undertaken to analyse the data. Findings The advent of natural language processing tools driven by AI technology chatbots has altered human-firm interaction. The research findings indicated that the push-pull-mooring (PPM) factors captured the phenomenon in the most comprehensive way. A total of 15 key factors influencing the adoption of AI technology-driven natural LLM-based chatbots by customers during firm customer interaction were identified in this study by the author. The thematic content analysis unraveled insights regarding transformed consumer adoptions towards AI-driven LLM-based chatbots by means of the PPM framework factors. Research limitations/implications The empirical research investigation contributed to the literature on the PPM theoretical framework. This was specifically in the context of adoption of AI technology-driven natural LLM-based chatbots by customers during firm customer interaction. Practical implications The research study insights would help managers to restructure and reconfigure their organizational processes. This would neccessiated a shift in firm-customer interactions as demanded because of the availability of AI technology-driven natural LLM-based chatbots by customers. Originality/value This research study was based upon the PPM theoretical framework. This study provided a unique analysis of the altered firm customer interaction needs and requirements. This was one of the first studies that applied the framework of PPM theory regarding the adoption of AI technology-driven natural LLM-based chatbots by customers.

The scientific knowledge of three large language models in cardiology: multiple choice questions examination-based performance

Background: The integration of artificial intelligence (AI) chatbots like Google’s Bard, OpenAI’s ChatGPT, and Microsoft’s Bing Chatbot into academic and professional domains, including cardiology, has been rapidly evolving. Their application in educational and research frameworks, however, raises questions about their efficacy, particularly in specialized fields like cardiology. This study aims to evaluate the knowledge depth and accuracy of these AI chatbots in cardiology using a multiple-choice question (MCQ) format. Methods: The study was conducted as an exploratory, cross-sectional study in November 2023 on a bank of 100 MCQs covering various cardiology topics that was created from authoritative textbooks and question banks. These MCQs were then used to assess the knowledge level of Google’s Bard, Microsoft Bing, and ChatGPT 4.0. Each question was entered manually into the chatbots, ensuring no memory retention bias. Results: The study found that ChatGPT 4.0 demonstrated the highest knowledge score in cardiology, with 87% accuracy, followed by Bing at 60% and Bard at 46%. The performance varied across different cardiology subtopics, with ChatGPT consistently outperforming the others. Notably, the study revealed significant differences in the proficiency of these chatbots in specific cardiology domains. Conclusion: This study highlights a spectrum of efficacy among AI chatbots in disseminating cardiology knowledge. ChatGPT 4.0 emerged as a potential auxiliary educational resource in cardiology, surpassing traditional learning methods in some aspects. However, the variability in performance among these AI systems underscores the need for cautious evaluation and continuous improvement, especially for chatbots like Bard, to ensure reliability and accuracy in medical knowledge dissemination.

ChatGPT in higher education: factors influencing ChatGPT user satisfaction and continued use intention

Artificial intelligence (AI) chatbots, represented by ChatGPT, have shown significant improvement in natural learning and problem analysis and solving, which could trigger a profound change in the education sector and have a far-reaching impact on educational practices. This study aimed to gain insights into the various impacts on users’ experience when using the AI tool ChatGPT in higher education. The study analyzed questionnaire data from 328 college students who used ChatGPT. Employing structural equation modeling, this study examined the technology acceptance model in the higher educational setting to identify factors influencing continued use of ChatGPT. The results show that ChatGPT’s compatibility positively affected users’ perceived ease of use and that efficiency positively affected perceived usefulness. Furthermore, perceived ease of use and perceived usefulness were identified as core factors affecting users’ satisfaction and continued use intentions, with user satisfaction showing a significant positive effect on continued use intention. The findings not only provide new perspectives on human–computer interaction theory in higher education but also further refine the functions of current smart devices. This study will help to promote the continued progress and improvement of educational technology, while also deepening understanding of the relationship between attitude and behavior.

Does the Internet Provide Quality Information on Metoidioplasty? Using the Modified Ensuring Quality Information for Patients Tool to Evaluate Artificial Intelligence-Generated and Online Information on Metoidioplasty.

Public interest in gender affirmation surgery has surged over the last decade. This spike in interest, combined with extensive free online medical knowledge, has led many to the Internet for more information on this complicated procedure. This study aimed to evaluate the quality of online information on metoidioplasty. Google Trends in searches on "metoidioplasty" from 2004 to present were assessed. "metoidioplasty" was searched on three popular search engines (Google, Yahoo, and Bing), and the first 100 websites from each search were extracted for inclusion (Fig. 1). Exclusion criteria included duplicates, websites requiring fees, photo libraries, and irrelevant websites. Websites were assigned a score (out of 36) using the modified Ensuring Quality Information for Patients (EQIP) instrument, which grades patient materials based on content (18), identification (6), and structure (12). ChatGPT was also queried for metoidioplasty-related information and responses were analyzed using EQIP. Google Trends analysis indicated relative search interest in "metoidioplasty" has more than quadrupled since 2013(Fig. 2). Of the 93 websites included, only 2 received an EQIP score greater than 27 (6%). Website scores ranged from 7 to 33, with a mean of 18.6 ± 4.8. Mean scores were highest for websites made by health departments (22.3) and lowest for those made by encyclopedias and academic institutions (16.0). Websites with the highest frequency were research articles, web portals, hospital websites, and private practice sites, which averaged scores of 18.2, 19.7, 19.0, and 17.8, respectively. Health department sites averaged the highest content points (11.25), and academic institutions averaged the lowest (5.5). The average content point across all websites was 7.9 of 18. ChatGPT scored a total score of 29: 17 content, 2 identification, and 10 structures. The artificial intelligence chatbot scored the second highest score among all included online resources. Despite the continued use of search engines, the quality of online information on metoidioplasty remains exceptionally poor across most website developers. This study demonstrates the need to improve these resources, especially as interest in gender-affirming surgery continues to grow. ChatGPT and other artificial intelligence chatbots may be efficient and reliable alternatives for those seeking to understand complex medical information.

“What Did You Say, ChatGPT?” The Use of AI in Black Women's HIV Self-Education: An Inductive Qualitative Data Analysis

Abstract The emergence of widely accessible artificial intelligence (AI) chatbots such as ChatGPT presents unique opportunities and challenges in public health self-education. This study examined simulations with ChatGPT for its use in public education of sexual health of Black women, specifically in HIV prevention and/or HIV PrEP use. The research questions guiding the study are as follows: (a) does the information ChatGPT offers about HIV prevention and HIV PrEP differ based on stated race? and (b) how could this relatively new platform inform public health education of Black women educating themselves about sexual health behaviors, diagnoses, and treatments? In addressing these questions, this study also uncovered notable differences in ChatGPT's tone when responding to users based on race. This study described valuable insights that can inform health care professionals, educators, and policymakers, ultimately advancing the cause of sexual health equity for Black women and underscoring the paradigm-shifting potential of AI in the field of public health education.

Evaluating the Performance of Artificial Intelligence Chatbots and Large Language Models in the FE and PE Structural Exams

Evaluating the Performance of Artificial Intelligence Chatbots and Large Language Models in the FE and PE Structural Exams

D124. Comparing Provider and Artificial Intelligence Chatbot Responses to Breast Reconstruction Patient Questions

D124. Comparing Provider and Artificial Intelligence Chatbot Responses to Breast Reconstruction Patient Questions

AI Enabled Chatbot

Abstract: As technology continues its rapid evolution, the integration of Artificial Intelligence (AI) into chatbot systems emerges as a pivotal frontier in reshaping human-computer interaction. This research paper embarks on a comprehensive exploration of the intricate synergy between AI technologies and chatbot development, with a primary focus on elevating user interactions to unprecedented levels of sophistication and intuitiveness. By delving into a nuanced understanding of methodologies, technologies, and challenges, this study aims to provide a holistic perspective on the present state and future potential of AIenabled chatbots. The journey begins with a retrospective analysis of the evolution of chatbots, tracing their trajectory from rulebased systems to the forefront of AI-driven conversational agents. The advent of AI, particularly Natural Language Processing (NLP) and machine learning algorithms, has ushered in a new era, enabling chatbots to not only comprehend but also respond intelligently, mirroring the complexities of human language.