Generative Pre-trained Transformer Research Articles

The role of chat generative pre-trained transformer in facilitating decision-making and the e-learning process in higher education

Digital technology and artificial intelligence technologies have been progressing rapidly, thus giving rise to intelligent chatbots such as chat generative pre-trained transformer (ChatGPT). These chatbots make searching for information more efficient and provide higher education institutions with assistance in decision-making. The goal of this research is to explore the capabilities of ChatGPT technology and its role in enhancing the e-learning process. Moreover, it seeks to determine whether ChatGPT can provide useful suggestions to improve the decision-making process in higher education. ChatGPT is effective in an e-learning environment for the following reasons: it facilitates personalized learning experiences, offers real-time support, and enhances decision-making by leveraging natural language processing capabilities. As suggested by the findings, ChatGPT has significant potential in higher education, as demonstrated by its ability to improve interactive participation, educational strategies, and educational outcomes. This study highlights the importance of incorporating ChatGPT into higher education settings to improve e-learning and decision-making.

The potential of ChatGPT technology in education: advantages, obstacles and future growth

<span lang="EN-US">Information and communication technology is becoming increasingly prevalent in our daily lives, with interactive communication modes such as social networks and instant messaging reaching unprecedented popularity. These tools are now widely utilised in various academic and research institutions by both faculty and students for communication and distance learning purposes. <a name="_Hlk155360394"></a>Chat generative pre-trained transformer (ChatGPT) and artificial intelligence tools hold the potential to revolutionise the way that students obtain knowledge and support. ChatGPT is a cutting-edge language technology capable of constructing intelligent, coherent texts, making it a valuable tool for writing and communication across different fields, including education. However, universities that incorporate ChatGPT as a teaching tool must address concerns regarding plagiarism and academic integrity. This investigation focuses on the advantages and obstacles of applying ChatGPT technology in the education field and its potential for future development. Findings reveal that through careful consideration of the ethical dilemmas and issues, academic institutions can leverage the maximum potential of ChatGPT to provide a more accessible, successful, and personalised learning experience for learners. The development prospects for ChatGPT appear promising, given its potential to grow and enhance its capabilities through on-going research and innovation.</span>

From RNN techniques to pre-trained models: emphasis on the use in Arabic machine translation

<p>Neural Machine Translation (NMT) has gained increasing attention in recent years due to its promising performance over conventional approaches such as Statistical Machine Translation. Nevertheless, when applied to languages with different structures, such as the pair (English, Arabic) that interests us in this work, its efficiency is degraded.<br />Alternatively, the use of unsupervised pre-training of large neural models has recently made a significant leap forward in the field of Natural Language Processing (NLP). By warm starting from published checkpoints, NLP practitioners have been able to push the state-of-the-art boundaries on multiple benchmarks while simultaneously saving considerable computational time. The emphasis so far has been mainly on natural language understanding problems. In this paper, we show the efficacy of pre-trained checkpoints for Arabic Machine Translation. We have evolved a transformer-based sequence-to-sequence model which is compatible with pre-trained checkpoints publicly available for Arabic Bidirectional Encoder Representations from Transformers (AraBERT) and Arabic Generative Pre-trained Transformer (AraGPT) and conducted a thorough empirical study on the usefulness of initializing our Arabic MT model, both encoder and decoder, with these checkpoints. Our models yield new state-of-the-art results in Arabic MT.</p>

Cyber Chatter

Abstract: Cyber Chatter, a web-based application built on the MERN stack, uses the power of GPT (Generative Pre-trained Transformer) technology to provide basic educational chat services and staff Building and ensure efficiency and they have been able to do it well. Cyber Chatter uses GPT-3.5, an advanced generic bid process, to provide accurate and meaningful answers to customer questions. With prior training in a large body of content knowledge, this demonstration demonstrates the incredible ability to develop and generate human-like emotions The application uses machine learning techniques to analyze customer data and continues to improve response quality over time. The use of Cyber Chatter has expanded to include Content Summarizer, Section Generator, Chat with Chatbot, Sci-fi image generator. Understudies can post questions, get answers, and engage in interesting conversations by using its natural user interface with GPT -a chatbot, Cyber Chatter. Cyber Chatter not only covers educational topics but also includes such features as content generators and JavaScript converters. Combining innovative features and an easy-to-use system, Cyber Chatter turns into a versatile punch where students can explore educational tools and interact with intelligent chatbots on a regular basis has enriched their learning experience

Comparing the Diagnostic Performance of GPT-4-based ChatGPT, GPT-4V-based ChatGPT, and Radiologists in Challenging Neuroradiology Cases.

To compare the diagnostic performance among Generative Pre-trained Transformer (GPT)-4-based ChatGPT, GPT‑4 with vision (GPT-4V) based ChatGPT, and radiologists in challenging neuroradiology cases. We collected 32consecutive "Freiburg Neuropathology Case Conference" cases from the journal Clinical Neuroradiology between March 2016 and December 2023. We input the medical history and imaging findings into GPT-4-based ChatGPT and the medical history and images into GPT-4V-based ChatGPT, then both generated adiagnosis for each case. Six radiologists (three radiology residents and three board-certified radiologists) independently reviewed all cases and provided diagnoses. ChatGPT and radiologists' diagnostic accuracy rates were evaluated based on the published ground truth. Chi-square tests were performed to compare the diagnostic accuracy of GPT-4-based ChatGPT, GPT-4V-based ChatGPT, and radiologists. GPT‑4 and GPT-4V-based ChatGPTs achieved accuracy rates of 22% (7/32) and 16% (5/32), respectively. Radiologists achieved the following accuracy rates: three radiology residents 28% (9/32), 31% (10/32), and 28% (9/32); and three board-certified radiologists 38% (12/32), 47% (15/32), and 44% (14/32). GPT-4-based ChatGPT's diagnostic accuracy was lower than each radiologist, although not significantly (all p > 0.07). GPT-4V-based ChatGPT's diagnostic accuracy was also lower than each radiologist and significantly lower than two board-certified radiologists (p = 0.02 and 0.03) (not significant for radiology residents and one board-certified radiologist [all p > 0.09]). While GPT-4-based ChatGPT demonstrated relatively higher diagnostic performance than GPT-4V-based ChatGPT, the diagnostic performance of GPT‑4 and GPT-4V-based ChatGPTs did not reach the performance level of either radiology residents or board-certified radiologists in challenging neuroradiology cases.

Evaluating the accuracy of Chat Generative Pre-trained Transformer version 4 (ChatGPT-4) responses to United States Food and Drug Administration (FDA) frequently asked questions about dental amalgam

BackgroundThe use of artificial intelligence in the field of health sciences is becoming widespread. It is known that patients benefit from artificial intelligence applications on various health issues, especially after the pandemic period. One of the most important issues in this regard is the accuracy of the information provided by artificial intelligence applications.ObjectiveThe purpose of this study was to the frequently asked questions about dental amalgam, as determined by the United States Food and Drug Administration (FDA), which is one of these information resources, to Chat Generative Pre-trained Transformer version 4 (ChatGPT-4) and to compare the content of the answers given by the application with the answers of the FDA.MethodsThe questions were directed to ChatGPT-4 on May 8th and May 16th, 2023, and the responses were recorded and compared at the word and meaning levels using ChatGPT. The answers from the FDA webpage were also recorded. The responses were compared for content similarity in “Main Idea”, “Quality Analysis”, “Common Ideas”, and “Inconsistent Ideas” between ChatGPT-4’s responses and FDA’s responses.ResultsChatGPT-4 provided similar responses at one-week intervals. In comparison with FDA guidance, it provided answers with similar information content to frequently asked questions. However, although there were some similarities in the general aspects of the recommendation regarding amalgam removal in the question, the two texts are not the same, and they offered different perspectives on the replacement of fillings.ConclusionsThe findings of this study indicate that ChatGPT-4, an artificial intelligence based application, encompasses current and accurate information regarding dental amalgam and its removal, providing it to individuals seeking access to such information. Nevertheless, we believe that numerous studies are required to assess the validity and reliability of ChatGPT-4 across diverse subjects.

Performance of an online chat-based artificial intelligence interface for patient education on atrial fibrillation ablation

Abstract Background Chat-based artificial intelligence (AI) web interfaces that aim to mimic human conversation have increasing utilization in healthcare to help with simple tasks such as scheduling appointments, and even more complex tasks such as providing patient educational responses to COVID-19 questions as done by the World Health Organization.1 Chat-based AI has also been shown to provide accurate responses to cardiovascular disease prevention questions.2 Its ability to provide patient education for more complex treatments like atrial fibrillation (AF) ablation has not been explored. Purpose To evaluate the quality of a popular chat-based AI program’s answers to patient questions about AF ablation. Methods Twenty commonly asked questions ("prompts") regarding AF ablation were entered into ChatGPT (Chat Generative Pre-trained Transformer), a large language model-based AI program (Fig. 1). Prompts were written in plain language; technical terms were avoided except for "radiofrequency", "cryoablation" and "pulsed field ablation" (PFA). SMOG readability calculator was used to assess responses for difficulty and grade-level, as healthcare organizations recommend ≤ 8th-grade level complexity for patient information. Response content was graded by 3 experienced cardiac electrophysiologists as "reasonable", "missing important elements/some inaccuracies" or "misleading/inappropriate". Responses are presented in mean +/- standard deviation and percentages. Results Responses averaged 118±67 words (Fig. 1). Of 20 responses, 17 (85%) were deemed reasonable, 3 (15%) missing important elements/some inaccuracies and none inappropriate or misleading; 16 (80%) emphasized discussion of issues with the healthcare team (Fig. 2). Responses missing important elements/some inaccuracies were those about risks/complications of ablation [missing phrenic nerve palsy, atrioesophageal fistula (AEF), potential need for emergent cardiac surgery or pacemaker, death], concerning symptoms post-procedure (missing symptoms of hematoma, AEF, stroke), and that PFA is not yet approved for use in all regions. Average reading grade level of responses was 13.8 (college level or "professional"): 17 (85%) responses were 12th grade level, 11 (55%) were college-level or higher, and 6 (30%) were college-graduate level ("extremely difficult"). None were ≤ 8th grade level (Fig. 2). Conclusions A majority of ChatGPT responses to common patient questions about AF ablation had reasonable content quality that frequently emphasized the importance of discussion with the healthcare team. However, responses to more difficult questions regarding risks, symptoms of potential complications, or newer technology missed important details; more than half of responses required college-level reading skills. While use of Chat-AI for patient education on EP topics appears promising, patients should be advised to use caution. Further AI training to improve content and readability should be explored.

#1772 Comparison of interpretation of acid-base disorder in patients with critical illness: nephrologist versus ChatGPT

Abstract Background and Aims The generative pre-trained transformer (GPT), a type of large language model (LLM), is now playing a huge role in driving innovation in fields of medical education, diagnosis and treatment and with a process of continuous improvement it will itself be able to become beacons of sustainability. Medical professionals expect that they may have help from LLMs to diagnose patients more accurately and efficiently, but it is unclear whether current LLMs are well-trained and validated on real-world clinical data. In this study, we compared the performance of ChatGPT, a representative LLM, MDCalc, an online medical calculator, and human nephrologist for their diagnostic accuracies of acid-base gas analysis in critically ill cases. Method This study included 130 patients admitted to intensive care unit with varying medical conditions. All variables were obtained during the first 24 hours after admission. Results The Fleiss’ Kappa between the interpretations of nephrologist, ChatGPT and MDCalc with the acid-base gas analysis for acid-base disorders showed that Fleiss’ Kappa value was −0.138 (95% CI −0.216 to −0.059), indicating no agreement among the judgments of human doctor, LLM and online medical calculator in critically ill patients for the interpretation of acid-base status. MDCalc showed that all patients had mixed acid-base disorders while nephrologist that 4 patients had simple acid-base disorder and 1 patient had normal acid-base balance. By contrast, ChatGPT reported that 51 patients had only simple acid-base disorder. Furthermore, according to ChatGPT, normal acid-base balance was found in 27 patients, who were all diagnosed as having acid-base disorders by MDCalc or nephrologist. Conclusion We found that current ChatGPT does not yet provide better diagnostic performance for interpreting acid-base balance in critically ill patients who with usually have mixed acid-base disorders compared with an existing online medical calculator or nephrologist.

Perceptions of Higher Education Faculty Regarding the Use of Chat Generative Pre-Trained Transformer (ChatGPT) in Education

This research delves into the perspectives of faculty members in higher education regarding the utilization of Chat Generative Pre-Trained Transformer (ChatGPT) in the realm of education. Additionally, it examines the probable impacts of this technology on social transformation and academic settings in the Philippines. A research study was conducted wherein 20 faculty members from a state university in the Philippines were questioned to acquire their perspectives and firsthand encounters regarding the utilization of ChatGPT in their academic instruction. The research's findings showed that ChatGPT had both beneficial and detrimental effects on higher education. On the positive side, ChatGPT can be a helpful learning tool for students, especially for those who are introverted, and can simplify the repetitive tasks that educators perform while encouraging the growth of students' self-directed critical thinking skills. The utilization of ChatGPT may have adverse effects on students' critical thinking skills and promote excessive dependence on technology. Moreover, the research revealed that although the faculty members were cognizant of ChatGPT's existence, they did not entirely depend on technology to execute highquality education. The efficacy of artificial intelligence and its incorporation in the realm of education, as per the respondents, was contingent upon the conscientiousness of the faculty and administrators in facilitating the enhancement of students' analytical and evaluative proficiencies.

ChatGPT — A Threat or an Aid in Teaching and Learning

The use of artificial intelligence (AI) in education can have a beneficial effect on learners' attitudes and self-assurance toward learning. However, it may also present a danger of substituting teachers and interpersonal interactions that are essential for learners' holistic development. This research investigated the use of AI, specifically the Chat Generative Pre-trained Transformer (ChatGPT), in education and whether it is an ally or a threat to teaching and learning. It discussed and elaborated how AI has changed the traditional practice, principles, culture, and theory in education. Moreover, it identified the advantages and disadvantages of using ChatGPT in education, potential paradigm shifts that may arise from its use, and its impact on science and mathematics education. Furthermore, this study presented an analysis of ChatGPT's responses to standardized marking schemes in science and mathematics and provided valuable insights into how AI can be integrated into the teaching and learning of science and mathematics. This study concluded that a review of the assessment approaches, as well as an openminded perspective in adapting the curriculum and pedagogies with the incorporation of AI, are needed to maximize the learning outcomes of 21st century learners.

Utilization of ChatGPT in Dental Healthcare

The integration of artificial intelligence (AI) in the field of dentistry has been gaining prominence, with Chatbot Generative Pretrained Transformer (ChatGPT), emerging as a pivotal tool. ChatGPT's applications span a myriad of functionalities including diagnostic aid, educational support, and patient interaction.

The Heap, the Hype, the Reality: Generative Pretrained Transformer for Systematic Reviews.

The Heap, the Hype, the Reality: Generative Pretrained Transformer for Systematic Reviews.

A bibliometric analysis of literature published on ChatGPT and GPT

Purpose This study aims to analyse the literature published on ChatGPT and generative pre-trained transformer (GPT) available through Scopus to identify the top countries, institutions, authors, journals and titles in terms of publishing and citations in the area.The research also intends to determine the collaborative trends among countries and authors as well as top-used keywords on the topic identified from the analysed literature. Design/methodology/approach The researchers searched the Scopus database to collect and assess the literature on the topic. The paper used six applications such as Biblioshiny, VosViewer, Python, MS Access and Excel and Endnote to collect and analyse the literature. Findings It is found that European countries such as the USA, the UK and Germany took the lead in terms of publishing and impact in the area but the USA stood first with 90 publications and 1,720 citations in this connection. Likewise, the organization “Rheinisch-Westfälische Technische Hochschule Aachen” scored the top position regarding publishing, but Open AI remained on top for getting the highest citations (1,384). Apropos, the author “Choi, Y” produced the highest number of publications. The research also shares the collaborative patterns, top journals and publications in the area, as well as the top-used keywords on the topic. Originality/value To the best of the authors’ knowledge, this is the first study that shares a bibliometric analysis of literature published on GPT and ChatGPT. The research not only fills the research gap on the topic but also shares implications for relevant stakeholders and future research directions for future researchers.

Integrating AI in Lipedema Management: Assessing the Efficacy of GPT-4 as a Consultation Assistant.

The role of artificial intelligence (AI) in healthcare is evolving, offering promising avenues for enhancing clinical decision making and patient management. Limited knowledge about lipedema often leads to patients being frequently misdiagnosed with conditions like lymphedema or obesity rather than correctly identifying lipedema. Furthermore, patients with lipedema often present with intricate and extensive medical histories, resulting in significant time consumption during consultations. AI could, therefore, improve the management of these patients. This research investigates the utilization of OpenAI's Generative Pre-Trained Transformer 4 (GPT-4), a sophisticated large language model (LLM), as an assistant in consultations for lipedema patients. Six simulated scenarios were designed to mirror typical patient consultations commonly encountered in a lipedema clinic. GPT-4 was tasked with conducting patient interviews to gather medical histories, presenting its findings, making preliminary diagnoses, and recommending further diagnostic and therapeutic actions. Advanced prompt engineering techniques were employed to refine the efficacy, relevance, and accuracy of GPT-4's responses. A panel of experts in lipedema treatment, using a Likert Scale, evaluated GPT-4's responses across six key criteria. Scoring ranged from 1 (lowest) to 5 (highest), with GPT-4 achieving an average score of 4.24, indicating good reliability and applicability in a clinical setting. This study is one of the initial forays into applying large language models like GPT-4 in specific clinical scenarios, such as lipedema consultations. It demonstrates the potential of AI in supporting clinical practices and emphasizes the continuing importance of human expertise in the medical field, despite ongoing technological advancements.

Limitations of Large Language Models in Propaganda Detection Task

Propaganda in the digital era is often associated with online news. In this study, we focused on the use of large language models and their detection of propaganda techniques in the electronic press to investigate whether it is a noteworthy replacement for human annotators. We prepared prompts for generative pre-trained transformer models to find spans in news articles where propaganda techniques appear and name them. Our study was divided into three experiments on different datasets—two based on an annotated SemEval2020 Task 11 corpora and one on an unannotated subset of the Polish Online News Corpus, which we claim to be an even bigger challenge as an example of an under-resourced language. Reproduction of the results of the first experiment resulted in a higher recall of 64.53% than the original run, and the highest precision of 81.82% was achieved for gpt-4-1106-preview CoT. None of our attempts outperformed the baseline F1 score. One of the attempts with gpt-4-0125-preview on original SemEval2020 Task 11 achieved an almost 20% F1 score, but it was below the baseline, which oscillated around 50%. Part of our work that was dedicated to Polish articles showed that gpt-4-0125-preview had a 74% accuracy in the binary detection of propaganda techniques and 69% in propaganda technique classification. The results for SemEval2020 show that the outputs of generative models tend to be unpredictable and are hardly reproducible for propaganda detection. For the time being, these are unreliable methods for this task, but we believe they can help to generate more training data.

How Factually Accurate is GPT-3? A Focused Case Study on Helping Malaysia’s B40s Through e-Commerce

GPT-3 (Generative Pre-trained Transformer 3) is an advanced natural language processing model utilizing unsupervised learning to generate sophisticated human-like text. GPT-3 has been lauded for its potential to revolutionize the field of natural language processing, with its capacity to generate a variety of text with a high degree of fluency and accuracy. We examine the ability of GPT-3’s to produce text related to a focused subject matter: alleviating poverty in Malaysia through e-Commerce. We especially examine GPT-3’s ability to produce factual responses within this narrow context. It was discovered that GPT-3 could produce plausible statements, albeit some of them being factually debatable or incorrect due to how its training data was sourced. We also discuss how GPT-3 could be used unscrupulously to either produce academic-sounding responses that appear to be a product of research, but possibly untrue or inaccurate and discuss its potential ramifications (such as propaganda and disinformation). We end the paper with some suggestions to the brilliant team at OpenAI to further improve GPT-3 for the advancement of humankind.

Assessing ChatGPT vs. Standard Medical Resources for Endoscopic Sleeve Gastroplasty Education: A Medical Professional Evaluation Study.

The Chat Generative Pre-Trained Transformer (ChatGPT) represents a significant advancement in artificial intelligence (AI) chatbot technology. While ChatGPT offers promising capabilities, concerns remain about its reliability and accuracy. This study aims to evaluate ChatGPT's responses to patients' frequently asked questions about Endoscopic Sleeve Gastroplasty (ESG). Expert Gastroenterologists and Bariatric Surgeons, with experience in ESG, were invited to evaluate ChatGPT-generated answers to eight ESG-related questions, and answers sourced from hospital websites. The evaluation criteria included ease of understanding, scientific accuracy, and overall answer satisfaction. They were also tasked with discerning whether each response was AI generated or not. Twelve medical professionals with expertise in ESG participated, 83.3% of whom had experience performing the procedure independently. The entire cohort possessed substantial knowledge about ESG. ChatGPT's utility among participants, rated on a scale of one to five, averaged 2.75. The raters demonstrated a 54% accuracy rate in distinguishing AI-generated responses, with a sensitivity of 39% and specificity of 60%, resulting in an average of 17.6 correct identifications out of a possible 31. Overall, there were no significant differences between AI-generated and non-AI responses in terms of scientific accuracy, understandability, and satisfaction, with one notable exception. For the question defining ESG, the AI-generated definition scored higher in scientific accuracy (4.33 vs. 3.61, p = 0.007) and satisfaction (4.33 vs. 3.58, p = 0.009) compared to the non-AI versions. This study underscores ChatGPT's efficacy in providing medical information on ESG, demonstrating its comparability to traditional sources in scientific accuracy.

Investigating the capability of ChatGPT for generating multiple-choice reading comprehension items

The development of multiple-choice reading comprehension items based on specific reading subskills is crucial in teaching, learning, and testing reading. But it remains a challenging task because this process is time-consuming and costly. This study aims to investigate the capability of ChatGPT (Chat Generative Pre-trained Transformer) for generating multiple-choice reading comprehension items. Psychometric models and human review were adopted to evaluate the item quality based on the benchmark of human-authored items. The results showed that ChatGPT-authored multiple-choice items were acceptable and comparable to human-authored items in terms of psychometric properties, and human review by questionnaire, expert judgment, and interview found that ChatGPT had the potential capability to serve as a test developer and assistant for teaching and learning reading. However, some shortcomings and potential pitfalls were also identified and room for improvement was discussed when ChatGPT is applied to generate items for educational purposes.

Global trends and hotspots of ChatGPT in medical research: a bibliometric and visualized study

ObjectiveWith the rapid advancement of Chat Generative Pre-Trained Transformer (ChatGPT) in medical research, our study aimed to identify global trends and focal points in this domain.MethodAll publications on ChatGPT in medical research were retrieved from the Web of Science Core Collection (WoSCC) by Clarivate Analytics from January 1, 2023, to January 31, 2024. The research trends and focal points were visualized and analyzed using VOSviewer and CiteSpace.ResultsA total of 1,239 publications were collected and analyzed. The USA contributed the largest number of publications (458, 37.145%) with the highest total citation frequencies (2,461) and the largest H-index. Harvard University contributed the highest number of publications (33) among all full-time institutions. The Cureus Journal of Medical Science published the most ChatGPT-related research (127, 10.30%). Additionally, Wiwanitkit V contributed the majority of publications in this field (20). “Artificial Intelligence (AI) and Machine Learning (ML),” “Education and Training,” “Healthcare Applications,” and “Data Analysis and Technology” emerged as the primary clusters of keywords. These areas are predicted to remain hotspots in future research in this field.ConclusionOverall, this study signifies the interdisciplinary nature of ChatGPT research in medicine, encompassing AI and ML technologies, education and training initiatives, diverse healthcare applications, and data analysis and technology advancements. These areas are expected to remain at the forefront of future research, driving continued innovation and progress in the field of ChatGPT in medical research.

The Use of Generative AI for Scientific Literature Searches for Systematic Reviews: ChatGPT and Microsoft Bing AI Performance Evaluation.

A large language model is a type of artificial intelligence (AI) model that opens up great possibilities for health care practice, research, and education, although scholars have emphasized the need to proactively address the issue of unvalidated and inaccurate information regarding its use. One of the best-known large language models is ChatGPT (OpenAI). It is believed to be of great help to medical research, as it facilitates more efficient data set analysis, code generation, and literature review, allowing researchers to focus on experimental design as well as drug discovery and development. This study aims to explore the potential of ChatGPT as a real-time literature search tool for systematic reviews and clinical decision support systems, to enhance their efficiency and accuracy in health care settings. The search results of a published systematic review by human experts on the treatment of Peyronie disease were selected as a benchmark, and the literature search formula of the study was applied to ChatGPT and Microsoft Bing AI as a comparison to human researchers. Peyronie disease typically presents with discomfort, curvature, or deformity of the penis in association with palpable plaques and erectile dysfunction. To evaluate the quality of individual studies derived from AI answers, we created a structured rating system based on bibliographic information related to the publications. We classified its answers into 4 grades if the title existed: A, B, C, and F. No grade was given for a fake title or no answer. From ChatGPT, 7 (0.5%) out of 1287 identified studies were directly relevant, whereas Bing AI resulted in 19 (40%) relevant studies out of 48, compared to the human benchmark of 24 studies. In the qualitative evaluation, ChatGPT had 7 grade A, 18 grade B, 167 grade C, and 211 grade F studies, and Bing AI had 19 grade A and 28 grade C studies. This is the first study to compare AI and conventional human systematic review methods as a real-time literature collection tool for evidence-based medicine. The results suggest that the use of ChatGPT as a tool for real-time evidence generation is not yet accurate and feasible. Therefore, researchers should be cautious about using such AI. The limitations of this study using the generative pre-trained transformer model are that the search for research topics was not diverse and that it did not prevent the hallucination of generative AI. However, this study will serve as a standard for future studies by providing an index to verify the reliability and consistency of generative AI from a user's point of view. If the reliability and consistency of AI literature search services are verified, then the use of these technologies will help medical research greatly.