Exam Questions Research Articles

Evaluating the Performance of ChatGPT 3.5 and 4.0 on StatPearls Oculoplastic Surgery Text- and Image-Based Exam Questions

Evaluating the Performance of ChatGPT 3.5 and 4.0 on StatPearls Oculoplastic Surgery Text- and Image-Based Exam Questions

Leveraging large language models to construct feedback from medical multiple-choice Questions.

Exams like the formative Progress Test Medizin can enhance their effectiveness by offering feedback beyond numerical scores. Content-based feedback, which encompasses relevant information from exam questions, can be valuable for students by offering them insight into their performance on the current exam, as well as serving as study aids and tools for revision. Our goal was to utilize Large Language Models (LLMs) in preparing content-based feedback for the Progress Test Medizin and evaluate their effectiveness in this task. We utilize two popular LLMs and conduct a comparative assessment by performing textual similarity on the generated outputs. Furthermore, we study via a survey how medical practitioners and medical educators assess the capabilities of LLMs and perceive the usage of LLMs for the task of generating content-based feedback for PTM exams. Our findings show that both examined LLMs performed similarly. Both have their own advantages and disadvantages. Our survey results indicate that one LLM produces slightly better outputs; however, this comes at a cost since it is a paid service, while the other is free to use. Overall, medical practitioners and educators who participated in the survey find the generated feedback relevant and useful, and they are open to using LLMs for such tasks in the future. We conclude that while the content-based feedback generated by the LLM may not be perfect, it nevertheless can be considered a valuable addition to the numerical feedback currently provided.

Supporting Peer Feedback at Scale

Schneider et al. (2017) conducted a systematic review of 38 meta-analyses investigating 105 variables linked to student performance. They reported peer assessment is the most influential factor in explaining students’ achievement. The benefits are even more significant when combined with self-assessment as students' meta-cognitive capacity is exercised as they calibrate their level of understanding and confidence to that of their peers (Power & Tanner, 2023). Assessing a peer digitally has been shown to be more effective than in-person settings (Li et al., 2020). Winstone & Boud (2022) recommend improving students’ learning by focusing on providing and receiving high-quality feedback rather than on grades. Despite the potential of technology to help students assess each other, there is sparse research in commerce studies, let alone on large-scale subjects, where technology use is essential to leverage students’ learning and encourage interaction, which otherwise would be limited. Such a study can shed light on students' learning and engagement benefits. Technology can also aid in creating a structured learning environment, with students being required to complete multiple tasks throughout the semester and engage with the platform repetitively, reinforcing improved curriculum retention (Schwerter et al., 2022). Our project involved implementing five tasks (a practice task and four assessed tasks) in a large-cohort first-year finance subject at a leading Australian university. The tasks contributed 15% to each student’s final mark and utilised the Feedback Fruits platform (integrated with Canvas LMS). For each task, students scanned their handwritten answers to two questions, uploaded them into the platform as a PDF file, completed self-assessment and reviewed a randomly assigned anonymous peer using a rubric designed to mimic the rubric used to mark final exam questions. We configured the system so that students had at least one week to upload their answers and another week to complete the self-assessment and peer review. Statistical analysis of the Feedback Fruits analytics files, which include completion rates and performance, involved regression analysis and propensity score matching and indicates a significant positive relationship between platform engagement (measured by time spent on the platform, provided comments length and task performance) and final exam performance, even after controlling for a student’s finance aptitude. The study also involved an end-of-semester survey and focus groups to capture students’ perceptions. Students indicated the positive learning benefits they gained from reviewing their peers, being reviewed, and reflecting on their performance. Furthermore, we found a high correlation between the number of comments and the average number of words per comment students left themselves and those left for their peers. The innovation of our research is in the scale, continuous use of the platform throughout the semester with multiple access points per task and better exam preparedness by simulating exam conditions utilising rubrics and handwriting requirements. This poster describes the project’s implementation process map, including task setup, timelines, and the support provided, which is the technical part of the bigger project. We review fundamental mechanisms to improve participation and the quality of the reviews. In addition, the poster presents the challenges faced and offers practical steps to overcome them.

Scaling Oral Assessment Authentically

The rise of AI and related threats to academic integrity in a post-pandemic academic environment where most assessments are not invigilated forces universities to revisit the question of how to conduct safe and fair assessments (Dawson, 2022). As invigilated exams and online proctoring have fallen out of favour with university administrators due to cost, space or privacy concerns, oral assessments have found renewed interest (Newell, 2023). These are effective in measuring students' knowledge whilst verifying that the responses truly belong to the student (Joughin, 1998). Though there are many established variations of oral assessment, from viva voce thesis defence to interactive orals (as widely conducted in Law and Medicine), they usually emphasise authentic live interaction between student and assessor. However, the sheer scale of some undergraduate programs (as common in Business Schools) along with the integrated nature of business curriculum requiring marking expertise from different graders makes truly interactive oral exams unfeasible. As a response, a large first-year undergraduate Business course (with student enrolments of between 500-1200 per term) at a G08 institution implemented an authentic recorded oral assessment, which mimics job interview practice. Students were recorded for a set time period while they responded to questions verbally with the resulting video and transcripts being later marked. We utilised an asynchronous, remotely invigilated oral assessment (Baird et al., 2022) for the final exam to improve academic integrity and to further embed transferable skills that is industry relevant whilst increasing student employability. The new assessment format was named ‘Live Recorded Video Exam (LRVE)’. The invigilation process for this exam was conducted on a one-on-one basis where one invigilator checked each student in their groupings. The invigilators were provided with a procedural checklist to follow to be consistent within all the groups. The process involved the invigilator calling the student into a Microsoft Teams channel, verifying the student’s identity, checking their immediate surroundings, sharing the screen of the device that the student was using, providing a password verbally for the student to access the exam questions and then to start the recording for invigilation purposes. Once the recording commenced, the invigilator would leave the meeting room and allow the student to complete the exam independently by articulating their responses. The graders had access to both the video recording as well as the captioned transcript to mark the exam and check for any student misconduct. Students were informed about the exam format through different channels from the teaching team as well as innovative student-led initiatives. They were given various opportunities to prepare throughout the term within established class structures and practice mock sessions. Out of 542 exams, none were reported for violations of academic integrity and the rate of special consideration applications was lower. This is, in part, due to potential problems being rectified during the student check-in process. Surveys conducted after the exam showed the LRVE format was well-received by markers and student representatives in part because of its flexibility, offering a complement to traditional viva voce models and demonstrating its scalability.

Performance of Chatgpt in ophthalmology exam; human versus AI.

This cross-sectional study focuses on evaluating the success rate of ChatGPT in answering questions from the 'Resident Training Development Exam' and comparing these results with the performance of the ophthalmology residents. The 75 exam questions, across nine sections and three difficulty levels, were presented to ChatGPT. The responses and explanations were recorded. The readability and complexity of the explanations were analyzed and The Flesch Reading Ease (FRE) score (0-100) was recorded using the program named Readable. Residents were categorized into four groups based on their seniority. The overall and seniority-specific success rates of the residents were compared separately with ChatGPT. Out of 69 questions, ChatGPT answered 37 correctly (53.62%). The highest success was in Lens and Cataract (77.77%), and the lowest in Pediatric Ophthalmology and Strabismus (0.00%). Of 789 residents, overall accuracy was 50.37%. Seniority-specific accuracy rates were 43.49%, 51.30%, 54.91%, and 60.05% for 1st to 4th-year residents. ChatGPT ranked 292nd among residents. Difficulty-wise, 11 questions were easy, 44 moderate, and 14 difficult. ChatGPT's accuracy for each level was 63.63%, 54.54%, and 42.85%, respectively. The average FRE score of responses generated by ChatGPT was found to be 27.56 ± 12.40. ChatGPT correctly answered 53.6% of questions in an exam for residents. ChatGPT has a lower success rate on average than a 3rd year resident. The readability of responses provided by ChatGPT is low, and they are difficult to understand. As difficulty increases, ChatGPT's success decreases. Predictably, these results will change with more information loaded into ChatGPT.

Evaluating Anatomy Education in Kastamonu Çatalzeytin Vocational Higher Schools through Student Feedbacks

Aims: The aim of this study is to assess the attitudes of students in Vocational Higher Schools by obtaining their opinions, regarding the Anatomy course. Methods: This study is a survey-based research aimed at evaluating the opinions of 639 students enrolled in different programs (Disability Care and Rehabilitation, Physiotherapy, and Elderly Care Programs) at Kastamonu University Çatalzeytin Vocational Higher School regarding the Anatomy course, which they have taken theoretically. The survey questionnaire consists of 29 questions related to the evaluation of theoretical classes and eight questions related to the evaluation of exams. The survey questions were created online, and the participating students were provided with a link to access the survey questionnaire. Results: When asked about the contribution of the Anatomy course content to their professional development, 60.5% of the students responded positively, while 18.2% responded negatively. In regards to the inclusion of practical classes alongside theoretical classes, 80.9% of the students responded positively, while 7.1% responded negatively. Regarding the evaluation of exams, 63.8% of the students responded positively to the question of including visual-based questions in theoretical exams, while 19.5% responded negatively. When asked about the inclusion of practical exams alongside theoretical exams, 50% responded positively, while 27.6% responded negatively. Conclusions: Based on the findings of this study, which incorporates the opinions of the students, it is emphasized that practical classes should be included alongside theoretical classes in Anatomy education. Furthermore, it is noted that reminder sessions would be beneficial in the following years due to the tendency of forgetting the content of the Anatomy course.

Motivasi dan Tantangan Tamatan Perawat Dalam Mencapai Kelulusan NCLEX-RN di California

Introduction: Significant factors in influencing students' ability to pass the competency test include academic, cognitive, and individual aspects. Several factors that influence the success of the competency test from the perspective of student preparation in learning include motivation, reading ability, record management, time management, and critical thinking skills. Objective: This study aims to analyze the motivation and challenges of nurse graduates from Universitas Advent Indonesia in achieving graduation on the NCLEX-RN exam in California. Method: This study is a qualitative study with a phenomenological study that collects research data through in-depth interviews consisting of six UNAI alumni participants in California. Results: the results of the study in the form of research themes, namely: preparing for and requirements for the NCLEX exam, factors causing failure to pass the NCLEX exam, and motivation to take and pass the NCLEX exam. Conclusion: Nurses’ motivations in achieving NCLEX RN graduation in California are to improve finances, flexible working hours, make family happy, and be involved in complex patient care. Nurses’ challenges in achieving NCLEX RN graduation in California are meeting the administrative requirements for registering for the NCLEX RN exam, managing study time, understanding the material and the purpose of the exam questions, and adapting to the CAT exam system

Using active student participation to cultivate critical thinking in examinations

ABSTRACT Exams can be a stressful time in a student’s life where they are required to review notes and memorize facts and figures. Many students leave these exams and forget the material relatively quickly. An alternative pedagogical exam style where students create their own exam questions is a method that engages students through active participation and fosters critical thinking. The current paper discusses a student-led exam approach in a criminal justice course where students work together to create short answer exam questions that focus on the key course content. The questions require students to synthesize what they have learned and apply this information to suggest solutions for issues presented throughout the class. The answers must provide justification for their solutions and move beyond regurgitation of their notes. Students in this course have praised the exam style, explaining how the method has helped them learn, apply, and retain the information. The exam style can be applied to a multitude of courses and will likely benefit students following their graduation.

Arabic Question Generation Using Transformers

After the increased reliance on online education, online assessment became an essential tool for educators to remotely monitor and evaluate students’ understanding in order to assist them properly. However, the laborious process of creating exam questions is a challenge for most teachers. Thus, automated Question Generation aims to assist teachers by generating questions from given data. Limited research has been conducted to tackle this issue in the Arabic Language due to the complexity of the language and the limited amount of available Arabic data. This paper explores different implementations of the transformer models, that demonstrated their superiority in natural language processing. Three approaches were introduced to tackle this problem with Arabic data using an Arabic-based transformer, an English-based transformer, and a multilingual-based transformer. Each of the fine-tuned models was trained using ARCD, XGLUE, DialectBench, and ArabicQA data sets and evaluated on automatic and manual metrics. Two of the proposed models achieve state-of-the-art results on the Arabic question generation task. The English transformer obtained a ROUGE score of 0.59 on XGLUE, while the Arabic transformer model achieves 0.49 on ARCD. Both of these models demonstrate excellent quality of questions through human-conducted evaluations by achieving low WER and high GC, U, and A scores.

Performance evaluation of ChatGPT 4.0 on cardiovascular questions from the medical knowledge self-assessment program

Abstract Background Medical trainees are increasingly using online chat-based artificial intelligence (AI) platforms as supplementary resources for board exam preparation and clinical decision support. Prior studies have evaluated the performance of AI chatbots like ChatGPT on various general standardized tests such as the United States Medical Licensing Examination (USMLE), but little is known about their performance on subspecialty-focused exam questions, particularly related to clinical management and treatment. Objective This study aims to evaluate the performance of ChatGPT version 4.0 on the cardiovascular questions from the Medical Knowledge Self-Assessment Program (MKSAP) 19, a widely used resource for board exam preparation in the United States. Methods We submitted all cardiovascular questions from MKSAP 19 to ChatGPT 4.0, covering a broad range of cardiology topics in a multiple-choice format. Performance was gauged against both the official MKSAP answer key and average trainee scores obtained from the MKSAP website. Out of 129 questions, 4 were invalidated due to post-publication data, and 18 were excluded due to reliance on visual aids, leaving 107 questions for analysis. Results ChatGPT 4.0 correctly answered 93 out of 107 questions, reflecting an 87% accuracy rate, compared to a 60% accuracy rate averaged among all human users for the same questions (p<0.0001). ChatGPT accuracy rates for each question category (e.g., heart failure, electrophysiology, etc.) are provided in the figure. On the 14 questions that ChatGPT answered incorrectly, human users averaged a 47% accuracy rate. All 14 incorrectly answered questions were related to clinical management and treatment (as opposed to diagnosis or epidemiology), and the majority (10/14) involved choosing treatment medications or choosing between medications versus interventions. Conclusion ChatGPT 4.0 surpassed the typical 70% accuracy threshold required for passing the internal medicine board certification but performed less well in answering questions about clinical management and treatment. This performance suggests that ChatGPT 4.0 could be an effective adjunct tool for cardiology education, but raises some concerns about its use in clinical decision support for trainees.

Performance Assessment of GPT 4.0 on the Japanese Medical Licensing Examination.

To evaluate the accuracy and parsing ability of GPT 4.0 for Japanese medical practitioner qualification examinations in a multidimensional way to investigate its response accuracy and comprehensiveness to medical knowledge. We evaluated the performance of the GPT 4.0 on Japanese Medical Licensing Examination (JMLE) questions (2021-2023). Questions are categorized by difficulty and type, with distinctions between general and clinical parts, as well as between single-choice (MCQ1) and multiple-choice (MCQ2) questions. Difficulty levels were determined on the basis of correct rates provided by the JMLE Preparatory School. The accuracy and quality of the GPT 4.0 responses were analyzed via an improved Global Qualily Scale (GQS) scores, considering both the chosen options and the accompanying analysis. Descriptive statistics and Pearson Chi-square tests were used to examine performance across exam years, question difficulty, type, and choice. GPT 4.0 ability was evaluated via the GQS, with comparisons made via the Mann-Whitney U or Kruskal-Wallis test. The correct response rate and parsing ability of the GPT4.0 to the JMLE questions reached the qualification level (80.4%). In terms of the accuracy of the GPT4.0 response to the JMLE, we found significant differences in accuracy across both difficulty levels and option types. According to the GQS scores for the GPT 4.0 responses to all the JMLE questions, the performance of the questionnaire varied according to year and choice type. GTP4.0 performs well in providing basic support in medical education and medical research, but it also needs to input a large amount of medical-related data to train its model and improve the accuracy of its medical knowledge output. Further integration of ChatGPT with the medical field could open new opportunities for medicine.

The association between family socio-demographic factors, parental mediation and adolescents’ digital literacy: a cross-sectional study

BackgroundDigital technology has a two-sided impact whether it is used appropriately or not. It is crucial to promote the benefits of digital technology and reduce its harms by bolstering adolescents’ digital literacy. The appropriateness of the strategies parents use to manage their children’s digital lives plays a key role. This study aims to investigate the separate effects of family socio-demographic factors and varied dimensions of parental mediation on adolescents’ digital literacy separately and simultaneously to explore the appropriate parental mediation strategies even in the absence of favorable family status.MethodsA cross-sectional survey was utilized with a sample of 1894 Chinese adolescents in junior and senior high schools (mean age = 14.01, 48.94% female), using a set of exam questions to test children’s objective digital literacy online. Family socio-demographic factors (parental marital status, economic status, and parental education levels) and varied dimensions of parental mediation were included as predictors of digital literacy. Hierarchical regression analysis was used to estimate the relation between the variables.ResultsHigher parental education levels were positively associated with adolescents’ parental mediation and digital literacy while better family economic status and parental divorce were negatively associated. Among the dimensions of parental mediation, active mediation, restrictive mediation and supervision were significantly associated with adolescents’ digital literacy, but not for co-use. Active mediation and supervision showed significant positive associations with adolescents’ digital literacy, whereas, co-use showed a significant negative association when all parental mediation dimensions were integrated into the model, The significant association between restrictive mediation and digital literacy disappeared. Parental mediation has diluted the impact of family socio-demographic factors on adolescents’ digital literacy.ConclusionsEffective interventions are needed to promote parental active mediation and supervision to help adolescents develop digital literacy and navigate the digital world, especially for adolescents who do not have highly educated parents or whose parents are divorced.

Evaluation and mitigation of cognitive biases in medical language models.

Increasing interest in applying large language models (LLMs) to medicine is due in part to their impressive performance on medical exam questions. However, these exams do not capture the complexity of real patient-doctor interactions because of factors like patient compliance, experience, and cognitive bias. We hypothesized that LLMs would produce less accurate responses when faced with clinically biased questions as compared to unbiased ones. To test this, we developed the BiasMedQA dataset, which consists of 1273 USMLE questions modified to replicate common clinically relevant cognitive biases. We assessed six LLMs on BiasMedQA and found that GPT-4 stood out for its resilience to bias, in contrast to Llama 2 70B-chat and PMC Llama 13B, which showed large drops in performance. Additionally, we introduced three bias mitigation strategies, which improved but did not fully restore accuracy. Our findings highlight the need to improve LLMs' robustness to cognitive biases, in order to achieve more reliable applications of LLMs in healthcare.

Evaluating the Performance of ChatGPT, Gemini, and Bing Compared with Resident Surgeons in the Otorhinolaryngology In-service Training Examination.

Large language models (LLMs) are used in various fields for their ability to produce human-like text. They are particularly useful in medical education, aiding clinical management skills and exam preparation for residents. To evaluate and compare the performance of ChatGPT (GPT-4), Gemini, and Bing with each other and with otorhinolaryngology residents in answering in-service training exam questions and provide insights into the usefulness of these models in medical education and healthcare. Eight otorhinolaryngology in-service training exams were used for comparison. 316 questions were prepared from the Resident Training Textbook of the Turkish Society of Otorhinolaryngology Head and Neck Surgery. These questions were presented to the three artificial intelligence models. The exam results were evaluated to determine the accuracy of both models and residents. GPT-4 achieved the highest accuracy among the LLMs at 54.75% (GPT-4 vs. Gemini p=0.002, GPT-4 vs. Bing p<0.001), followed by Gemini at 40.50% and Bing at 37.00% (Gemini vs. Bing p=0.327). However, senior residents outperformed all LLMs and other residents with an accuracy rate of 75.5% (p<0.001). The LLMs could only compete with junior residents. GPT- 4 and Gemini performed similarly to juniors, whose accuracy level was 46.90% (p=0.058 and p=0.120, respectively). However, juniors still outperformed Bing (p=0.019). The LLMs currently have limitations in achieving the same medical accuracy as senior and mid-level residents. However, they outperform in specific subspecialties, indicating the potential usefulness in certain medical fields.

Enhancing exam question quality in medical education through bootstrapping.

Medical schools are required to assess and evaluate their curricula and to develop exam questions with strong reliability and validity evidence, often based on data derived from statistically small samples of medical students. Achieving a large enough sample to reliably and validly evaluate courses, assessments, and exam questions would require extensive data collection over many years, which is inefficient, especially in the fast-changing educational environment of medical schools. This article demonstrates how advanced quantitative methods, such as bootstrapping, can provide reliable data by resampling a single dataset to create many simulated samples. This economic approach, among others, allows for the creation of confidence intervals and, consequently, the accurate evaluation of exam questions as well as broader course and curriculum assessments. Bootstrapping offers a robust alternative to traditional methods, improving the psychometric quality of exam questions, and contributing to fair and valid assessments in medical education.

Implementation of HOTS-based Instructions in Pesantren: A Case Study in Pondok Modern Tazakka Batang

The development of Higher Order Thinking Skills (HOTS) in students at Islamic boarding schools (pesantren) is crucial due to their numerous advantages, such as the capacity to tackle social, scientific, and practical problems with effectiveness. Whether or not Pondok Modern Tazakka successfully integrated the use of HOTS questions in exams is the goal of this study. Through the application of descriptive phenomenological research methods and data collection techniques, including the tabulation of 132 test sheets in 44 subjects tested on KMI Class 6 students and interviews with 62 teachers in 8 subjects, it was discovered that 8 subjects exhibited no HOTS questions at all (19%), 6 subjects exhibited a decreasing number of HOTS questions (14%), 19 subjects exhibited a sharp increase in HOTS questions (45%), and 9 subjects exhibited a constant number of HOTS questions. The study's findings also demonstrate that, as long as supervisors (senior teachers) take a more proactive approach to giving junior teachers comprehensive training and ensure that all exam questions in each subject are HOTS exam questions, it is possible to guarantee that all exam questions will contain HOTS.

An analysis of calculus application questions in grade 12 mathematics exams in South Africa

This study employed a quantitative survey methodology to analyze the 2021-2023 grade 12 calculus application exam questions in the context of South Africa. A total of 11 grade 12 teachers and 220 grade 12 students participated in the study. Analysis of the survey data and students’ test results points to the view that examiners sometimes misjudge the level of difficulty of the exam questions and include questions that are very difficult, which fail to discriminate between strong and weak candidates. It was noticed that calculus applications questions that most students could not correctly answer included multiple mathematical skills that were compacted in a short statement. This paper recommends that mathematics examiners should continuously reflect on their own work and find ways to improve their skills in setting future exam papers. The inclusion of item difficulty and item discrimination indices in post-exam analysis of results could be part of the examiners’ reflective practice. Feedback from the grade 12 mathematics teachers on the standard of each exam question is just as important to the examiners as the examiners’ reports are to the teachers.

The Analysis of the Characters of Wording for High School Classical Chinese

Classical Chinese carries the splendid culture of China. It is the important learning content for Chinese in high school. At the same time, the scoring rate is not good in the corresponding examination. Reading and comprehending classical Chinese in high school is the hardest part, especially for students who are not good at language learning. It is meaningful to analyze the characters of wording for classical Chinese. In this paper, NLP (Natural Language Processing) technology is applied to analyze the word usage for classical Chinese in high school. Firstly, ancient poems and literature in teaching materials in high school and classical Chinese exam questions of the past 15 years are collected. Secondly, raw data is preprocessed, cut words, and calculated word frequencies. Finally, top N words are generated and the experimental data is analyzed. According to the experimental results, the methods proposed in this paper can be used to analyze words for classical Chinese in high school. The Top N words are calculated and analyzed. It provides a good reference for high school students to learn classical Chinese.

7870 Exploring The Role Of ChatGPT In Pediatric Endocrinology Education; Exam Preparation And Question Generation

Abstract Disclosure: J. Tarkoff: None. A.G. Martinez Sanchez: None. Large language models (LLMs) hold substantial promise for improving physician knowledge and expertise. Their role in medical education and the generation of diverse diagnoses can be crucial, potentially leading to positive effects on clinical outcomes. The New England Journal of Medicine pediatric case challenges have examined ChatGPT [1]. Yet, its performance in specialized areas like Pediatric Endocrinology remains unexplored. This study assesses the effectiveness of ChatGPT 4 in responding to questions from the Pediatric Endocrine Self-Assessment Program (PESAP). It also examines if the AI is suitable for creating an educational quiz for residents and fellows. Methods: ChatGPT 4 underwent testing with questions from the 2021-2022 version of PESAP, utilizing the prompt: “Can you assist from the perspective of a pediatric endocrinologist with the following patient case”. Responses were evaluated for initial correctness, and performance was analyzed across various competency categories corresponding to the 7 “umbrella sections” of the tool (Adrenal, Bone, Carbohydrate and Lipid Metabolism/Obesity, Growth, Pituitary, Reproductive System, and Thyroid). Subsequently, we personalized the model by incorporating the questions and detailed responses from the PESAP. This customization resulted in a model that was then utilized to create a 10-question proof-of-concept quiz for four board-certified pediatric endocrinologists. The quiz included a scoring system designed to measure the extent and depth of ChatGPT knowledge. Results: ChatGPT 4 accurately answered 52% of PESAP questions, demonstrating varying performance across specific categories, ranging from 30% (Adrenal) to 78% (Reproductive System). In 16 questions, ChatGPT 4 did not provide an initial answer, requiring a specific request for a response. For questions related to thyroid cancer, explicit prompts were necessary to instruct responses based on the American Thyroid Association 2015 guidelines. In the endocrinologist quiz, the average score was 80%, ranging from 60% to 100%. Discussion: Currently, using ChatGPT 4 as the final diagnostic tool, especially in pediatric endocrinology, should be approached with caution, based on our assessment. However, when incorporating distinct Pediatric Endocrinology case studies, ChatGPT 4 successfully generated valuable educational questions, reinforcing fundamental concepts in the field. We anticipate that as LLMs advance and receive direct medical training, the out-of-the-box diagnostic accuracy will improve, leading to a transformative impact on medical education through this technology.

Performance of ChatGPT on Hand Surgery Board-Style Examination Questions

Background The performance of large-language models, such as ChatGPT, on medical and sub-specialty examinations has been preliminarily explored in fields such as radiology, obstetrics and gynecology, and orthopedic surgery. However, no literature assessing ChatGPT’s ability to answer hand surgery exam questions exists. This study’s purpose was to evaluate ChatGPT’s performance on hand surgery board-style examination questions. Methods All questions from the American Society for Surgery of the Hand (ASSH) Hand 100 Exam, Beginner, and Intermediate Assessment tools were entered into ChatGPT-3.5. Responses were regenerated two times to identify inconsistencies. Duplicate questions, questions with figures and/or videos, and questions that ChatGPT refused to provide a response to were excluded. ChatGPT’s correct response rate, answer modifications, and human accuracy were recorded. Results 117 questions from the 3 assessment tools were analyzed: 49 from the ASSH Hand 100, 32 from the Beginner, and 36 from the Intermediate Assessment tools. On ChatGPT’s initial attempt, 40.82% (20/49), 50.0% (16/32), 38.89% (14/36) of questions were correctly answered, respectively. Overall, ChatGPT correctly answered 50/117 (42.7%) of questions on the first try. ChatGPT excelled in topics (&gt;60% correct) of mass/tumor, nerve, wrist, and performed poorly (&lt;40% correct) on topics regarding anatomy/basic science/imaging, brachial plexus, congenital, elbow, tendon, and vascular disorders, trauma. On the Beginner and Intermediate Exams, humans correctly answered 56.64% and 62.73% of questions, respectively. Conclusions ChatGPT can correctly answer simpler hand surgery questions but performed poorly when compared to humans on higher-difficulty questions.