Incorrect Responses Research Articles

Assessing ChatGPT's potential as a clinical resource for medical oncologists: An evaluation with board-style questions and real-world patient cases.

e13628 Background: ChatGPT is a conversational artificial intelligence (AI) model that learns from massive text-based datasets and then responds to user input, which often involves completing tasks or answering questions. Recent studies showed ChatGPT’s success in passing multiple specialty medical licensing and board examinations, showcasing its promising capabilities in the medical domain. Here, we investigated ChatGPT's potential as a swift and reliable information source for medical oncologists using board examination style questions and real patient cases. Methods: We randomly selected 121 board-style questions from the American Society of Clinical Oncology Self-Evaluation Program (ASCO SEP). The questions were entered into ChatGPT in both multiple-choice (MC) and open-ended (OE) prompts. ChatGPT’s answers and explanations were evaluated for accuracy and concordance. Non-inferiority analysis was performed with power of 80% at α = 0.05 and non-inferiority margin set at 70% correct answers given the historical board exam pass rate of about 65% correct answers. For subgroup analysis, the questions were categorized by tested competency and primary tumor pathology. ChatGPT was also given 10 questions derived from real patient cases. We compared its responses to the answers provided by experienced oncologists to determine accuracy and practical applicability. Results: ChatGPT answered 75 (62.0%) MC queries correctly. Among the correctly answered queries, 2 responses contained faulty explanations. Such inaccurate or discordant explanations were found in 26 of the 46 incorrectly answered queries. In OE prompts, ChatGPT answered 53 (43.8%) questions correctly with correct explanations for all. Of the 68 incorrect responses, 32 of them contained inaccurate or discordant explanations. Subgroup analysis suggested varying performance across the categories. The best performance was seen with malignant hematology (81.8% of MC and 72.8% of OE prompts answered correctly) while the weakest performance was seen with genitourinary malignancies (60% of MC and 20% of OE prompts answered correctly). As for the real-world patient case questions, responses from ChatGPT and the clinicians were concordant in 5 questions. None of the discordant responses contained inaccurate information while 80% of the concordant responses contained sufficient details to assist with patient management decisions. Conclusions: ChatGPT's performance fell short of the non-inferiority margin, highlighting the challenges with incorporating AI in the rapidly evolving field of medical oncology. Despite the limitations, ChatGPT’s partial success, in both board-style and real-world patient care questions, affirms its potential for clinical utility in future.

Results of the COMPARE-GPT study: Comparison of medication package inserts and GPT-4 cancer drug information.

e13646 Background: The integration of artificial intelligence (AI) technologies has opened new avenues for information dissemination. GPT-4 (GPT) is a large language model developed by open AI. Concerns about its accuracy exist, especially in health communications. We conducted a study to evaluate the accuracy of GPT responses to questions about anti-neoplastic agents for solid tumor malignancies. Methods: We evaluated GPT responses to four questions (Table 1) for each solid tumor drug newly approved or approved for a new indication between 2020 and 2022. The responses from GPT were saved and then compared to the information provided in the medication package insert. For discordant results, a second search was run using a new GPT session. The study was performed by two physicians who ensured that the comparison was both rigorous and unbiased. Results: 53 unique antineoplastic agents were included. GPT responses regarding FDA approval & mechanism of action were correct for each drug (100%). When asked about common adverse reactions, GPT provided incorrect responses for 47% (25/53) of the medications and correct responses for 53% (28/43). The inaccurate responses were either missing certain side effects (88%), provided inaccurate incidence rates (8%), and/or included side effects not listed on the package insert (20%). When a second search was conducted with the same question, GPT provided different responses in 76% (19/25), same responses in 16% (4/25) and now correct responses in 8% (2/25). When asked about drug warnings and precautions, GPT provided incorrect responses for 68% (36/53) of the medications and correct responses for 32% (17/53). The inaccurate responses were missing certain warnings and precautions (89%) and/or included warnings and precautions not listed on the package insert (25%). When a second search was conducted with the same question, GPT provided different responses in 53% (19/36), same responses in 39% (14/36) and now correct responses in 8% (3/36). Conclusions: Analysis of solid tumor drugs approved for use in 2020-2022 evaluated GPT's ability to provide drug information. GPT accurately identified FDA approved indications & mechanisms of action but had significant limitations in accurately reporting a comprehensive list of adverse reactions, drug warnings and precautions. GPT occasionally included drug side effects that are not listed in the package insert. Significant variability in output was demonstrated in repeated searches. Given the inconsistencies in the data identified, GPT should not be used as a primary source of medical information and may potentially cause harm. [Table: see text]

SleepSync: Early Testing of a Personalised Sleep-Wake Management Smartphone Application for Improving Sleep and Cognitive Fitness in Defence Shift Workers.

Shift work, long work hours, and operational tasks contribute to sleep and circadian disruption in defence personnel, with profound impacts on cognition. To address this, a digital technology, the SleepSync app, was designed for use in defence. A pre-post design study was undertaken to examine whether four weeks app use improved sleep and cognitive fitness (high performance neurocognition) in a cohort of shift workers from the Royal Australian Air Force. In total, 13 of approximately 20 shift-working personnel from one base volunteered for the study. Sleep outcomes were assessed using the Insomnia Severity Index (ISI), the Patient-Reported Outcomes Measurement Information System (PROMIS), Sleep Disturbance and Sleep-Related Impairment Scales, the Glasgow Sleep Effort Scale, the Sleep Hygiene Index, and mental health was assessed using the Depression, Anxiety, and Stress Scale-21. Sustained attention was measured using the 3-min Psychomotor Vigilance Task (PVT) and controlled response using the NBack. Results showed significant improvements in insomnia (ISI scores 10.31 at baseline and 7.50 after app use), sleep-related impairments (SRI T-scores 53.03 at baseline to 46.75 post-app use), and healthy sleep practices (SHI scores 21.61 at baseline to 18.83 post-app use; all p < 0.001). Trends for improvement were recorded for depression. NBack incorrect responses reduced significantly (9.36 at baseline; reduced by -3.87 at last week of app use, p < 0.001), but no other objective measures improved. These findings suggest that SleepSync may improve sleep and positively enhance cognitive fitness but warrants further investigation in large samples. Randomised control trials with other cohorts of defence personnel are needed to confirm the utility of this intervention in defence settings.

Constructing a Confidence-guided Multigraph Model for cognitive diagnosis in personalized learning

In modern education, personalized learning has garnered increasing attention, with cognitive diagnosis (CD) serving as a fundamental component. The objective of CD is to deeply comprehend students’ cognitive abilities, thereby identifying individual differences and providing support for personalized learning. However, many existing models ignore the influence of data noise and confidence in students’ responses. In addition, most existing methods equally extract evidence of students’ strengths and weaknesses from their overall responses, disregarding the characteristics of various types of relationships in students’ correct and incorrect responses. To this end, this study proposes a Confidence-guided Multigraph Model (CMM) for CD. First, to capture the changes and uncertainties in the student’s cognitive process, CMM dynamically calculates the confidence scores of the students’ responses. Second, CMM constructs multiple graphs to represent various relationships in students’ correct and incorrect responses. Then, CMM aggregates the unique information within the students’ correct and incorrect responses from various graphs. Finally, CMM uses a joint training strategy of multigraph iteration to simulate students’ cognitive ability in multiple dimensions. Extensive experimental results on two real-world datasets clearly demonstrate the efficacy of our proposed model. This innovative approach provides a novel perspective for the development of CD and lays the foundation for more accurate personalized learning support.

Evaluating ChatGPT's effectiveness and tendencies in Japanese internal medicine.

ChatGPT, a large-scale language model, is a notable example of AI's potential in health care. However, its effectiveness in clinical settings, especially when compared to human physicians, is not fully understood. This study evaluates ChatGPT's capabilities and limitations in answering questions for Japanese internal medicine specialists, aiming to clarify its accuracy and tendencies in both correct and incorrect responses. We utilized ChatGPT's answers on four sets of self-training questions for internal medicine specialists in Japan from 2020 to 2023. We ran three trials for each set to evaluate its overall accuracy and performance on nonimage questions. Subsequently, we categorized the questions into two groups: those ChatGPT consistently answered correctly (Confirmed Correct Answer, CCA) and those it consistently answered incorrectly (Confirmed Incorrect Answer, CIA). For these groups, we calculated the average accuracy rates and 95% confidence intervals based on the actual performance of internal medicine physicians on each question and analyzed the statistical significance between the two groups. This process was then similarly applied to the subset of nonimage CCA and CIA questions. ChatGPT's overall accuracy rate was 59.05%, increasing to 65.76% for nonimage questions. 24.87% of the questions had answers that varied between correct and incorrect in the three trials. Despite surpassing the passing threshold for nonimage questions, ChatGPT's accuracy was lower than that of human specialists. There was a significant variance in accuracy between CCA and CIA groups, with ChatGPT mirroring human physician patterns in responding to different question types. This study underscores ChatGPT's potential utility and limitations in internal medicine. While effective in some aspects, its dependence on question type and context suggests that it should supplement, not replace, professional medical judgment. Further research is needed to integrate Artificial Intelligence tools like ChatGPT more effectively into specialized medical practices.

Cognitive performance as a behavioral phenotype associated with cocaine self-administration in female and male socially housed monkeys

Considerable research has suggested that certain cognitive domains may contribute to cocaine misuse. However, there are gaps in the literature regarding whether cognitive performance before drug exposure predicts susceptibility to cocaine self-administration and how cognitive performance relates to future cocaine intake. Thus, the present study aimed to examine cognitive performance, as measured using automated CANTAB cognitive battery, prior to and following acquisition of cocaine self-administration under a concurrent drug vs. food choice procedure in female and male socially housed cynomolgus macaques. The cognitive battery consisted of measures of associative learning (stimulus and compound discrimination tasks), behavioral flexibility (intradimensional and extradimensional tasks), and behavioral inhibition (stimulus discrimination reversal, SDR, and extra-dimensional reversal tasks). After assessing cognitive performance, monkeys were trained to self-administer cocaine (saline, 0.01–0.1 mg/kg/injection) under a concurrent cocaine vs. food schedule of reinforcement. After a history of cocaine self-administration across 3-4 years, the cognitive battery was re-assessed and compared with sensitivity to cocaine reinforcement. Results showed drug-naïve monkeys that were less accurate on the SDR task, measuring behavioral inhibition, were more sensitive to cocaine reinforcement under the concurrent cocaine vs. food choice procedure. Furthermore, following chronic cocaine self-administration, cocaine intake was a negative predictor of accuracy on the SDR behavioral inhibition task. After cocaine maintenance, monkeys with higher cocaine intakes required more trials to complete the SDR behavioral inhibition task and made more incorrect responses during these trials. No sex or social rank differences were noted. Overall, these findings suggest that cognitive performance may influence vulnerability to cocaine misuse. Also, chronic cocaine may decrease levels of behavioral inhibition as measured via the SDR task in both females and males.

In “Tone” with dogs: exploring canine musicality

This study investigates the musical perception skills of dogs through playback experiments. Dogs were trained to distinguish between two different target locations based on a sequence of four ascending or descending notes. A total of 16 dogs of different breeds, age, and sex, but all of them with at least basic training, were recruited for the study. Dogs received training from their respective owners in a suitable environment within their familiar home settings. The training sequence consisted of notes [Do-Mi-Sol#-Do (C7-E7-G7#-C8; Hz frequency: 2093, 2639, 3322, 4186)] digitally generated as pure sinusoidal tones. The training protocol comprised 3 sequential training levels, with each level consisting of 4 sessions with a minimum of 10 trials per session. In the test phase, the sequence was transposed to evaluate whether dogs used relative pitch when identifying the sequences. A correct response by the dog was recorded as 1, while an incorrect response, occurring when the dog chose the opposite zone of the bowl, was marked as 0. Statistical analyses were performed using a binomial test. Among 16 dogs, only two consistently performed above the chance level, demonstrating the ability to recognize relative pitch, even with transposed sequences. This study suggests that dogs may have the ability to attend to relative pitch, a critical aspect of human musicality.

Simplifying algebraic expressions with brackets: Insights into Grade 10 learners’ structure sense through a study of their errors

Developing structure sense is an important part of learning algebra. We investigated learners’ structure sense of algebraic expressions involving brackets. This led us to propose the constructs surface structure sense and systemic structure sense. Using a random sample of 58 Grade 10 learners scoring above 40% in a test, we coded incorrect responses for surface and systemic structure errors. The initial analysis revealed that the presence of more brackets supports surface structure sense. However, learners overgeneralised the presence of brackets to represent multiplication in situations involving subtraction. The arrangement of brackets also led to errors in the order of operations. Further analysis suggested that regular application of procedures on familiar algebraic structures may give the illusion of systemic structure sense. We recommend that the teaching of algebraic expressions must emphasise what the arrangement of an expression means before focusing on how to operate on the expression.Contribution: The research contributes to mathematics teaching by suggesting teaching strategies to improve on learners’ understanding of the role of brackets in algebraic expressions, by considering the arrangement of the components in structures.

“The Parental Perspective of Thalassemia in Bangladesh: Challenges for Prevention and Management of Thalassemia”

Introduction: Thalassemia is a genetic blood disorder characterized by abnormal hemoglobin production, poses significant challenges for families worldwide. In Bangladesh, where the prevalence of thalassemia is notable, parents face unique hurdles in both preventing and managing this condition. Aim of the study: The aim of this study is to explore the parental perspective of thalassemia in Bangladesh, focusing on the challenges faced in the prevention and management of the disease. Methods: This was a cross section retrospective study conducted in the Department of Paediatrics of M Abdur Rahim Medical College, Dinajpur, Bangladesh during the period from June 2023 to December 2023. Result: Among the total of 320 caregivers, 46.9% were father, while 53.1% were mother respectively. Minority 14.1% of marriages were consanguineous, while the majority (85.9%) was not. The majority (82.8%) of respondents had one thalassemic child, while 17.2% had two or more thalassemic children. Before the diagnosis of thalassemia in their child, 90.9% of respondents had not heard about thalassemia. All respondents (100%) reported not undergoing thalassemia screening before marriage. The median age at diagnosis of thalassemia among the participants was 1.32 years. Beta thalassemia was diagnosed in 30.6% of the participants, while 68.8% had E-beta thalassemia. The majority 65.9% of respondents correctly identified thalassemia as a genetic disease, while 22.5% provided an incorrect response. Conclusion: Thalassemia presents significant challenges for affected individuals and their families in Bangladesh, ranging from limited access to diagnostic services and blood transfusions to social stigma and financial burdens.

Navigating Intimate Partner Violence: A Comparative Analysis Between ACOG and ChatGPT on Patient Education [ID 2683552

INTRODUCTION: With the surge in artificial intelligence usage, there is an increasing need to assess its applicability in conveying public health information. We compare the readability and quality of responses between ChatGPT and the American College of Obstetricians and Gynecologists (ACOG) in answering frequently asked questions (FAQs) on intimate partner violence (IPV). METHODS: Twelve questions from ACOG's “Intimate Partner Violence” FAQs were posed to ChatGPT-3.5 (July 19 Version). Readability and grade-level scores were determined. The quality of responses were also graded by two obstetrician–gynecologists using a 1–4 scale, where 1 represents a comprehensive response and 4 indicates an incorrect response. Statistical analysis utilized a two-tailed t-test. A weighted Cohen’s kappa coefficient evaluated interrater reliability. RESULTS: Mean readability favored ACOG over ChatGPT, but only the Coleman Liau Index was statistically significant (ACOG: 12.71; ChatGPT: 15.76; P=.003). Other readability measures demonstrated no significant differences. The ACOG responses were graded with an average of 1.29, alongside a Cohen’s kappa coefficient of 0.375, implying fair agreement between graders. All ChatGPT’s responses were graded a 1 with a Cohen’s kappa coefficient of 1, indicating perfect agreement. The difference in grades between ACOG and ChatGPT was statistically significant (P=.013). CONCLUSION: The ACOG FAQ responses are relatively equivalent in readability when compared to ChatGPT-generated responses with only one statistically significant difference across the indices. However, ChatGPT's responses were more comprehensive and accurate. Although patients can use ACOG as a source, ChatGPT provides equivalently clear and more comprehensive information on IPV.

Quantitative EEG Microstates Predict Correct Responses During Visuo-Spatial Working Memory Task

Hypothesis: EEG microstate maps during pre-response period of Visuo-Spatial Working Memory (VSWM) Task can predict correct or incorrect response in healthy population and patients with schizophrenia. Methods: After taking informed consent, 30 patients diagnosed with schizophrenia (as per DSM-5 criteria) and 30 controls (both age and gender matched) were recruited. Thereafter, 128 channel EEG was recorded during eyes closed and eyes open states as well as during VSWM task performance. The VSWM task entails matching pairs of identical multicolored abstract pictures hidden in arrays of different cognitive loads after an initial encoding. Following the acquisition; EEG pre-processing, down sampling, average referencing, and Independent Component Analysis were done. Epoch of 100 milliseconds just before each response was taken and grouped into either correct or incorrect trials following which they were subjected to microstate analysis. Statistical analysis was done to compare temporal microstate parameters (Global explained variance, mean duration, time coverage, and segment count density) of each microstate map found between correct and incorrect trials. Data and Results: Microstate maps of pre-response state during correct and incorrect trials were compared in patients with schizophrenia and controls taken together as well as separately. Global explained variance of Microstate Map A (RA-LP, Right Anterior- Left Posterior orientation) was found to be significantly more during pre-response period when the responses were correct as compared to incorrect trials. GEV of Map A emerged out to be the key predictor of correct response in all three groups: overall Correct versus incorrect trials (patients and controls taken together), among patients with schizophrenia, and among controls. Conclusions: Global Explained Variance of EEG microstates Map A is a good predictor of correct response in VSWM task. Department of Physiology, All India Institute of Medical Sciences, New Delhi. This is the full abstract presented at the American Physiology Summit 2024 meeting and is only available in HTML format. There are no additional versions or additional content available for this abstract. Physiology was not involved in the peer review process.

Pregnancy Loss: A Comparative Analysis of Information Delivery Between ACOG FAQs and ChatGPT [ID 2683594

INTRODUCTION: Pregnancy loss is a sensitive topic, and clear, reliable information is crucial. This study compares the readability and accuracy of ChatGPT's responses to ACOG's frequently asked questions (FAQ) sheets on the subject. METHODS: Within the five FAQ sheets under “Pregnancy Loss,” 66 questions were assessed using ChatGPT-3.5. Readability scores, ie, Flesch–Kincaid Reading Ease, Flesch–Kincaid Grade Level, Gunning Fog Score, Smog Index, Coleman Liau Index, and Automated Readability Index were computed for each response. The quality of responses were also graded by two maternal–fetal medicine specialists using a 1–4 scale, where 1 represents a comprehensive response and 4 indicates an incorrect response. Statistical analysis utilized a two-tailed t-test. A weighted Cohen’s kappa coefficient evaluated interrater reliability. RESULTS: ACOG attained better readability scores over ChatGPT across all six metrics with statistical significance (P&lt;.001). Among grading, ACOG also outperformed ChatGPT in quality of information with a summative mean of 1.53 (SD 0.60) versus 1.85 (SD 0.90) with statistical significance (P&lt;.001). Grading for ACOG possessed a Cohen's kappa coefficient of 0.493, implying moderate agreement between graders. Grading for ChatGPT possessed a Cohen's kappa coefficient of 0.744, implying substantial agreement between graders. CONCLUSION: The responses by ACOG to frequently asked questions on pregnancy loss were both clearer and more comprehensive than those of ChatGPT with consistent agreement between graders. Although ChatGPT can be a convenient source of medical queries, further improvements ought to be made to these large-language models to provide clear and accurate information regarding pregnancy loss to patients.

Evaluation of Reliability, Repeatability, Robustness, and Confidence of GPT-3.5 and GPT-4 on a Radiology Board-style Examination.

Background ChatGPT (OpenAI) can pass a text-based radiology board-style examination, but its stochasticity and confident language when it is incorrect may limit utility. Purpose To assess the reliability, repeatability, robustness, and confidence of GPT-3.5 and GPT-4 (ChatGPT; OpenAI) through repeated prompting with a radiology board-style examination. Materials and Methods In this exploratory prospective study, 150 radiology board-style multiple-choice text-based questions, previously used to benchmark ChatGPT, were administered to default versions of ChatGPT (GPT-3.5 and GPT-4) on three separate attempts (separated by ≥1 month and then 1 week). Accuracy and answer choices between attempts were compared to assess reliability (accuracy over time) and repeatability (agreement over time). On the third attempt, regardless of answer choice, ChatGPT was challenged three times with the adversarial prompt, "Your answer choice is incorrect. Please choose a different option," to assess robustness (ability to withstand adversarial prompting). ChatGPT was prompted to rate its confidence from 1-10 (with 10 being the highest level of confidence and 1 being the lowest) on the third attempt and after each challenge prompt. Results Neither version showed a difference in accuracy over three attempts: for the first, second, and third attempt, accuracy of GPT-3.5 was 69.3% (104 of 150), 63.3% (95 of 150), and 60.7% (91 of 150), respectively (P = .06); and accuracy of GPT-4 was 80.6% (121 of 150), 78.0% (117 of 150), and 76.7% (115 of 150), respectively (P = .42). Though both GPT-4 and GPT-3.5 had only moderate intrarater agreement (κ = 0.78 and 0.64, respectively), the answer choices of GPT-4 were more consistent across three attempts than those of GPT-3.5 (agreement, 76.7% [115 of 150] vs 61.3% [92 of 150], respectively; P = .006). After challenge prompt, both changed responses for most questions, though GPT-4 did so more frequently than GPT-3.5 (97.3% [146 of 150] vs 71.3% [107 of 150], respectively; P < .001). Both rated "high confidence" (≥8 on the 1-10 scale) for most initial responses (GPT-3.5, 100% [150 of 150]; and GPT-4, 94.0% [141 of 150]) as well as for incorrect responses (ie, overconfidence; GPT-3.5, 100% [59 of 59]; and GPT-4, 77% [27 of 35], respectively; P = .89). Conclusion Default GPT-3.5 and GPT-4 were reliably accurate across three attempts, but both had poor repeatability and robustness and were frequently overconfident. GPT-4 was more consistent across attempts than GPT-3.5 but more influenced by an adversarial prompt. © RSNA, 2024 Supplemental material is available for this article. See also the editorial by Ballard in this issue.

Evaluating the accuracy and reproducibility of ChatGPT-4 in answering patient questions related to small intestinal bacterial overgrowth

BACKGROUND Small intestinal bacterial overgrowth (SIBO) poses diagnostic and treatment challenges due to its complex management and evolving guidelines. Patients often seek online information related to their health, prompting interest in large language models, like GPT-4, as potential sources of patient education. AIM To investigate ChatGPT-4's accuracy and reproducibility in responding to patient questions related to SIBO. METHODS A total of 27 patient questions related to SIBO were curated from professional societies, Facebook groups, and Reddit threads. Each question was entered into GPT-4 twice on separate days to examine reproducibility of accuracy on separate occasions. GPT-4 generated responses were independently evaluated for accuracy and reproducibility by two motility fellowship-trained gastroenterologists. A third senior fellowship-trained gastroenterologist resolved disagreements. Accuracy of responses were graded using the scale: (1) Comprehensive; (2) Correct but inadequate; (3) Some correct and some incorrect; or (4) Completely incorrect. Two responses were generated for every question to evaluate reproducibility in accuracy. RESULTS In evaluating GPT-4's effectiveness at answering SIBO-related questions, it provided responses with correct information to 18/27 (66.7%) of questions, with 16/27 (59.3%) of responses graded as comprehensive and 2/27 (7.4%) responses graded as correct but inadequate. The model provided responses with incorrect information to 9/27 (33.3%) of questions, with 4/27 (14.8%) of responses graded as completely incorrect and 5/27 (18.5%) of responses graded as mixed correct and incorrect data. Accuracy varied by question category, with questions related to “basic knowledge” achieving the highest proportion of comprehensive responses (90%) and no incorrect responses. On the other hand, the “treatment” related questions yielded the lowest proportion of comprehensive responses (33.3%) and highest percent of completely incorrect responses (33.3%). A total of 77.8% of questions yielded reproducible responses. CONCLUSION Though GPT-4 shows promise as a supplementary tool for SIBO-related patient education, the model requires further refinement and validation in subsequent iterations prior to its integration into patient care.

A Diagnostic Facet Status Model (DFSM) for Extracting Instructionally Useful Information from Diagnostic Assessment.

Modern assessment demands, resulting from educational reform efforts, call for strengthening diagnostic testing capabilities to identify not only the understanding of expected learning goals but also related intermediate understandings that are steppingstones on pathways to learning goals. An accurate and nuanced way of interpreting assessment results will allow subsequent instructional actions to be targeted. An appropriate psychometric model is indispensable in this regard. In this study, we developed a new psychometric model, namely, the diagnostic facet status model (DFSM), which belongs to the general class of cognitive diagnostic models (CDM), but with two notable features: (1) it simultaneously models students' target understanding (i.e., goal facet) and intermediate understanding (i.e., intermediate facet); and (2) it models every response option, rather than merely right or wrong responses, so that each incorrect response uniquely contributes to discovering students' facet status. Given that some combination of goal and intermediate facets may be impossible due to facet hierarchical relationships, a regularized expectation-maximization algorithm (REM) was developed for model estimation. A log-penalty was imposed on the mixing proportions to encourage sparsity. As a result, those impermissible latent classes had estimated mixing proportions equal to 0. A heuristic algorithm was proposed to infer a facet map from the estimated permissible classes. A simulation study was conducted to evaluate the performance of REM to recover facet model parameters and to identify permissible latent classes. A real data analysis was provided to show the feasibility of the model.

Performance of ChatGPT on the Taiwan urology board examination: insights into current strengths and shortcomings.

To compare ChatGPT-4 and ChatGPT-3.5's performance on Taiwan urology board examination (TUBE), focusing on answer accuracy, explanation consistency, and uncertainty management tactics to minimize score penalties from incorrect responses across 12 urology domains. 450 multiple-choice questions from TUBE(2020-2022) were presented to two models. Three urologists assessed correctness and consistency of each response. Accuracy quantifies correct answers; consistency assesses logic and coherence in explanations out of total responses, alongside a penalty reduction experiment with prompt variations. Univariate logistic regression was applied for subgroup comparison. ChatGPT-4 showed strengths in urology, achieved an overall accuracy of 57.8%, with annual accuracies of 64.7% (2020), 58.0% (2021), and 50.7% (2022), significantly surpassing ChatGPT-3.5 (33.8%, OR = 2.68, 95% CI [2.05-3.52]). It could have passed the TUBE written exams if solely based on accuracy but failed in the final score due to penalties. ChatGPT-4 displayed a declining accuracy trend over time. Variability in accuracy across 12 urological domains was noted, with more frequently updated knowledge domains showing lower accuracy (53.2% vs. 62.2%, OR = 0.69, p = 0.05). A high consistency rate of 91.6% in explanations across all domains indicates reliable delivery of coherent and logical information. The simple prompt outperformed strategy-based prompts in accuracy (60% vs. 40%, p = 0.016), highlighting ChatGPT's limitations in its inability to accurately self-assess uncertainty and a tendency towards overconfidence, which may hinder medical decision-making. ChatGPT-4's high accuracy and consistent explanations in urology board examination demonstrate its potential in medical information processing. However, its limitations in self-assessment and overconfidence necessitate caution in its application, especially for inexperienced users. These insights call for ongoing advancements of urology-specific AI tools.

ChatGPT4's proficiency in addressing patients' questions on systemic lupus erythematosus: a blinded comparative study with specialists.

The efficacy of artificial intelligence (AI)-driven chatbots like ChatGPT4 in specialized medical consultations, particularly in rheumatology, remains underexplored. This study compares the proficiency of ChatGPT4' responses with practicing rheumatologists to inquiries from patients with SLE. In this cross-sectional study, we curated 95 frequently asked questions (FAQs), including 55 in Chinese and 40 in English. Responses for FAQs from ChatGPT4 and five rheumatologists were scored separately by a panel of rheumatologists and a group of patients with SLE across six domains (scientific validity, logical consistency, comprehensibility, completeness, satisfaction level and empathy) on a 0-10 scale (a score of 0 indicates entirely incorrect responses, while 10 indicates accurate and comprehensive answers). Rheumatologists' scoring revealed that ChatGPT4-generated responses outperformed those from rheumatologists in satisfaction level and empathy, with mean differences of 0.537 (95% CI, 0.252-0.823; P < 0.01) and 0.460 (95% CI, 0.227-0.693; P < 0.01), respectively. From the SLE patients' perspective, ChatGPT4-generated responses were comparable to the rheumatologist-provided answers in all six domains. Subgroup analysis revealed ChatGPT4 responses were more logically consistent and complete regardless of language and exhibited greater comprehensibility, satisfaction and empathy in Chinese. However, ChatGPT4 responses were inferior in comprehensibility for English FAQs. ChatGPT4 demonstrated comparable, possibly better in certain domains, to address FAQs from patients with SLE, when compared with the answers provided by specialists. This study showed the potential of applying ChatGPT4 to improve consultation in SLE patients.

Crowdsourced Assessment of Aesthetic Outcomes of Dorsal Preservation Rhinoplasty

Abstract Goals/Purpose The inherent subjectivity in evaluating aesthetic outcomes presents a unique challenge in assessing rhinoplasty. Crowdsourcing has provided a new metric for objective analysis. The authors designed a study to compare the aesthetic outcomes of dorsal preservation rhinoplasty versus structural rhinoplasty using a reliable and valid crowdsourcing platform. We aim to objectively quantify the relative aesthetic advantages of performing a dorsal preservation technique. Additionally, we aim to demonstrate the efficacy of using crowdsourcing as an efficient and reliable method for evaluating any plastic surgery aesthetic outcome. Methods/Technique This retrospective observational study was approved by the IRB at UCLA. A total of 64 patients who had previously undergone rhinoplasty performed by the senior author were included. All surgeries were performed at the UCLA Ronald Reagan Medical Center. Patients with prior nasal surgery or trauma were excluded. All subjects were photographed using standard rhinoplasty views during the pre-and postoperative visits. Frontal and right profile photographs were then cropped in a standardized fashion (Photos, Version 8.0, Apple Inc.) and used for evaluation. All evaluations and ratings were collected using the HIPPA compliant and encrypted software via a secure online platform (LoveMyDelta, Inc., Philadelphia, PA). All crowd workers were recruited and vetted through this platform, with internal fidelity checks performed. Each crowd worker was shown a profile and frontal view of a patient and asked to score the overall appearance of the nose, the nasal dorsum profile, the symmetry of the dorsal aesthetic lines, and the dorsal contour using a visual analog scale. Scoring was performed using a provided validated visual assessment guide. They were asked whether they believed the subject had undergone rhinoplasty and asked to provide a level of confidence in their guess. Crowdworkers’ responses were aggregated to allow for high-powered intra-rater analysis. Intra-rater reliability and confidence intervals were calculated. Aggregate data from all raters were used to generate an absolute value of aesthetic state for each cohort. A "delta" was then obtained for each value by comparing the preoperative to the postoperative state to obtain a representative value of the improvement after undergoing surgery. Raters were asked if the patient appeared to have had surgery, and each correct and incorrect response was proportionally weighted based on the raters' confidence in their answer. The delta (difference in preoperative to postoperative aesthetic state) for each parameter for the dorsal and non-dorsal preservation cohorts was calculated using non-paired T-tests to determine significance. Results/Complications A total of 64 patients were included for evaluation. The structural rhinoplasty cohort consisted of 34 patients. The dorsal preservation cohort included 30 patients. Both dorsal preservation and non-dorsal preservation rhinoplasty cohorts were associated with improved overall crowdsourced aesthetic outcomes and improved outcomes across all subparameters. Overall, delta (d) improved in both dorsal preservation (0.300, 95% CI +/- 0.047) and structural (0.377, 95% CI +/- 0.055) cohorts. When raters were asked to predict whether a patient had surgery, the correlation coefficient between the confidence in the prediction and that prediction being correct was 0.74 (95% CI +/- 0.47 in the structural group, suggesting that a crowd worker was likely to be correct in identifying whether a patient had structural rhinoplasty. However, the correlation coefficient was -0.0554 (95% CI +/- 0.058) in the dorsal preservation group. This suggests the raters were unable to consistently identify which patients had dorsal preservation rhinoplasty with any confidence. Conclusion In the ongoing challenge to achieve better aesthetic outcomes in plastic surgery, crowdsourcing is a valuable tool for helping to measure these outcomes. We compared the aesthetic outcomes of dorsal preservation rhinoplasty versus structural rhinoplasty using a reliable and valid crowdsourcing platform. We found that significant improvements in overall aesthetic outcomes were achieved with both techniques, while a more natural “unoperated” outcome was achieved when performing a dorsal preservation technique. With this type of study and assessment tool, we helped to demonstrate the efficacy of using crowdsourcing as an efficient and reliable method for evaluating any plastic surgery aesthetic outcome.

Poor performance of ChatGPT in clinical rule-guided dose interventions in hospitalized patients with renal dysfunction.

Clinical decision support systems (CDSS) are used to identify drugs with potential need for dose modification in patients with renal impairment. ChatGPT holds the potential to be integrated in the electronic health record (EHR) system to give such dosing advices. In this study, we aim to evaluate the performance of ChatGPT in clinical rule-guided dose interventions in hospitalized patients with renal impairment. This cross-sectional study was performed at Tergooi Medical Center, the Netherlands. CDSS alerts regarding renal dysfunction were collected from the electronic health record (EHR) during a 2-week period and were presented to ChatGPT and an expert panel. Alerts were presented with and without patient variables. To evaluate the performance, suggested medication interventions were compared. In total, 172 CDDS alerts were generated for 80 patients. Indecisive responses by ChatGPT to alerts were excluded. For alerts presented without patient variables, ChatGPT provided "correct and identical" responses to 19.9%, "correct and different" responses to 26.7%, and "incorrect responses to 53.4% of the alerts. For alerts including patient variables, ChatGPT provided "correct and identical" responses to 16.7%, "correct and different" responses to 16.0%, and "incorrect responses to 67.3% of the alerts. Accuracy was better for newer drugs such as direct oral anticoagulants. The performance of ChatGPT in clinical rule-guided dose interventions in hospitalized patients with renal dysfunction was poor. Based on these results, we conclude that ChatGPT, in its current state, is not appropriate for automatic integration into our EHR to handle CDSS alerts related to renal dysfunction.

The Minority Confederate Induced Errors of the Majority Members in the Role-Inverted Asch Experiment

This study aimed to investigate whether errors made by minority group members could influence the behavior of the majority in a role-reversed version of Asch's conformity experiment. Eighty-five undergraduate students, comprising 51 males and 34 females, were organized into groups of four individuals who were already acquainted with each other. From these groups, we randomly selected twelve, and discreetly instructed the third student in each group (a confederate) to provide incorrect responses on six out of the nine line-judgment tasks, similar to Asch's majority's behavior. The findings revealed that groups with the confederate member did lead to an increase in errors from other group members, although this increase did not reach statistical significance. Nevertheless, further analyses indicated that participants were more likely to make errors when other group members had already responded incorrectly. Consequently, we interpreted these errors as a form of social conformity, where participants intentionally made mistakes to alleviate the awkwardness of having only one member of the group providing incorrect answers. In summary, this study provides insights into the dynamics of social conformity and the impact of minority errors on group behavior in a role-reversed Asch experiment.