Response Accuracy Research Articles

Assessment of Knowledge About High Blood Pressure Among Syrians

Hypertension, often referred to the silent killer, is a prevalent health issue that impacts more than a billion individuals worldwide. The purpose of this study was to investigate the understanding of hypertension among Syrian individuals. 104 individuals filled out surveys. Females made up 64.4% of the group that was surveyed. 28.8% of the participants in the study were between the ages of 46 and 60. More than half of the participants had a study or a work related to the medicinal field. The participants showed good medication adherence where the score of accurate responses to the questions regarding medication compliance was very high. Most participants were familiar with the normal values of blood pressure levels and were optimistic about the possibility of treating the condition. A good percentage of individuals surveyed stated that smoking and obesity are contributing factors to developing high blood pressure. Most participants believed that consuming fatty foods raises plasma cholesterol levels. Participants had a good overall understanding of high blood pressure. However, they still need improvement in medical information about high blood pressure to enhance compliance with treatments.

Effects of posttransfer feedback informativeness in a transitive inference task.

Transitive inference (TI), referring to one's ability to learn that if A > B and B > C, one can infer that A > C, is a form of serial learning that has been tested using a variety of experimental protocols. An element of most of these protocols is the presentation of some form of visual corrective feedback to help inform naïve participants about the nature of the task. Therefore, corrective feedback is often used as a critical tool for experimental TI. To further explore this tool, we presented varying forms/presence of visual feedback during TI performance: corrective, none, and ambiguous. Of the three conditions, presentation of the ambiguous feedback yielded the clearest disruption of TI performance, obvious in measurements of both response accuracy and reaction times. Participants appears to remain engaged with feedback was withheld entirely, but to disengage when presented with ambiguous feedback. Therefore, the choice regarding the form of feedback provided during testing may be critical for information processing of inferences.

Comparing the Accuracy and Readability of Glaucoma-related Question Responses and Educational Materials by Google and ChatGPT

Comparing the Accuracy and Readability of Glaucoma-related Question Responses and Educational Materials by Google and ChatGPT

Large language models in patient education: a scoping review of applications in medicine.

Large Language Models (LLMs) are sophisticated algorithms that analyze and generate vast amounts of textual data, mimicking human communication. Notable LLMs include GPT-4o by Open AI, Claude 3.5 Sonnet by Anthropic, and Gemini by Google. This scoping review aims to synthesize the current applications and potential uses of LLMs in patient education and engagement. Following the PRISMA-ScR checklist and methodologies by Arksey, O'Malley, and Levac, we conducted a scoping review. We searched PubMed in June 2024, using keywords and MeSH terms related to LLMs and patient education. Two authors conducted the initial screening, and discrepancies were resolved by consensus. We employed thematic analysis to address our primary research question. The review identified 201 studies, predominantly from the United States (58.2%). Six themes emerged: generating patient education materials, interpreting medical information, providing lifestyle recommendations, supporting customized medication use, offering perioperative care instructions, and optimizing doctor-patient interaction. LLMs were found to provide accurate responses to patient queries, enhance existing educational materials, and translate medical information into patient-friendly language. However, challenges such as readability, accuracy, and potential biases were noted. LLMs demonstrate significant potential in patient education and engagement by creating accessible educational materials, interpreting complex medical information, and enhancing communication between patients and healthcare providers. Nonetheless, issues related to the accuracy and readability of LLM-generated content, as well as ethical concerns, require further research and development. Future studies should focus on improving LLMs and ensuring content reliability while addressing ethical considerations.

Adults report positive perceptions of ground rule instructions in mock investigative interviews

Ground rules establish the conversational expectations during an interview and are intended to reduce interviewee discomfort and increase response accuracy towards problematic interview questions. This study explored perceptions of ground rule instructions in adult interviews. Younger (18–40 years) and older (60+ years) adults (N = 168) were interviewed about a film depicting an implied sexual assault or a personal event. Participants received either three ground rules as statements (‘Don’t Know’, ‘Don’t Understand’ and ‘Correct Me’) or the three statements with practice questions. Participants were interviewed about their perceptions of ground rules. Overall, ground rules were perceived as useful (85%) and as having a positive impact on interviewees’ emotional state (55%) and perceptions of the interviewer (35%). Perceptions did not vary based on contextual factors, but female and younger adults endorsed ground rules as useful more often than other groups. The results suggest that ground rules may improve adults’ experience of being interviewed.

Response accuracy of GPT-4 across languages: insights from an expert-level diagnostic radiology examination in Japan.

This study aims to investigate the effects of language selection and translation quality on Generative Pre-trained Transformer-4 (GPT-4)'s response accuracy to expert-level diagnostic radiology questions. We analyzed 146 diagnostic radiology questions from the Japan Radiology Board Examination (2020-2022), with consensus answers provided by two board-certified radiologists. The questions, originally in Japanese, were translated into English by GPT-4 and DeepL and into German and Chinese by GPT-4. Responses were generated by GPT-4 five times per question set per language. Response accuracy was compared between languages using one-way ANOVA with Bonferroni correction or the Mann-Whitney U test. Scores on selected English questions translated by a professional service and GPT-4 were also compared. The impact of translation quality on GPT-4's performance was assessed by linear regression analysis. The median scores (interquartile range) for the 146 questions were 70 (68-72) (Japanese), 89 (84.5-95.5) (GPT-4 English), 64 (55.5-67) (Chinese), and 56 (46.5-67.5) (German). Significant differences were found between Japanese and English (p = 0.002) and between Japanese and German (p = 0.022). The counts of correct responses across five attempts for each question were significantly associated with the quality of translation into English (GPT-4, DeepL) and German (GPT-4). In a subset of 31 questions where English translations yielded fewer correct responses than Japanese originals, professionally translated questions yielded better scores than those translated by GPT-4 (13 versus 8 points, p = 0.0079). GPT-4 exhibits higher accuracy when responding to English-translated questions compared to original Japanese questions, a trend not observed with German or Chinese translations. Accuracy improves with higher-quality English translations, underscoring the importance of high-quality translations in improving GPT-4's response accuracy to diagnostic radiology questions in non-English languages and aiding non-native English speakers in obtaining accurate answers from large language models.

Asking to thrive: structures supporting young men’s behavioural health in Belfast, Northern Ireland

Abstract Background Men in Northern Ireland are three times more likely to die by suicide and psychiatric morbidity is significantly higher than for the United Kingdom overall, in part due to the Troubles, a 30-year conflict during which there were 34,000 shootings, 14,000 bombings, and 3,500 deaths. Prior studies suggest strong friendships and high collective efficacy in communities are protective against suicide and self-harm in adolescents and that inquiring about supportive people and structures directly-rather than through parental intermediaries-yields more accurate responses. Methods Semi-structured interviews were conducted with 30 adolescent males aged 16-19 from working-class communities across Belfast. Participants were recruited via trusted community partners and often had experiences with sectarianism and restorative justice. A deductive thematic analysis was conducted of interview transcripts, supplemented by responses to the Self-Harm Inventory and a modified Collective Efficacy Scale. Analyses were completed in NVivo and RStudio. Results In addition to male peer friendships in which they could share confidences, participants strongly endorsed the value of youth work organisations as keystones in supporting not only their own behavioural health through positive modelling but also the reduction of antisocial and sectarian behaviours in their communities. Participants named specific aspects of these structures they found most useful, including demonstrating “masculine success” though community engagement and family life, providing a safe venue for socialising, and delivering restorative justice programming. Conclusions Findings from this study suggest that youth work programming is an important component of both adolescent behavioural health promotion and community safety through the reduction of sectarian and other antisocial behaviours. Participants expressed dismay at the reduction of these services and endorsed both their personal and wider community importance. Key messages • Youth workers may provide valuable positive modelling and behavioural health support to the young men. • Youth work programmes may function both for behavioural health promotion and community violence reduction in Belfast, Northern Ireland.

Coupling Residual Stress Field to Enhance the Simulation Accuracy of Crack Propagation and Mechanical Responses of High Entropy Ceramics Reinforced by Graphene and Alumina

Coupling Residual Stress Field to Enhance the Simulation Accuracy of Crack Propagation and Mechanical Responses of High Entropy Ceramics Reinforced by Graphene and Alumina

Algorithmic Bias in BERT for Response Accuracy Prediction: A Case Study for Investigating Population Validity

AbstractPretrained large language models (LLMs) have gained popularity in recent years due to their high performance in various educational tasks such as learner modeling, automated scoring, automatic item generation, and prediction. Nevertheless, LLMs are black box approaches where models are less interpretable, and they may carry human biases and prejudices because historical human data have been used for pretraining these large‐scale models. For these reasons, the prediction tasks based on LLMs require scrutiny to ensure that the prediction models are fair and unbiased. In this study, we used BERT—a pretrained encoder‐only LLM for predicting response accuracy using action sequences extracted from the 2012 PIAAC assessment. We selected three countries (i.e., Finland, Slovakia, and the United States) representing different performance levels in the overall PIAAC assessment. We found promising results for predicting response accuracy using the fine‐tuned BERT model. Additionally, we examined algorithmic bias in the prediction models trained with different countries. We found differences in model performance, suggesting that some trained models are not free from bias, and thus the models are less generalizable across countries. Our results highlighted the importance of investigating algorithmic fairness in prediction models utilizing algorithmic systems to ensure models are bias‐free.

Data-assisted non-intrusive model reduction for forced nonlinear finite elements models

AbstractSpectral submanifolds (SSMs) have emerged as accurate and predictive model reduction tools for dynamical systems defined either by equations or data sets. While finite-elements (FE) models belong to the equation-based class of problems, their implementations in commercial solvers do not generally provide information on the nonlinearities required for the analytical construction of SSMs. Here, we overcome this limitation by developing a data-driven construction of SSM-reduced models from a small number of unforced FE simulations. We then use these models to predict the forced response of the FE model without performing any costly forced simulation. This approach yields accurate forced response predictions even in the presence of internal resonances or quasi-periodic forcing, as we illustrate on several FE models. Our examples range from simple structures, such as beams and shells, to more complex geometries, such as a micro-resonator model containing more than a million degrees of freedom. In the latter case, our algorithm predicts accurate forced response curves in a small fraction of the time it takes to verify just a few points on those curves by simulating the full forced-response.

Non-target screening analysis of hazardous noxious substances using gas chromatography-quadrupole time-of-flight mass spectrometry

The forensic investigation of hazardous noxious substances (HNS) is paramount for an effective response to chemical spill emergencies and other accidents. Analyzing unknown emergency samples poses a challenge due to the limited availability of background information, making the selection of appropriate sample preparation and analytical methodologies difficult. The utilization of high-resolution mass spectrometers (HRMS) in screening both target and non-target substances proves instrumental in revealing hazardous substances that may be overlooked alongside the intended analytes. In this study, a gas chromatography-quadruple time-of-flight mass spectrometer (GC-QTOF-MS) was employed to identify numerous organic compounds in an indoor dust sample. The compounds detected encompassed normal alkanes, fatty acids (saturated and unsaturated), alcohols, phenols, sterols, drugs, polycyclic aromatic hydrocarbons (PAHs), pesticides, flame retardants (such as polybrominated biphenyl ethers, PBDEs), plasticizers (such as phthalates and phosphates), among others. Notably, concentrations of n-alkanes, fatty acids, and phthalates were relatively high, while PAHs and pesticides were present at trace levels. The application of GC-QTOF-MS provides a swift and confirmative approach for analyzing target, suspect, and non-target compounds in both routine and emergency scenarios. This methodology proves invaluable in enhancing our capability to comprehensively assess and address chemical incidents, ensuring a more thorough and accurate response.

Performance Assessment of GPT 4.0 on the Japanese Medical Licensing Examination.

To evaluate the accuracy and parsing ability of GPT 4.0 for Japanese medical practitioner qualification examinations in a multidimensional way to investigate its response accuracy and comprehensiveness to medical knowledge. We evaluated the performance of the GPT 4.0 on Japanese Medical Licensing Examination (JMLE) questions (2021-2023). Questions are categorized by difficulty and type, with distinctions between general and clinical parts, as well as between single-choice (MCQ1) and multiple-choice (MCQ2) questions. Difficulty levels were determined on the basis of correct rates provided by the JMLE Preparatory School. The accuracy and quality of the GPT 4.0 responses were analyzed via an improved Global Qualily Scale (GQS) scores, considering both the chosen options and the accompanying analysis. Descriptive statistics and Pearson Chi-square tests were used to examine performance across exam years, question difficulty, type, and choice. GPT 4.0 ability was evaluated via the GQS, with comparisons made via the Mann-Whitney U or Kruskal-Wallis test. The correct response rate and parsing ability of the GPT4.0 to the JMLE questions reached the qualification level (80.4%). In terms of the accuracy of the GPT4.0 response to the JMLE, we found significant differences in accuracy across both difficulty levels and option types. According to the GQS scores for the GPT 4.0 responses to all the JMLE questions, the performance of the questionnaire varied according to year and choice type. GTP4.0 performs well in providing basic support in medical education and medical research, but it also needs to input a large amount of medical-related data to train its model and improve the accuracy of its medical knowledge output. Further integration of ChatGPT with the medical field could open new opportunities for medicine.

Accuracy and Consistency of Gemini Responses Regarding the Management of Traumatized Permanent Teeth.

The aim of this cross-sectional observational analytical study was to assess the accuracy and consistency of responses provided by Google Gemini (GG), a free-access high-performance multimodal large language model, to questions related to the European Society of Endodontology position statement on the management of traumatized permanent teeth (MTPT). Three academic endodontists developed a set of 99 yes/no questions covering all areas of the MTPT. Nine general dentists and 22 endodontic specialists evaluated these questions for clarity and comprehension through an iterative process. Two academic dental trauma experts categorized the knowledge required to answer each question into three levels. The three academic endodontists submitted the 99 questions to the GG, resulting in 297 responses, which were then assessed for accuracy and consistency. Accuracy was evaluated using the Wald binomial method, while the consistency of GG responses was assessed using the kappa-Fleiss coefficient with a confidence interval of 95%. A 5% significance level chi-squared test was used to evaluate the influence of question level of knowledge on accuracy and consistency. The responses generated by Gemini showed an overall moderate accuracy of 80.81%, with no significant differences found between the responses of the academic endodontists. Overall, high consistency (95.96%) was demonstrated, with no significant differences between GG responses across the three accounts. The analysis also revealed no correlation between question level of knowledge and accuracy or consistency, with no significant differences. The results of this study could significantly impact the potential use of Gemini as a free-access source of information for clinicians in the MTPT.

CIPHER: Cybersecurity Intelligent Penetration-Testing Helper for Ethical Researcher.

Penetration testing, a critical component of cybersecurity, typically requires extensive time and effort to find vulnerabilities. Beginners in this field often benefit from collaborative approaches with the community or experts. To address this, we develop Cybersecurity Intelligent Penetration-testing Helper for Ethical Researchers (CIPHER), a large language model specifically trained to assist in penetration testing tasks as a chatbot. Unlike software development, penetration testing involves domain-specific knowledge that is not widely documented or easily accessible, necessitating a specialized training approach for AI language models. CIPHER was trained using over 300 high-quality write-ups of vulnerable machines, hacking techniques, and documentation of open-source penetration testing tools augmented in an expert response structure. Additionally, we introduced the Findings, Action, Reasoning, and Results (FARR) Flow augmentation, a novel method to augment penetration testing write-ups to establish a fully automated pentesting simulation benchmark tailored for large language models. This approach fills a significant gap in traditional cybersecurity Q&A benchmarks and provides a realistic and rigorous standard for evaluating LLM's technical knowledge, reasoning capabilities, and practical utility in dynamic penetration testing scenarios. In our assessments, CIPHER achieved the best overall performance in providing accurate suggestion responses compared to other open-source penetration testing models of similar size and even larger state-of-the-art models like Llama 3 70B and Qwen1.5 72B Chat, particularly on insane difficulty machine setups. This demonstrates that the current capabilities of general large language models (LLMs) are insufficient for effectively guiding users through the penetration testing process. We also discuss the potential for improvement through scaling and the development of better benchmarks using FARR Flow augmentation results.

A comparison of drug information question responses by a drug information center and by ChatGPT.

In an effort to expedite the publication of articles, AJHP is posting manuscripts online as soon as possible after acceptance. Accepted manuscripts have been peer-reviewed and copyedited, but are posted online before technical formatting and author proofing. These manuscripts are not the final version of record and will be replaced with the final article (formatted per AJHP style and proofed by the authors) at a later time. A study was conducted to assess the accuracy and ability of Chat Generative Pre-trained Transformer (ChatGPT) to systematically respond to drug information inquiries relative to responses of a drug information center (DIC). Ten drug information questions answered by the DIC in 2022 or 2023 were selected for analysis. Three pharmacists created new ChatGPT accounts and submitted each question to ChatGPT at the same time. Each question was submitted twice to identify consistency in responses. Two days later, the same process was conducted by a fourth pharmacist. Phase 1 of data analysis consisted of a drug information pharmacist assessing all 84 ChatGPT responses for accuracy relative to the DIC responses. In phase 2, 10 ChatGPT responses were selected to be assessed by 3 blinded reviewers. Reviewers utilized an 8-question predetermined rubric to evaluate the ChatGPT and DIC responses. When comparing the ChatGPT responses (n = 84) to the DIC responses, ChatGPT had an overall accuracy rate of 50%. Accuracy across the different question types varied. In regards to the overall blinded score, ChatGPT responses scored higher than the responses by the DIC according to the rubric (overall scores of 67.5% and 55.0%, respectively). The DIC responses scored higher in the categories of references mentioned and references identified. Responses generated by ChatGPT have been found to be better than those created by a DIC in clarity and readability; however, the accuracy of ChatGPT responses was lacking. ChatGPT responses to drug information questions would need to be carefully reviewed for accuracy and completeness.

Assessing Performance of Cloud-based Heterogeneous Chatbot Systems.

Cloud-based heterogeneous chatbot systems represent a significant advancement in automating diverse tasks across various platforms, including customer service, healthcare, and education. These systems are designed with multi-modal capabilities, allowing them to interact via text, voice, and visual interfaces, and are built on a cloud infrastructure that ensures scalability and reliability. The cloud-based architecture allows dynamic resource allocation, enabling the system to handle high volumes of traffic without compromising performance. These chatbots also benefit from global availability, as cloud regions distributed across the world ensure low latency and high availability.The integration of advanced AI and NLP models, such as GPT and BERT, enhances the chatbot's response accuracy and contextual understanding, making interactions more precise and human-like. Moreover, the cloud infrastructure supports continuous learning through machine learning models that refine chatbot performance over time. With multitenant capabilities, these chatbots can cater to different users or organizations, providing tailored services while ensuring that performance are not affected. Additionally, their ability to integrate with external enterprise systems, like CRMs or ERPs, allows for real-time data exchange and task automation.In terms of security, cloud-based chatbots benefit from encryption protocols and compliance with industry regulations, ensuring safe handling of sensitive data, particularly in sectors like healthcare and finance. Despite the advantages, challenges such as network dependency and cost management need careful consideration, as performance can be affected by poor connectivity or high processing costs, especially when using complex AI models. Nonetheless, cloud-based heterogeneous chatbots excel in delivering scalable, reliable, and intelligent interactions, adapting to growing user demands and a wide range of applications.

Anxious/depressed individuals exhibit disrupted frontotemporal synchrony during cognitive conflict encoding.

Anxiety and depressive disorders are associated with cognitive control deficits, yet their underlying neural mechanisms remain poorly understood. Here, we used high-resolution stereotactic EEG (sEEG) to determine how anxiety and/or depression modulates neural and behavioral responses when cognitive control is engaged in individuals with medically refractory epilepsy undergoing sEEG monitoring for surgical evaluation. We analyzed sEEG data recorded from frontotemporal regions of 29 participants (age range: 19-55, mean age: 35.5, female: 16/29) while they performed a Multi-Source Interference Task (MSIT) designed to elicit cognitive conflict. Neurobehavioral interviews, symptom rating scales, and clinical documentation were used to categorize participants as demonstrating anxiety and/or depression symptoms (A/D, n=13) or as epilepsy controls (EC, n=16). Generalized linear mixed-effects (GLME) models were used to analyze behavioral and neural data. Models of oscillatory power were used to identify brain regions within conflict-encoding networks in which coherence and phase locking values (PLV) were examined in A/D and EC. A/D participants demonstrated a greater conflict effect (response time slowing with higher cognitive load), without impairment in response time (RT) or accuracy compared to EC. A/D participants also showed significantly enhanced conflict-evoked theta (4-8Hz) and alpha (8-15Hz) power in the dorsolateral prefrontal cortex (dlPFC) and amygdala as well as widespread broadband activity in the lateral temporal lobe (LTL) compared to EC. Additionally, theta coherence and PLV between dlPFC-LTL and dlPFC-amygdala were reduced by conflict in A/D. Our findings suggest individuals with anxiety/depression symptoms exhibit heightened frontotemporal oscillatory activity and disrupted frontotemporal synchrony during cognitive conflict encoding, which may indicate a greater need for cognitive resources due to ineffective cognitive processing. These results highlight a potential role of frontotemporal circuits in conflict encoding that are altered in anxiety/depression, and may further inform future therapeutic interventions aimed at enhancing cognitive control in these populations.

Psychometrics of drift-diffusion model parameters derived from the Eriksen flanker task: Reliability and validity in two independent samples.

The flanker task is a widely used measure of cognitive control abilities. Drift-diffusion modeling of flanker task behavior can yield separable parameters of cognitive control-related subprocesses, but the parameters' psychometrics are not well-established. We examined the reliability and validity of four behavioral measures: (1) raw accuracy, (2) reaction time (RT) interference, (3) NIH Toolbox flanker score, and (4) two drift-diffusion model (DDM) parameters-drift rate and boundary separation-capturing evidence accumulation efficiency and speed-accuracy trade-off, respectively. Participants from two independent studies - one cross-sectional (N = 381) and one with three timepoints (N = 83) - completed the flanker task while electroencephalography data were collected. Across both studies, drift rate and boundary separation demonstrated comparable split-half and test-retest reliability to accuracy, RT interference, and NIH Toolbox flanker score, but better incremental convergent validity with psychophysiological measures (i.e., the error-related negativity; ERN) and neuropsychological measures of cognitive control than the other behavioral indices. Greater drift rate (i.e., faster and more accurate responses) to congruent and incongruent stimuli, and smaller boundary separation to incongruent stimuli were related to 1) larger ERN amplitudes (in both studies) and 2) faster and more accurate inhibition and set-shifting over and above raw accuracy, reaction time, and NIH Toolbox flanker scores (in Study 1). Computational models, such as DDM, can parse behavioral performance into subprocesses that exhibit comparable reliability to other scoring approaches, but more meaningful relationships with other measures of cognitive control. The application of these computational models may be applied to existing data and enhance the identification of cognitive control deficits in psychiatric disorders.

Performance of Multimodal Artificial Intelligence Chatbots Evaluated on Clinical Oncology Cases

Multimodal artificial intelligence (AI) chatbots can process complex medical image and text-based information that may improve their accuracy as a clinical diagnostic and management tool compared with unimodal, text-only AI chatbots. However, the difference in medical accuracy of multimodal and text-only chatbots in addressing questions about clinical oncology cases remains to be tested. To evaluate the utility of prompt engineering (zero-shot chain-of-thought) and compare the competency of multimodal and unimodal AI chatbots to generate medically accurate responses to questions about clinical oncology cases. This cross-sectional study benchmarked the medical accuracy of multiple-choice and free-text responses generated by AI chatbots in response to 79 questions about clinical oncology cases with images. A unique set of 79 clinical oncology cases from JAMA Network Learning accessed on April 2, 2024, was posed to 10 AI chatbots. The primary outcome was medical accuracy evaluated by the number of correct responses by each AI chatbot. Multiple-choice responses were marked as correct based on the ground-truth, correct answer. Free-text responses were rated by a team of oncology specialists in duplicate and marked as correct based on consensus or resolved by a review of a third oncology specialist. This study evaluated 10 chatbots, including 3 multimodal and 7 unimodal chatbots. On the multiple-choice evaluation, the top-performing chatbot was chatbot 10 (57 of 79 [72.15%]), followed by the multimodal chatbot 2 (56 of 79 [70.89%]) and chatbot 5 (54 of 79 [68.35%]). On the free-text evaluation, the top-performing chatbots were chatbot 5, chatbot 7, and the multimodal chatbot 2 (30 of 79 [37.97%]), followed by chatbot 10 (29 of 79 [36.71%]) and chatbot 8 and the multimodal chatbot 3 (25 of 79 [31.65%]). The accuracy of multimodal chatbots decreased when tested on cases with multiple images compared with questions with single images. Nine out of 10 chatbots, including all 3 multimodal chatbots, demonstrated decreased accuracy of their free-text responses compared with multiple-choice responses to questions about cancer cases. In this cross-sectional study of chatbot accuracy tested on clinical oncology cases, multimodal chatbots were not consistently more accurate than unimodal chatbots. These results suggest that further research is required to optimize multimodal chatbots to make more use of information from images to improve oncology-specific medical accuracy and reliability.

Accuracy of natural language processors for patients seeking inguinal hernia information.

NLPs such as ChatGPT are novel sources of online healthcare information that are readily accessible and integrated into internet search tools. The accuracy of NLP-generated responses to health information questions is unknown. We queried four NLPs (ChatGPT 3.5 and 4, Bard, and Claude 2.0) for responses to simulated patient questions about inguinal hernias and their management. Responses were graded on a Likert scale (1 poor to 5 excellent) for relevance, completeness, and accuracy. Responses were compiled and scored collectively for readability using the Flesch-Kincaid score and for educational quality using the DISCERN instrument, a validated tool for evaluating patient information materials. Responses were also compared to two gold-standard educational materials provided by SAGES and the ACS. Evaluations were performed by six hernia surgeons. The average NLP response scores for relevance, completeness, and accuracy were 4.76 (95% CI 4.70-4.80), 4.11 (95% CI 4.02-4.20), and 4.14 (95% CI 4.03-4.24), respectively. ChatGPT4 received higher accuracy scores (mean 4.43 [95% CI 4.37-4.50]) than Bard (mean 4.06 [95% CI 3.88-4.26]) and Claude 2.0 (mean 3.85 [95% CI 3.63-4.08]). The ACS document received the best scores for reading ease (55.2) and grade level (9.2); however, none of the documents achieved the readibility thresholds recommended by the American Medical Association. The ACS document also received the highest DISCERN score of 63.5 (57.0-70.1), and this was significantly higher compared to ChatGPT 4 (50.8 [95% CI 46.2-55.4]) and Claude 2.0 (48 [95% CI 41.6-54.4]). The evaluated NLPs provided relevant responses of reasonable accuracy to questions about inguinal hernia. Compiled NLP responses received relatively low readability and DISCERN scores, although results may improve as NLPs evolve or with adjustments in question wording. As surgical patients expand their use of NLPs for healthcare information, surgeons should be aware of the benefits and limitations of NLPs as patient education tools.