Open Software Development Research Articles

Enhancing industrial IoT with edge computing and computer vision: An analog gauge visual digitization approach

The advent of Industry 4.0 has led to the widespread adoption of Industrial Internet of Things (IIoT) and Computer Vision technologies in manufacturing. However, handling the massive data generated by IIoT poses challenges. To address this, we propose a software and hardware architecture that combines edge computing with IIoT to enable computer vision applications on manufacturing floors. This research contributes to integrating edge computing and IIoT with Computer Vision technologies in manufacturing facilities, offering a cost-effective and scalable solution for machine vision applications. This approach benefits Small and Medium Manufacturers (SMMs) by utilizing low-cost hardware and open-source technologies, enabling them to harness Industry 4.0 advantages. Our approach involves IoT devices using Raspberry Pi microcomputers and USB webcams as clients and a Python-based edge node server software as the central component. Our test case study results show that a centralized edge-computing solution can offer several useability benefits and a significant performance increase over using a non-edge-enabled IIoT approach. The proposed architecture is tested, validated, and verified through a case study focused on the digitization of the readings of analog gauges. Our experimentation results can offer valuable insights and guidance for the broader adoption of edge-enabled IIoT solutions in manufacturing contexts.

Predicting solvation free energies for neutral molecules in any solvent with openCOSMO-RS

In this study, we introduce openCOSMO-RS 24a, an improved version of the open-source COSMO-RS model parameterized using quantum chemical calculations from ORCA 6.0, leveraging a comprehensive dataset that includes solvation free energies, partition coefficients, and infinite dilution activity coefficients for various solutes and solvents mainly at 25 °C. This is the first version of the model also capable of predicting solvation free energies based on ORCA calculations. Additionally, we develop a Quantitative Structure-Property Relationships model to predict molar volumes of the solvents, an essential requirement for predicting solvation free energies and partition coefficients from structure alone. Our results show that openCOSMO-RS 24a achieves an average absolute deviation of 0.45 kcal mol1 for solvation free energies, 0.76 for the logarithm of the partition coefficients, and 0.51 for the logarithm of infinite dilution activity coefficients, demonstrating improvements over the previous openCOSMO-RS 22 parameterization and comparable results to COSMOtherm 24 BP-TZVP. The user interface was extended to be able use it as solvation model directly from within ORCA 6.0 or from the command line to provide researchers with a robust tool for applications in chemical and materials science.

Impact of applying machine learning to the electronic medical record on prediction of cancer-associated thrombosis.

409 Background: Cancer-associated thrombosis (CAT) is preventable among high-risk individuals with prophylactic anticoagulation. Risk assessment tools focus only on the initial period after cancer diagnosis; however, some patients may face different risks during different periods of their cancer journey. We developed a longitudinal machine learning system to predict the risk of CAT throughout cancer treatment. Methods: Using electronic health record data at Princess Margaret Cancer Centre, we assembled a cohort of people with thoracic and gastrointestinal cancer receiving systemic treatments between August 1, 2017 and December 31, 2019. We manually labelled 5,000 CT scans and 500 Doppler ultrasounds for the occurrence of CAT. We fine-tuned open-source large language models (LLMs) on those labelled reports, which we then used to automatically detect CAT among all 37,000 CT scans and Dopplers. Finally, we trained longitudinal machine learning systems to predict CAT within 90 days after each cancer treatment. LLMs and the CAT risk prediction system were evaluated among the held-out test cohort of people whose first treatment occurred in 2019. Results: The overall cohort included 2,031 patients and 21,375 treatment sessions. When classifying radiology reports, the fine-tuned LLM achieved an area under the receiver operating characteristic curve (AUROC) of 0.758 for DVT and 0.988 for PE in the test cohort. CAT occurred within 90 days after 6.23% of treatment sessions. The best ML system predicted the risk of CAT within 90 days with an AUROC of 0.651 (95% CI, 0.620-0.678). Considering only the first treatment per patient, treatments within the first 90 days after the first treatment, and treatments after 90 days, the system achieved AUROC of 0.639 (95% CI, 0.568-0.704), 0.599 (95% CI, 0.559-0.642), and 0.741 (95% CI, 0.699-0.783), respectively. Conclusions: Machine learning can longitudinally predict CAT among patients receiving systemic therapy for aerodigestive cancers. These systems could personalize prophylactic anticoagulation by identifying specific periods when patients face sufficient risk to warrant prevention.

OphGLM: An ophthalmology large language-and-vision assistant

Vision computer-aided diagnostic methods have been used in early ophthalmic disease screening and diagnosis. However, the limited output formats of these methods lead to poor human–computer interaction and low clinical applicability value. Thus, ophthalmic visual question answering is worth studying. Unfortunately, no practical solutions exist before Large Language Models(LLMs). In this paper, we investigate the ophthalmic visual diagnostic interaction problem. We construct an ophthalmology large language-and-vision assistant, OphGLM, consisting of an image encoder, a text encoder, a fusion module, and an LLM module. We establish a new Chinese ophthalmic fine-tuning dataset, FundusTuning-CN, including the fundus instruction and conversation sets. Based on FundusTuning-CN, we establish a novel LLM-tuning strategy to introduce visual model understanding and ophthalmic knowledge into LLMs at a low cost and high efficiency. Leveraging the pre-training of the image encoder, OphGLM demonstrates strong visual understanding and surpasses open-source visual language models in common fundus disease classification tasks. The FundusTuning-CN enables OphGLM to surpass open-source medical LLMs in both ophthalmic knowledge and interactive capabilities. Our proposed OphGLM has the potential to revolutionize clinical applications in ophthalmology. The dataset, code, and models will be publicly available at https://github.com/ML-AILab/OphGLM.

A ferry route in the Skagerrak optimised via VISIR-2

Abstract Urgent measures are needed for the shipping industry to contribute proportionately to climate change mitigation. To date, the potential of weather routing has not been reproducibly assessed, especially for short sea shipping. We employ the open-source VISIR-2 model on a ferry passage in the Skagerrak (the strait among Norway, Sweden, and Denmark), considering both sea currents and waves. By computing 5,840 least-CO2 optimal routes for 2023, we find promising results, particularly for voyages from Denmark. The savings follow a bi-exponential distribution, with over 10% CO2 emissions savings achievable on about ten days per year. Specifically, the Norwegian coastal current can be harnessed to enhance speed over ground and reduce overall emissions. The vessel performance curve needs to be tailored to specific ships in service, and the optimisation algorithm should be developed to enable just-in-time arrivals.

Simulation-free prediction of atrial fibrillation inducibility with the fibrotic kernel signature

Computational models of atrial fibrillation (AF) can help improve success rates of interventions, such as ablation. However, evaluating the efficacy of different treatments requires performing multiple costly simulations by pacing at different points and checking whether AF has been induced or not, hindering the clinical application of these models. In this work, we propose a classification method that can predict AF inducibility in patient-specific cardiac models without running additional simulations. Our methodology does not require re-training when changing atrial anatomy or fibrotic patterns. To achieve this, we develop a set of features given by a variant of the heat kernel signature that incorporates fibrotic pattern information and fiber orientations: the fibrotic kernel signature (FKS). The FKS is faster to compute than a single AF simulation, and when paired with machine learning classifiers, it can predict AF inducibility in the entire domain. To learn the relationship between the FKS and AF inducibility, we performed 2371 AF simulations comprising 6 different anatomies and various fibrotic patterns, which we split into training and a testing set. We obtain a median F1 score of 85.2% in test set and we can predict the overall inducibility with a mean absolute error of 2.76 percent points, which is lower than alternative methods. We think our method can significantly speed-up the calculations of AF inducibility, which is crucial to optimize therapies for AF within clinical timelines. An example of the FKS for an open source model is provided in https://github.com/tbanduc/FKS_AtrialModel_Ferrer.git.

Driving sustainable energy transitions with a multi-source RAG-LLM system

By 2035, the UK aims to upgrade all homes to achieve a net-zero economy by 2050, thereby reducing energy consumption, household costs, and improving living conditions. Small and Medium-sized Enterprises (SMEs) play a crucial role in this transition. However, many SME contractors lack essential information on Sustainable Energy Initiatives (SEIs) and the relevant Energy landscape necessary for driving Sustainable Energy Transitions (SETs). This knowledge gap poses risks to SME interventions, potentially leading to increased costs and inefficiencies. Accessing timely information on SEIs including government policies, funding, technologies, and environmental impacts from various media sources is essential for guiding effective SETs and understanding the relevant Energy landscape, thereby facilitating informed decision-making. Currently, SMEs lack an integrated system that consolidates data from diverse media sources into a centralized Information System (IS), limiting their ability to effectively navigate SEIs. To address this gap, this research introduces an Energy Chatbot, a sustainable IS that utilizes Large Language Models (LLMs) integrated with multi-source Retrieval Augmented Generation (RAG). This system encompasses diverse media sources, including news articles, government reports, industry publications, academic research, and social media. The Energy Chatbot is designed to enhance decision-making for SMEs by providing comprehensive Energy sector insights through a Question Answering (QA) system. Key findings emphasize that this approach reduces costs by utilizing open-source models. Moreover, the Energy Chatbot provides SMEs with access to up-to-date information, enabling them to identify long-term sustainability strategies and maintain a competitive edge in the evolving Energy landscape.

Towards building multilingual language model for medicine

The development of open-source, multilingual medical language models can benefit a wide, linguistically diverse audience from different regions. To promote this domain, we present contributions from the following: First, we construct a multilingual medical corpus, containing approximately 25.5B tokens encompassing 6 main languages, termed as MMedC, enabling auto-regressive domain adaptation for general LLMs; Second, to monitor the development of multilingual medical LLMs, we propose a multilingual medical multi-choice question-answering benchmark with rationale, termed as MMedBench; Third, we have assessed a number of open-source large language models (LLMs) on our benchmark, along with those further auto-regressive trained on MMedC. Our final model, MMed-Llama 3, with only 8B parameters, achieves superior performance compared to all other open-source models on both MMedBench and English benchmarks, even rivaling GPT-4. In conclusion, in this work, We present a large-scale corpus, a benchmark and a series of models to support the development of multilingual medical LLMs.

An open-source fine-tuned large language model for radiological impression generation: a multi-reader performance study

BackgroundThe impression section integrates key findings of a radiology report but can be subjective and variable. We sought to fine-tune and evaluate an open-source Large Language Model (LLM) in automatically generating impressions from the remainder of a radiology report across different imaging modalities and hospitals.MethodsIn this institutional review board-approved retrospective study, we collated a dataset of CT, US, and MRI radiology reports from the University of California San Francisco Medical Center (UCSFMC) (n = 372,716) and the Zuckerberg San Francisco General (ZSFG) Hospital and Trauma Center (n = 60,049), both under a single institution. The Recall-Oriented Understudy for Gisting Evaluation (ROUGE) score, an automatic natural language evaluation metric that measures word overlap, was used for automatic natural language evaluation. A reader study with five cardiothoracic radiologists was performed to more strictly evaluate the model’s performance on a specific modality (CT chest exams) with a radiologist subspecialist baseline. We stratified the results of the reader performance study based on the diagnosis category and the original impression length to gauge case complexity.ResultsThe LLM achieved ROUGE-L scores of 46.51, 44.2, and 50.96 on UCSFMC and upon external validation, ROUGE-L scores of 40.74, 37.89, and 24.61 on ZSFG across the CT, US, and MRI modalities respectively, implying a substantial degree of overlap between the model-generated impressions and impressions written by the subspecialist attending radiologists, but with a degree of degradation upon external validation. In our reader study, the model-generated impressions achieved overall mean scores of 3.56/4, 3.92/4, 3.37/4, 18.29 s,12.32 words, and 84 while the original impression written by a subspecialist radiologist achieved overall mean scores of 3.75/4, 3.87/4, 3.54/4, 12.2 s, 5.74 words, and 89 for clinical accuracy, grammatical accuracy, stylistic quality, edit time, edit distance, and ROUGE-L score respectively. The LLM achieved the highest clinical accuracy ratings for acute/emergent findings and on shorter impressions.ConclusionsAn open-source fine-tuned LLM can generate impressions to a satisfactory level of clinical accuracy, grammatical accuracy, and stylistic quality. Our reader performance study demonstrates the potential of large language models in drafting radiology report impressions that can aid in streamlining radiologists’ workflows.

Developing Robust Human Liver Microsomal Stability Prediction Models: Leveraging Inter-Species Correlation with Rat Data.

Objectives: Pharmacokinetic issues were the leading cause of drug attrition, accounting for approximately 40% of all cases before the turn of the century. To this end, several high-throughput in vitro assays like microsomal stability have been developed to evaluate the pharmacokinetic profiles of compounds in the early stages of drug discovery. At NCATS, a single-point rat liver microsomal (RLM) stability assay is used as a Tier I assay, while human liver microsomal (HLM) stability is employed as a Tier II assay. We experimentally screened and collected data on over 30,000 compounds for RLM stability and over 7000 compounds for HLM stability. Although HLM stability screening provides valuable insights, the increasing number of hits generated, along with the time- and resource-intensive nature of the assay, highlights the need for alternative strategies. One promising approach is leveraging in silico models trained on these experimental datasets. Methods: We describe the development of an HLM stability prediction model using our in-house HLM stability dataset. Results: Employing both classical machine learning methods and advanced techniques, such as neural networks, we achieved model accuracies exceeding 80%. Moreover, we validated our model using external test sets and found that our models are comparable to some of the best models in literature. Additionally, the strong correlation observed between our RLM and HLM data was further reinforced by the fact that our HLM model performance improved when using RLM stability predictions as an input descriptor. Conclusions: The best model along with a subset of our dataset (PubChem AID: 1963597) has been made publicly accessible on the ADME@NCATS website for the benefit of the greater drug discovery community. To the best of our knowledge, it is the largest open-source model of its kind and the first to leverage cross-species data.

Open-source hand prosthesis: evaluation of mechanical feasibility and additive manufacturing potential

This study delves into the risks associated with 3D-printed hand prostheses lacking mechanical feasibility studies. Such unvalidated prostheses may exhibit various issues like durability shortcomings, anatomical incompatibility, functional safety concerns, manufacturing quality deficiencies, and health risks due to inappropriate materials. These issues can lead to damage or premature failure during use, discomfort, skin irritation, injuries, inability to withstand functional loads, and health hazards from toxic or allergenic substances. Therefore, ensuring the safety, quality, and effectiveness of these prostheses is crucial. The study focuses on a mechanical feasibility study conducted through Finite Element Analysis (FEA) simulations on an open-source hand prosthesis model. It evaluates mechanical properties, stress concentration areas, and displacement on the prosthesis surface. The methodology comprises three key steps: acquiring the virtual model, conducting computational simulations, and selecting the 3D printing material. The simulations assess the prosthesis’s ability to withstand compressive forces and identify stress concentration areas. Results from the study indicate that using PETG as the constituent material demonstrates mechanical viability and satisfactory performance under static force conditions. This finding underscores the importance of rigorous testing and adherence to standards in developing 3D-printed hand prostheses. Such studies contribute significantly to enhancing these devices’ safety and effectiveness, facilitating their broader adoption in healthcare settings. In summary, this study highlights the critical need for mechanical feasibility studies in the development of 3D-printed hand prostheses. It emphasizes the significance of following strict standards and regulations to ensure these devices’ safety, quality, and functionality. By doing so, it paves the way for the widespread use of these prostheses in healthcare, benefiting users and advancing the field of prosthetic technology.

Servicing open-source large language models for oncology

Servicing open-source large language models for oncology

Toponym resolution leveraging lightweight and open-source large language models and geo-knowledge

Toponym resolution is crucial for extracting geographic information from natural language texts, such as social media posts and news articles. Despite the advancements in current methods, including state-of-the-art deep learning solutions like GENRE and a sophisticated voting system that integrates seven individual methods, further enhancing their accuracy is essential. To achieve this goal, we propose a novel method that combines lightweight and open-source large language models and geo-knowledge. Specifically, we first fine-tune Mistral (7B), Baichuan2 (7B), Llama2 (7B & 13B), and Falcon (7B) to estimate toponyms’ unambiguous reference (e.g., city, state, country) given their contexts. Subsequently, we correct inaccuracies in generated references and determine their geo-coordinates via sequentially querying GeoNames, Nominatim, and ArcGIS geocoders until a successful geocoding result is achieved. Our methods demonstrate enhanced performance compared to 20 existing methods, as evidenced across seven challenging datasets including 83,365 toponyms worldwide, with the Mistral-based method leading, followed by Baichuan2, Llama2, and Falcon-based methods. Specifically, the Mistral-based method achieves an Accuracy@161km of 0.91, surpassing GENRE, the best individual method, by 17% and the seven-methods composite voting system by 7%. Moreover, our methods are computationally efficient, operable on one general GPU, have modest memory requirements (14 GB for 7B models and 27 GB for 13B models), and exceed both GENRE and the voting system in inferring speed.

Optimising the review of electronic monitoring information for management of commercial fisheries

Electronic monitoring (EM) systems incorporating cameras and other devices can collect a broad range of data to support fisheries management. We reviewed the data collection capabilities of EM and considered approaches to increasing efficiency, including cost effectiveness, of EM review. EM can provide information on catch, effort, catch handling, bycatch mitigation, fishing gear and operational data, which are relevant for fisheries management including by Regional Fisheries Management Organisations (RFMOs). Methods to increase efficiency and decrease costs of EM review apply from the programme design phase, through data collection and review. At review, costs may be reduced by sampling imagery optimally to meet monitoring objectives. Considering RFMOs as users of EM-collected information, we applied EMoptim, an open-source simulation model developed in R that estimates the amount of EM review necessary to meet one or more user-specified monitoring objectives. EMoptim uses stratification to increase review efficiency and incorporates a function to explore review costs against the monitoring objectives set. We evaluated the amount of EM review needed to estimate catch with specified precision, using fishery data available from the Western and Central Pacific Fisheries Commission. Model outputs show that EM review requirements increase as catch frequency decreases, dispersion of catch events increases, and when more precise catch estimates are required. Geographical stratification reduced the amount of review required for more commonly caught species and when catch events were focused in a limited area. Optimising review rates across multiple monitoring objectives was most effective for more commonly caught species. We highlight opportunities for future use and development of this prototype modelling package.

The potential role of concentrated solar power for off-grid green hydrogen and ammonia production

This paper investigates the potential role of concentrated solar power (CSP) in off-grid green electrolytic hydrogen and ammonia production using an open-source techno-economic assessment optimisation model. The green ammonia plant comprises a flexible Haber-Bosch loop, an electrolyser park, a desalination plant, storage systems (thermal, electrical, and hydrogen), and a power supply optionally including wind power, solar photovoltaic (PV), and CSP (solar tower or parabolic trough collectors). Various system configurations and cost assumptions are tested to evaluate when CSP technologies are economically viable for green fuel production. Results demonstrate that CSP and thermal energy storage (TES) technologies can play a minor but useful role in providing nighttime power supply to the ammonia plant, in combination with onshore wind power and electrical storage. Solar PV with 1-axis tracking remains the primary power provider, and the optimal system always includes large-scale hydrogen storage. In the cheapest configuration, the LCOA reaches 646 €/tNH3. With an 80% reduction in CSP and TES costs, CSP technology becomes a major player in power production. In scenarios where hydrogen storage is not feasible and the ammonia plant lacks full flexibility, a 20% cost reduction (to 2750 €/kW) is enough for them to become major power providers.

Comparative Assessment of Otolaryngology Knowledge Among Large Language Models

ObjectiveThe purpose of this study was to evaluate the performance of advanced large language models from OpenAI (GPT‐3.5 and GPT‐4), Google (PaLM2 and MedPaLM), and an open source model from Meta (Llama3:70b) in answering clinical test multiple choice questions in the field of otolaryngology—head and neck surgery.MethodsA dataset of 4566 otolaryngology questions was used; each model was provided a standardized prompt followed by a question. One hundred questions that were answered incorrectly by all models were further interrogated to gain insight into the causes of incorrect answers.ResultsGPT4 was the most accurate, correctly answering 3520 of 4566 questions (77.1%). MedPaLM correctly answered 3223 of 4566 (70.6%) questions, while llama3:70b, GPT3.5, and PaLM2 were correct on 3052 of 4566 (66.8%), 2672 of 4566 (58.5%), and 2583 of 4566 (56.5%) questions. Three hundred and sixty‐nine questions were answered incorrectly by all models. Prompts to provide reasoning improved accuracy in all models: GPT4 changed from incorrect to correct answer 31% of the time, while GPT3.5, Llama3, PaLM2, and MedPaLM corrected their responses 25%, 18%, 19%, and 17% of the time, respectively.ConclusionLarge language models vary in their understanding of otolaryngology‐specific clinical knowledge. OpenAI's GPT4 has a strong understanding of core concepts as well as detailed information in the field of otolaryngology. Its baseline understanding in this field makes it well‐suited to serve in roles related to head and neck surgery education provided that the appropriate precautions are taken and potential limitations are understood.Level of EvidenceN/A Laryngoscope, 2024

Knowledge-based model generation for aircraft cabin noise prediction from pre-design data

The climate targets set out in the “European Green Deal” call for the consideration and implementation of climate-friendly propulsion concepts and sustainable fuels in future aircraft configurations. This puts the use of very efficient propeller engines into the focus of aircraft design. However, these pose major challenges, especially to cabin acoustics, due to high, tonal sound pressure levels on the fuselage. The design of noise control measures requires the competence to predict the resulting interior noise as early as possible in the design process of the vehicle. This requires a high level of geometric and structural detail regarding the fuselage structure and cabin components. With increasing frequency range, the necessary structural details also increase, which have to be resolved in the simulation models due to the associated decreasing structural wavelength. Especially in the context of aircraft pre-design, there is usually not enough information available for a detailed vibro-acoustic modeling of the fuselage and cabin components, which allows a meaningful prediction of the vibrations and thus the cabin noise. Therefore, the knowledge-based tool Fuselage Geometry Assembler (FUGA) is developed for the targeted enrichment of preliminary design data with knowledge for detailed numerical analyses. This paper describes the knowledge-based geometry and model generation in FUGA, which can consider the necessary (increasing) level of detail for the vibro-acoustic prediction already in the preliminary design. For this purpose, aircraft data sets in the preliminary design data format Common Parametric Aircraft Configuration Schema (CPACS) form the modeling basis. Originating from the aircraft preliminary design, these initially describe the outer shell of the vehicle and are extended by detailed structural information that defines the geometric boundary conditions for component placement in cabin design. For the cabin components, the open-source geometry kernel Open Cascade Technology (OCCT) is used to provide geometries at the level of detail required for subsequent analyses. The geometry models are then discretized in open source (e.g. Gmsh) or commercial meshers and further used for numerical analysis. Finally, the prediction of cabin noise is demonstrated as a Proof of Concept using the example of a short-haul propeller-driven aircraft and the feasibility of the proposed method is indicated by investigating the sensitivity of resulting simulation models to the fuselage skin thickness.

Quantitative aortic Na[18F]F positron emission tomography computed tomography as a tool to associate vascular calcification with major adverse cardiovascular events

Abstract Purpose Sodium[18F]fluoride (Na[18F]F) used in positron emission tomography (PET) binds to active calcification and correlates consistently with higher cardiovascular risk. This study aims to investigate the feasibility of aortic Na[18F]F-PET in hybrid combination with low-dose computed tomography (CT) as a risk model for major adverse cardiovascular events (MACE). Methods Patient data and Na[18F]F-PET/CT scans from January 2019 to February 2022 were retrospectively collected at the University Medical Center Groningen (UMCG), the Netherlands. MACE-outcome was a composite of time to first documented myocardial infarction, cerebral vascular accident (CVA), acute heart failure hospitalization, and aortic aneurysms. MACE dates were recorded from the day of the scan until follow-up in December 2023. The aorta was manually segmented in all low-dose CT scans. To minimize spill-over effects from the vertebrae, the vertebrae were automatically segmented using an open-source model, dilated with 10 mm, and subtracted from the aortic mask. The total aortic Na[18F]F corrected maximum standardized uptake value (cSUVmax) and total aortic Agatston score were automatically calculated using SEQUOIA. Kaplan–Meier and Cox regression survival analysis were performed, stratifying patients into high, medium, and low cSUVmax and Agatston categories. Cox regression models were adjusted for age. Results Out of 280 identified scans, 216 scans of unique patients were included. During a median follow-up of 3.9 years, 12 MACE occurred. Kaplan–Meier survival analysis demonstrated a significant difference in MACE-free survival among the high cSUVmax group compared to the medium and low groups (p = 0.03 and p < 0.01, respectively). Similarly, patients with high Agatston scores had a significantly lower MACE-free survival probability compared to those with medium and low scores (both p < 0.01). Conclusion This study highlights the potential clinical utility of Na[18F]F-PET/CT as an imaging tool to predict the risk of MACE. Clinical validation of this novel proof-of-concept method is needed to confirm these results and expand the clinical context.

Assessing the Capability of Advanced AI Models in Cardiovascular Symptom Recognition: A Comparative Study

The field of medical informatics has been significantly transformed in recent years with the emergence of Natural Language Understanding (NLU) and Large Language Models (LLM), providing new opportunities for innovative patient care solutions. This study aims to evaluate the effectiveness of publicly available LLMs as symptom checkers for cardiological diseases by comparing their diagnostic capabilities in real disease cases. We employed a set of 9 models, including ChatGPT-4, OpenSource models, Google PaLM 2, and Meta’s LLaMA, to assess their diagnostic accuracy, reliability, and safety across various clinical scenarios. Our methodology involved presenting these LLMs with symptom descriptions and test results in Spanish, requiring them to provide specialist diagnoses and recommendations in English. This approach allowed us to compare the performance of each model, highlighting their respective strengths and limitations in a healthcare context. The results revealed varying levels of accuracy, precision, and sensitivity among the models, demonstrating the potential of LLMs to enhance medical education and patient care. By analysing the capabilities of each model, our study contributes to a deeper understanding of artificial intelligence’s role in medical diagnosis. We argue for the strategic implementation of LLMs in healthcare, emphasizing the importance of balancing sensitivity and realism to optimize patient outcomes.

Tailoring Risk Prediction Models to Local Populations.

Risk estimation is an integral part of cardiovascular care. Local recalibration of guideline-recommended models could address the limitations of existing tools. To provide a machine learning (ML) approach to augment the performance of the American Heart Association's Predicting Risk of Cardiovascular Disease Events (AHA-PREVENT) equations when applied to a local population while preserving clinical interpretability. This cohort study used a New England-based electronic health record cohort of patients without prior atherosclerotic cardiovascular disease (ASCVD) who had the data necessary to calculate the AHA-PREVENT 10-year risk of developing ASCVD in the event period (2007-2016). Patients with prior ASCVD events, death prior to 2007, or age 79 years or older in 2007 were subsequently excluded. The final study population of 95 326 patients was split into 3 nonoverlapping subsets for training, testing, and validation. The AHA-PREVENT model was adapted to this local population using the open-source ML model (MLM) Extreme Gradient Boosting model (XGBoost) with minimal predictor variables, including age, sex, and AHA-PREVENT. The MLM was monotonically constrained to preserve known associations between risk factors and ASCVD risk. Along with sex, race and ethnicity data from the electronic health record were collected to validate the performance of ASCVD risk prediction in subgroups. Data were analyzed from August 2021 to February 2024. Consistent with the AHA-PREVENT model, ASCVD events were defined as the first occurrence of either nonfatal myocardial infarction, coronary artery disease, ischemic stroke, or cardiovascular death. Cardiovascular death was coded via government registries. Discrimination, calibration, and risk reclassification were assessed using the Harrell C index, a modified Hosmer-Lemeshow goodness-of-fit test and calibration curves, and reclassification tables, respectively. In the test set of 38 137 patients (mean [SD] age, 64.8 [6.9] years, 22 708 [59.5]% women and 15 429 [40.5%] men; 935 [2.5%] Asian, 2153 [5.6%] Black, 1414 [3.7%] Hispanic, 31 400 [82.3%] White, and 2235 [5.9%] other, including American Indian, multiple races, unspecified, and unrecorded, consolidated owing to small numbers), MLM-PREVENT had improved calibration (modified Hosmer-Lemeshow P > .05) compared to the AHA-PREVENT model across risk categories in the overall cohort (χ23 = 2.2; P = .53 vs χ23 > 16.3; P < .001) and sex subgroups (men: χ23 = 2.1; P = .55 vs χ23 > 16.3; P < .001; women: χ23 = 6.5; P = .09 vs. χ23 > 16.3; P < .001), while also surpassing a traditional recalibration approach. MLM-PREVENT maintained or improved AHA-PREVENT's calibration in Asian, Black, and White individuals. Both MLM-PREVENT and AHA-PREVENT performed equally well in discriminating risk (approximate ΔC index, ±0.01). Using a clinically significant 7.5% risk threshold, MLM-PREVENT reclassified a total of 11.5% of patients. We visualize the recalibration through MLM-PREVENT ASCVD risk charts that highlight preserved risk associations of the original AHA-PREVENT model. The interpretable ML approach presented in this article enhanced the accuracy of the AHA-PREVENT model when applied to a local population while still preserving the risk associations found by the original model. This method has the potential to recalibrate other established risk tools and is implementable in electronic health record systems for improved cardiovascular risk assessment.