Information Extraction Research Articles

Neural architecture search for multi-sensor information fusion-based intelligent fault diagnosis

The multi-sensor information fusion-based intelligent fault diagnosis (MIF-IFD) method using deep learning aims to provide more accurate and reliable ways of identifying equipment health status than single-sensor methods. However, designing feature extraction modules in deep networks and determining information fusion execution layers still requires expertise and extensive experimentation, imposing high development costs. This limitation impedes the development and application of MIF-IFD. This paper proposes an architecture to mitigate the development challenges of MIF-IFD. Firstly, the basic module of the network consists of searchable feature extraction cells and a mask self-attention mechanism to achieve the extraction and fusion of multi-sensor information. Secondly, the optimal feature extraction cells and fusion starting layers are searched by constructing a continuous search space to compose the optimal fusion diagnostic structure, respectively. Finally, validation and analysis are conducted using two experimental datasets. The results demonstrate that the proposed method not only ensures diagnostic accuracy but also significantly improves development efficiency.

Multi-view brain functional connectivity and hierarchical fusion for EEG-based emotion recognition

Emotional states evolve gradually from their onset to full manifestation, reflected in changes in brain functional connectivity. Existing research often focuses only on the final emotional state, missing the rich, dynamic information of emotional evolution. This paper proposes a novel network framework leveraging global and local brain functional connectivity through a multi-view approach. We introduce a global and local connectivity modeling method using electroencephalogram signals to simulate brain connectivity changes across time and specific moments. A residual-based local detail feature extractor combined with a coordinate attention module captures long-range dependencies and maintains positional accuracy. A global feature extractor is also designed for the effective extraction of emotional information. Finally, a hierarchical multi-scale cross-attention feature fusion module integrates features across different levels to enhance emotion recognition performance and robustness. Extensive experiments on three publicly available datasets demonstrate the framework’s superiority and generalization capability.

Mid-infrared hyperspectral microscopy with broadband 1-GHz dual frequency combs

Mid-infrared microscopy is an important tool for biological analyses, allowing a direct probe of molecular bonds in their low energy landscape. In addition to the label-free extraction of spectroscopic information, the application of broadband sources can provide a third dimension of chemical specificity. However, to enable widespread deployment, mid-infrared microscopy platforms need to be compact and robust while offering high speed, broad bandwidth, and high signal-to-noise ratio. In this study, we experimentally showcase the integration of a broadband, high-repetition-rate dual-comb spectrometer (DCS) in the mid-infrared range with a scanning microscope. We employ a set of 1-GHz mid-infrared frequency combs, demonstrating their capability for high-speed and broadband hyperspectral imaging of polymers and ovarian tissue. The system covers 1000 cm−1 at νc = 2941 cm−1 with 12.86 kHz spectra acquisition rate and 5 µm spatial resolution. Taken together, our experiments and analysis elucidate the trade-off between bandwidth and speed in DCS as it relates to microscopy. This provides a roadmap for the future advancement and application of high-repetition-rate DCS hyperspectral imaging.

Application of Facial Analysis Based on Convolutional Neural Network and Iterative Decision Tree for Teaching Evaluation in Smart Classrrom

The creation of intelligent classrooms has been expedited by the rapid advancement of the Internet and computer vision, and intelligent teaching has increased the interactivity and effectiveness of learning. Teachers’ teaching and students’ classroom learning state ultimately affect the teaching effect. Students’ facial expressions during class can reflect emotional changes and the current learning state. The computer camera in the smart classroom collects students’ face image data, uses texture-based information, edge-based information, geometric information, and global and local feature extraction to identify and analyze and process the students’ facial expressions. Research has shown that the combination of expression recognition and an intelligent teaching classroom can accurately identify and analyze students’ emotions and learning status, and can effectively evaluate the teaching effect of the intelligent classroom, which helps to improve teaching quality and learning efficiency. Therefore, applying facial expression recognition in the intelligent teaching classroom has far-reaching significance.

Faint Space Target Information Extraction Based on Small Aperture Telescope in Complex Background

AbstractThere are many problems in space debris monitoring with ground‐based telescopes, such as too many stars in the same field of view, uneven background and optical distortion in the optical system. We propose a two‐stage weak debris detection algorithm. In the first stage, wavelet transform is used to extract different components of three frames of images, and the median of corresponding components of the images is taken respectively to eliminate the influence of stars. In the second stage, an improved version of the faint space target extraction based on principal component analysis. The algorithm uses a smooth‐detection idea to extract target information. Based on a 150 mm aperture telescope, we improved the existing method of faint space debris extraction based on principal component analysis by introducing the smooth‐detection idea, and transformed the target detection problem into the separation problem of sparse matrix and low‐rank matrix. We applied a certain preprocessing consisting of wavelet‐based star removal and median pre‐filtering to keep as little noise and other contaminants as possible. After experimental measurements by observers, the algorithm demonstrated advanced detection capabilities on multiple indicators.

Short-term heating load forecasting model based on SVMD and improved informer

Accurate heat load forecasting can not only effectively reduce energy consumption, but also improve the efficiency of the heating system and user comfort. In order to improve the accuracy of heat load forecasting, we proposed a forecasting method combining SVMD algorithm and improved Informer model and applied it to the heat load forecasting of district heating system. First, SVMD algorithm is used to decompose the heat load data, and then relative position coding is introduced into Informer model, and causal convolution and jump connection mechanisms are adopted to better capture the dependence of sequence data and enhance the ability of local information extraction. In order to verify the model performance, experiments were conducted on multiple data sets. The results show that the R2 evaluation index of the model reaches 98.7 %, which is 10.8 %, 17.1 %, 6.3 %, 4.9 %, 2.3 %, 1.5 % and 16.6 % higher than SVR, RNN, LSTM, Transformer, SH-Informer, iTransformer and DLinear respectively. The validity of the model is further verified by DM test, which provides a reference for further improving the timeliness of intelligent heating.

Uncovering condition information loss in medical text extraction: The challenge of non-contiguous spans

We investigated the limitations of conventional named entity recognition (NER) and entity linking (EL) methods in accurately extracting patient condition information from medical texts, focusing on the challenges posed by non-contiguous spans and the potential information loss. We utilized a corpus with entity-relation annotations, analyzing the frequency and nature of non-contiguous spans that include irrelevant entities within gaps. The corpus was further analyzed to pinpoint the types of entity representations predominantly linked with peripheral spans—those not encompassing central symptom-describing terms—with a focus on items, body parts, and clinical tests. Our analysis revealed that 18.6 % of patient condition expressions were non-contiguous spans containing irrelevant entities, suggesting an accuracy ceiling of 81.4 % for conventional NER and EL approaches in the worst-case scenario. The study highlights the importance of entity types such as items, body parts, and clinical tests in these expressions, indicating that conventional extraction methods incur considerable information loss. The findings underscore the need for more sophisticated information extraction techniques capable of handling the complexities of medical texts, including non-contiguous spans. Adapting methods that allow gaps within entities or employing graph-based term assignments can enhance the accuracy and comprehensiveness of medical text annotation.

A Method for Extracting Power Entity Relationships Based on Hybrid Neural Networks

In response to the challenges of entity relationship extraction in unstructured text, the author proposes a power entity relationship extraction method based on a hybrid neural network. This method aims to overcome the limitations of existing models in accurately representing contextual environment information, thereby improving the accuracy of the extraction model to meet practical application needs. Firstly, a Bidirectional Gated Recurrent Unit (BiGRU) was designed to better capture contextual information in text sequences. This helps the model to better understand the relationships between entities. Secondly, an attention mechanism was adopted to enable the model to automatically focus on sequence features that have a significant impact on relationships. This helps the model to extract entity relationships more accurately in complex text environments. Finally, a segmented convolutional neural network (PCNN) was introduced to further improve the accuracy of relationship extraction by learning the environmental feature information in the adjusted sequence. This enables the model to better understand the contextual relationships between entities. On the publicly available English dataset SemEval2010Task8, this method achieved satisfactory results, achieving an F1 value of 85.62%. These experiments have confirmed the effectiveness of our method, providing new ideas and support for automatic extraction of entity relationships, and are expected to play an important role in the field of information extraction.

Causal-relationship representation enhanced joint extraction model for elements and relationships

The joint extraction of elements such as entities, relations, events, and their arguments, along with their specific interrelationships, is a critical task in natural language processing. Existing research often employs implicit handling of interactions between tasks through techniques like shared encodings or parameter sharing, lacking explicit modeling of the specific relationships between tasks. This limitation hinders the full utilization of inter-task correlation information and impacts effective collaboration between tasks. To address this, we propose a model for joint extraction of elements and relations based on causal relationship representation enhancement (CRE). This model aims to capture specific relationships between tasks in multiple stages, facilitating finer adjustments and optimization of subtasks and thereby enhancing the overall model performance. Specifically, CRE comprises three key modules: feature adaptation, feature interaction, and feature fusion. The feature adaptation module selects and adjusts features from shared encodings based on the requirements of specific tasks to better adapt to semantic differences between different tasks. The feature interaction module employs causal reasoning to comprehensively capture the causal relationships between tasks while mitigating negative transfer brought about by interference from semantic information. The feature fusion module further integrates features to obtain optimized task-specific representations. Ultimately, CRE exhibits a significant improvement in average performance across multiple information extraction tasks.

Disambiguating Clinical Abbreviations by One-to-All Classification: Algorithm Development and Validation Study.

Electronic medical records store extensive patient data and serve as a comprehensive repository, including textual medical records like surgical and imaging reports. Their utility in clinical decision support systems is substantial, but the widespread use of ambiguous and unstandardized abbreviations in clinical documents poses challenges for natural language processing in clinical decision support systems. Efficient abbreviation disambiguation methods are needed for effective information extraction. This study aims to enhance the one-to-all (OTA) framework for clinical abbreviation expansion, which uses a single model to predict multiple abbreviation meanings. The objective is to improve OTA by developing context-candidate pairs and optimizing word embeddings in Bidirectional Encoder Representations From Transformers (BERT), evaluating the model's efficacy in expanding clinical abbreviations using real data. Three datasets were used: Medical Subject Headings Word Sense Disambiguation, University of Minnesota, and Chia-Yi Christian Hospital from Ditmanson Medical Foundation Chia-Yi Christian Hospital. Texts containing polysemous abbreviations were preprocessed and formatted for BERT. The study involved fine-tuning pretrained models, ClinicalBERT and BlueBERT, generating dataset pairs for training and testing based on Huang et al's method. BlueBERT achieved macro- and microaccuracies of 95.41% and 95.16%, respectively, on the Medical Subject Headings Word Sense Disambiguation dataset. It improved macroaccuracy by 0.54%-1.53% compared to two baselines, long short-term memory and deepBioWSD with random embedding. On the University of Minnesota dataset, BlueBERT recorded macro- and microaccuracies of 98.40% and 98.22%, respectively. Against the baselines of Word2Vec + support vector machine and BioWordVec + support vector machine, BlueBERT demonstrated a macroaccuracy improvement of 2.61%-4.13%. This research preliminarily validated the effectiveness of the OTA method for abbreviation disambiguation in medical texts, demonstrating the potential to enhance both clinical staff efficiency and research effectiveness.

TourismNER: A Tourism Named Entity Recognition method based on entity boundary joint prediction

Tourism named entity recognition is indispensable in tourism information extraction, and plays a crucial role in constructing tourism knowledge map and enhancing tourism knowledge quiz system. The difficulty of tourism named entity recognition lies in its complex nested structure, and the lengthy entity naming length. To address these existing problems, we propose a tourism named entity recognition model that jointly predicts entity boundaries, adopting a training strategy of data preprocessing to enhance the model’s ability for tourism named entity boundary recognition, while our model introduces a pre-trained Bert model as well as BiLSTM coding to enhance the representation of the model’s contexts, and uses a combined predictor of Biaffine and MLP to enhance the model’s recognition performance for boundaries, as well as introducing label smoothing cross entropy to smooth the target labels during the training process. Experiments are conducted on three datasets with different granularities. From the analysis of the experimental results, it can be seen that the named entity recognition method achieves higher accuracy and F1 value compared with the optimal baseline model, and also proves the effectiveness and generality of the modeling method proposed in this paper.

The versatile multi-functional substance NMN: its unique characteristics, metabolic properties, pharmacodynamic effects, clinical trials, and diverse applications.

β-nicotinamide mononucleotide (NMN) is a naturally occurring biologically active nucleotide widely present in organisms and an inherent substance in the human body. As a critical intermediate in synthesizing coenzyme I (NAD+), it widely participates in multiple biochemical reactions in the human body and is closely related to immunity, metabolism, and other factors. In recent years, NMN has rapidly developed and made significant progress in medicine, food, and healthcare. However, there is currently a lack of comprehensive reports on the research progress of NMN, as well as exploration and analysis of the current research achievements and progress of NMN. Therefore, this review is based on retrieving relevant research on NMN from multiple databases at home and abroad, with the retrieval time from database establishment to 20 May 2024. Subsequently, literature search, reading, key information extraction, organization, and summarization were conducted with the aim of providing a comprehensive and in-depth analysis of the characteristics, metabolic pathways, pharmacological effects, progress in human clinical trials, and wide applications of NMN in drug development and food applications. Furthermore, it offers personal insights into NMN's potential future developments and advancements to present the current development state and existing challenges comprehensively. Ultimately, this review aims to provide guidance and serve as a reference for the future application, innovation, and progression of NMN research.

The joint extraction of fact-condition statement and super relation in scientific text with table filling method

The fact-condition statements are of great significance in scientific text, via which the natural phenomenon and its precondition are detailly recorded. In previous study, the extraction of fact-condition statement and their relation (super relation) from scientific text is designed as a pipeline that the fact-condition statement and super relation are extracted successively, which leads to the error propagation and lowers the accuracy. To solve this problem, the table filling method is firstly adopted for joint extraction of fact-condition statement and super relation, and the Biaffine Convolution Neural Network model (BCNN) is proposed to complete the task. In the BCNN, the pretrained language model and Biaffine Neural Network work as the encoder, while the Convolution Neural Network is added into the model as the decoder that enhances the local semantic information. Benefiting from the local semantic enhancement, the BCNN achieves the best F1 score with different pretrained language models in comparison with other baselines. Its F1 scores in GeothCF (geological text) reach 73.17% and 71.04% with BERT and SciBERT as pretrained language model, respectively. Moreover, the local semantic enhancement also increases its training efficiency, via which the tags’ distribution can be more easily learned by the model. Besides, the BCNN trained with GeothCF also exhibits the best performance in BioCF (biomedical text), which indicates that it can be widely applied for the information extraction in all scientific domains. Finally, the geological fact-condition knowledge graph is built with BCNN, showing a new pipeline for construction of scientific fact-condition knowledge graph.

Evolution and spatiotemporal analysis of earthquake public opinion based on social media data

As critical conduits for the dissemination of online public opinion, social media platforms offer a timely and effective means for managing emergencies during major disasters, such as earthquakes. This study focuses on the analysis of online public opinions following the Maduo M7.4 earthquake in Qinghai Province and the Yangbi M6.4 earthquake in Yunnan Province. By collecting, cleaning, and organizing post-earthquake Sina Weibo (short for Weibo) data, we employed the Latent Dirichlet Allocation (LDA) model to extract information pertinent to public opinion on these earthquakes. This analysis included a comparison of the nature and temporal evolution of online public opinions related to both events. An emotion analysis, utilizing an emotion dictionary, categorized the emotional content of post-earthquake Weibo posts, facilitating a comparative study of the characteristics and temporal trends of online public emotions following the earthquakes. The findings were visualized using Geographic Information System (GIS) techniques. The analysis revealed certain commonalities in online public opinion following both earthquakes. Notably, the peak of online engagement occurred within the first 24 hours post-earthquake, with a rapid decline observed between 24 to 48 hours thereafter. The variation in popularity of online public opinion was linked to aftershock occurrences. Adjusted for population factors, online engagement in areas surrounding the earthquake sites and in Sichuan Province was significantly high. Initially dominated by feelings of “fear” and “surprise”, the public sentiment shifted towards a more positive outlook with the onset of rescue operations. However, distinctions in the online public response to each earthquake were also noted. Following the Yangbi earthquake, Yunnan Province reported the highest number of Weibo posts nationwide; in contrast, Qinghai Province ranked third post-Maduo earthquake, attributable to its smaller population size and extensive damage to communication infrastructure. This research offers a methodological approach for the analysis of online public opinion related to earthquakes, providing insights for the enhancement of post-disaster emergency management and public mental health support.

Temperature, Crime, and Violence: A Systematic Review and Meta-Analysis.

Heat is known to affect many health outcomes, but more evidence is needed on the impact of rising temperatures on crime and/or violence. We conducted a systematic review with meta-analysis regarding the influence of hot temperatures on crime and/or violence. In this systematic review and meta-analysis, we evaluated the relationship between increase in temperature and crime and/or violence for studies across the world and generated overall estimates. We searched MEDLINE and Web of Science for articles from the available database start year (1946 and 1891, respectively) to 6 November 2023 and manually reviewed reference lists of identified articles. Two investigators independently reviewed the abstracts and full-text articles to identify and summarize studies that analyzed the relationship between increasing temperature and crime, violence, or both and met a priori eligibility criteria. PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines were used to extract information from included articles. Some study results were combined using a profile likelihood random-effects model for meta-analysis for a subset of outcomes: violent crime (assault, homicide), property crime (theft, burglary), and sexual crime (sexual assault, rape). This review is registered at PROSPERO, CRD42023417295. We screened 16,634 studies with 83 meeting the inclusion criteria. Higher temperatures were significantly associated with crime, violence, or both. A 10°C (18°F) increase in short-term mean temperature exposure was associated with a 9% [95% confidence interval (CI): 7%, 12%] increase in the risk of violent crime (; eight studies). Studies had differing definitions of crime and/or violence, exposure assessment methods, and confounder assessments. Our findings summarize the evidence supporting the association between elevated temperatures, crime, and violence, particularly for violent crimes. Associations for some categories of crime and/or violence, such as property crimes, were inconsistent. Future research should employ larger spatial/temporal scales, consistent crime and violence definitions, advanced modeling strategies, and different populations and locations. https://doi.org/10.1289/EHP14300.

A simple model of dust extinction in gamma-ray burst host galaxies

Context. Gamma-ray burst (GRB) afterglows are powerful probes for studying the different properties of their host galaxies (e.g., the interstellar dust) at all redshifts. By fitting their spectral energy distribution (SED) over a large range of wavelengths, we can gain direct insights into the properties of the interstellar dust by studying the extinction curves. Unlike the dust extinction templates, such as those of the average Milky Way (MW) or the Small and Large Magellanic Cloud (SMC and LMC), the extinction curves of galaxies outside the Local Group exhibit deviation from these laws. Altogether, X-ray and gamma-ray satellites as well as ground-based telescopes, such as Neil Gehrels Swift Observatory (Swift) and Gamma-Ray Optical and Near-Infrared Detector (GROND), provide measurements of the afterglows from the X-ray to the NIR, which can be used to extract information on dust extinction curves along their lines of sight (LoS). The study presented in this paper undertakes such a photometric study, comprising a preparatory work for the SVOM mission and its ground-based follow-up telescope COLIBRI. Aims. We propose a simple approach to parameterize the dust extinction curve of GRB host galaxies. The model used in this analysis is based on a power law form with the addition of a Loretzian-like Drude profile with two parameters: the extinction slope, γ, and the 2175 Å bump amplitude, Eb. Methods. Using the g′r′i′z′JHKs GROND filter bands, we tested our dust extinction model and explored the parameter space in extinction and redshift by fitting SEDs of simplified simulations of GRB afterglow spectra based on different extinction curve templates. From a final sample of 10 real Swift/GROND extinguished GRBs, we determined the quantities of the dust extinction in their host and measured their extinction curves. Results. We find that our derived extinction curves are in agreement with the spectroscopic measurements reported for four GRBs in the literature. We compared four other GRBs to the results of photometric studies where fixed laws were used to fit their data. We additionally derived two new GRB extinction curves. The measured average extinction curve is given by a slope of γ = 1.051 ± 0.129 and Eb = 0.070 ± 0.036, which is equivalent to a quasi-featureless in-between SMC-LMC template. This is consistent with previous studies aimed at deriving the dust host galaxy extinction where we expect that small dust grains dominate in GRB environment, yielding a steeper curve than the mean MW extinction curve.

Personal and Psychological Predictors of Glycemic Control among Patients with Type 2 Diabetes at a Tertiary Care Center in Ghana

Abstract Background: As diabetes is a chronic disease, patients must make adjustments to their lifestyle and self-care practices to live optimally with the disease and reduce associated morbidity. Their ability to make these adjustments is partly dependent on their coping capacity and resilience, which may enhance glycemic control. Materials and Methods: A cross-sectional study involving 156 consented and randomly selected type 2 diabetes patients was conducted at Korle-Bu Teaching Hospital, a national health facility in Accra, Ghana. Structured questionnaires were used to extract information on sociodemographic characteristics and clinical parameters (including glycemic control), while the Africultural Coping Systems Inventory and the Adult Resilience Measure were administered to assess cultural variables and resilience. Stata version 16.1 was used to analyze the data, and the significance level was set at P ≤ 0.05. Results: Ethnicity, age, and working status were significantly associated with glycemic control. Adjusted binary logistic regression revealed that the odds of good glycemic control among nonhypertensive patients was 2.8 times greater compared with hypertensive patients (adjusted odds ratio [aOR]: 2.80, 95% confidence interval [CI]: 1.02–7.69); while the odds of good glycemic control increase by 17% with every 1-point increase in the resilience caregiver arm score (aOR: 1.17, 95% CI: 1.01–1.37). Conclusions: Efforts should be made toward implementing proactive measures aimed at enhancing income levels, coping strategies, and resilience among patients diagnosed with diabetes mellitus. Such initiatives are crucial for assisting individuals in adapting to the chronic nature of the condition and its associated treatment regimen, ultimately leading to improved glycemic control and treatment outcomes.

MSCMNet: Multi-scale Semantic Correlation Mining for Visible-Infrared Person Re-Identification

The main challenge in the Visible-Infrared Person Re-Identification (VI-ReID) task lies in extracting discriminative features from different modalities for matching purposes. While existing studies primarily focus on reducing modal discrepancies, the modality information fails to be thoroughly exploited. To solve this problem, the Multi-scale Semantic Correlation Mining network (MSCMNet) is proposed to comprehensively exploit semantic features at multiple scales. The network fuses shallow-level features into the deep network through dimensionality reduction and mapping, and the fused features are utilized to minimize modality information loss in feature extraction. Firstly, considering the effective utilization of modality information, the Multi-scale Information Correlation Mining Block (MIMB) is designed to fuse features at different scales and explore the semantic correlation of fusion features. Secondly, in order to enrich the semantic information that MIMB can utilize, the Quadruple-stream Feature Extractor (QFE) with non-shared parameters is specifically designed to extract information from different dimensions of the dataset. Finally, the Quadruple Center Triplet Loss (QCT) is further proposed to address the information discrepancy in the comprehensive features. Extensive experiments on the SYSU-MM01, RegDB, and LLCM datasets demonstrate that the proposed MSCMNet achieves the greatest accuracy. We release the source code on https://github.com/Hua-XC/MSCMNet.

Extragalactic globular cluster near-infrared spectroscopy

Context. One way to constrain the evolutionary histories of galaxies is to analyse their stellar populations. In the local Universe, our understanding of the stellar population properties of galaxies has traditionally relied on the study of optical absorption and emission-line features. Aims. In order to overcome limitations intrinsic to this wavelength range, such as the age-metallicity degeneracy and the high sensitivity to dust reddening, we must use wavelength ranges beyond the optical. The near-infrared (NIR) offers a possibility to extract information on spectral signatures that are not as obvious in traditional optical bands. Moreover, with the current and forthcoming generation of instrumentation focusing on the NIR, it is mandatory to explore possibilities within this wavelength range for nearby-Universe galaxies. However, although the NIR shows great potential, we are only beginning to understand it. Widely used techniques such as a full spectral fitting and line strength indices need to be tested on systems that are as close to simple stellar populations as possible, and the result from the techniques need to be compared to the yields from a traditional optical analysis. Methods. We present a NIR spectral survey of extragalactic globular clusters (GCs). The set was composed of 21 GCs from the Centaurus A galaxy that were obtained with SOAR/TripleSpec4, which covered the ∼1.0–2.4 μm range with a spectral resolution (R = λ/Δλ) of 3500. These spectra cover Hβ equivalent widths between 0.98 Å and 4.32 Å, and [MgFe]′ between 0.24 Å and 3.76 Å. Results. This set was ideal for performing absorption band measurements and a full spectral fitting, and it can be used for kinematic studies and age and abundance measurements. With this library, we expect to be able to probe the capabilities of NIR models, as well as to further improve stellar population estimates for the GCs around the Centaurus A galaxy.

Comparing Commercial and Open-Source Large Language Models for Labeling Chest Radiograph Reports.

Background Rapid advances in large language models (LLMs) have led to the development of numerous commercial and open-source models. While recent publications have explored OpenAI's GPT-4 to extract information of interest from radiology reports, there has not been a real-world comparison of GPT-4 to leading open-source models. Purpose To compare different leading open-source LLMs to GPT-4 on the task of extracting relevant findings from chest radiograph reports. Materials and Methods Two independent datasets of free-text radiology reports from chest radiograph examinations were used in this retrospective study performed between February 2, 2024, and February 14, 2024. The first dataset consisted of reports from the ImaGenome dataset, providing reference standard annotations from the MIMIC-CXR database acquired between 2011 and 2016. The second dataset consisted of randomly selected reports created at the Massachusetts General Hospital between July 2019 and July 2021. In both datasets, the commercial models GPT-3.5 Turbo and GPT-4 were compared with open-source models that included Mistral-7B and Mixtral-8 × 7B (Mistral AI), Llama 2-13B and Llama 2-70B (Meta), and Qwen1.5-72B (Alibaba Group), as well as CheXbert and CheXpert-labeler (Stanford ML Group), in their ability to accurately label the presence of multiple findings in radiograph text reports using zero-shot and few-shot prompting. The McNemar test was used to compare F1 scores between models. Results On the ImaGenome dataset (n = 450), the open-source model with the highest score, Llama 2-70B, achieved micro F1 scores of 0.97 and 0.97 for zero-shot and few-shot prompting, respectively, compared with the GPT-4 F1 scores of 0.98 and 0.98 (P > .99 and < .001 for superiority of GPT-4). On the institutional dataset (n = 500), the open-source model with the highest score, an ensemble model, achieved micro F1 scores of 0.96 and 0.97 for zero-shot and few-shot prompting, respectively, compared with the GPT-4 F1 scores of 0.98 and 0.97 (P < .001 and > .99 for superiority of GPT-4). Conclusion Although GPT-4 was superior to open-source models in zero-shot report labeling, few-shot prompting with a small number of example reports closely matched the performance of GPT-4. The benefit of few-shot prompting varied across datasets and models. © RSNA, 2024 Supplemental material is available for this article.