Radiology Reports Research Articles

Using natural language processing to identify emergency department patients with incidental lung nodules requiring follow-up.

For emergency department (ED) patients, lung cancer may be detected early through incidental lung nodules (ILNs) discovered on chest CTs. However, there are significant errors in the communication and follow-up of incidental findings on ED imaging, particularly due to unstructured radiology reports. Natural language processing (NLP) can aid in identifying ILNs requiring follow-up, potentially reducing errors from missed follow-up. We sought to develop an open-access, three-step NLP pipeline specifically for this purpose. This retrospective used a cohort of 26,545 chest CTs performed in three EDs from 2014 to 2021. Randomly selected chest CT reports were annotated by MD raters using Prodigy software to develop a stepwise NLP "pipeline" that first excluded prior or known malignancy, determined the presence of a lung nodule, and then categorized any recommended follow-up. NLP was developed using a RoBERTa large language model on the SpaCy platform and deployed as open-access software using Docker. After NLP development it was applied to 1000 CT reports that were manually reviewed to determine accuracy using accepted NLP metrics of precision (positive predictive value), recall (sensitivity), and F1 score (which balances precision and recall). Precision, recall, and F1 score were 0.85, 0.71, and 0.77, respectively, for malignancy; 0.87, 0.83, and 0.85 for nodule; and 0.82, 0.90, and 0.85 for follow-up. Overall accuracy for follow-up in the absence of malignancy with a nodule present was 93.3%. The overall recommended follow-up rate was 12.4%, with 10.1% of patients having evidence of known or prior malignancy. We developed an accurate, open-access pipeline to identify ILNs with recommended follow-up on ED chest CTs. While the prevalence of recommended follow-up is lower than some prior studies, it more accurately reflects the prevalence of truly incidental findings without prior or known malignancy. Incorporating this tool could reduce errors by improving the identification, communication, and tracking of ILNs.

CXR-LLaVA: a multimodal large language model for interpreting chest X-ray images.

This study aimed to develop an open-source multimodal large language model (CXR-LLaVA) for interpreting chest X-ray images (CXRs), leveraging recent advances in large language models (LLMs) to potentially replicate the image interpretation skills of human radiologists. For training, we collected 592,580 publicly available CXRs, of which 374,881 had labels for certain radiographic abnormalities (Dataset 1) and 217,699 provided free-text radiology reports (Dataset 2). After pre-training a vision transformer with Dataset 1, we integrated it with an LLM influenced by the LLaVA network. Then, the model was fine-tuned, primarily using Dataset 2. The model's diagnostic performance for major pathological findings was evaluated, along with the acceptability of radiologic reports by human radiologists, to gauge its potential for autonomous reporting. The model demonstrated impressive performance in test sets, achieving an average F1 score of 0.81 for six major pathological findings in the MIMIC internal test set and 0.56 for six major pathological findings in the external test set. The model's F1 scores surpassed those of GPT-4-vision and Gemini-Pro-Vision in both test sets. In human radiologist evaluations of the external test set, the model achieved a 72.7% success rate in autonomous reporting, slightly below the 84.0% rate of ground truth reports. This study highlights the significant potential of multimodal LLMs for CXR interpretation, while also acknowledging the performance limitations. Despite these challenges, we believe that making our model open-source will catalyze further research, expanding its effectiveness and applicability in various clinical contexts. Question How can a multimodal large language model be adapted to interpret chest X-rays and generate radiologic reports? Findings The developed CXR-LLaVA model effectively detects major pathological findings in chest X-rays and generates radiologic reports with a higher accuracy compared to general-purpose models. Clinical relevance This study demonstrates the potential of multimodal large language models to support radiologists by autonomously generating chest X-ray reports, potentially reducing diagnostic workloads and improving radiologist efficiency.

Cardiac Computed Tomography Measurements in Pulmonary Embolism Associated with Clinical Deterioration

Introduction: Most pulmonary embolism response teams (PERT) use a radiologist-determined right ventricle to left ventricle ratio (RV:LV) cut-off of 1.0 to risk-stratify pulmonary embolism (PE) patients. Continuous measurements from computed tomography pulmonary angiograms (CTPAs) may improve risk stratification. We assessed associations of CTPA cardiac measurements with acute clinical deterioration and use of advanced PE interventions. Methods: This was a retrospective study of a PE registry used by eight affiliated emergency departments. We used an artificial intelligence (AI) algorithm to measure RV:LV on anonymized CTPAs from registry patients for whom the PERT was activated (2018–2023) by institutional guidelines. Primary outcome was in-hospital PE-related clinical deterioration defined as cardiac arrest, vasoactive medication use for hypotension, or rescue respiratory interventions. Secondary outcome was advanced intervention use. We used bivariable and multivariable analyses. For the latter, we used least absolute shrinkage and selection operator (LASSO) and random forest (RF) to determine associations of all candidate variables with the primary outcome (clinical deterioration), and the Youden index to determine RV:LV optimal cut-offs for primary outcome. Results: Artificial intelligence analyzed 1,467 CTPAs, with 88% agreement on RV:LV categorization with radiologist reports (kappa 0.36, 95% confidence interval [CI] 0.28–0.43). Of 1,639 patients, 190 (11.6%) had PE-related clinical deterioration, and 314 (19.2%) had advanced interventions. Mean RV:LV were 1.50 (0.39) vs 1.30 (0.32) for those with and without clinical deterioration and 1.62 (0.33) vs 1.35 (0.32) for those with and without advanced intervention use. The RV:LV cut-off of 1.0 by AI and radiologists had 0.02 and 0.53 P-values for clinical deterioration, respectively. With adjusted LASSO, top clinical deterioration predictors were cardiac arrest at presentation, lowest systolic blood pressure, and intensive care unit admission. The RV:LV measurement was a top 10 predictor of clinical deterioration by RF. Optimal cut-off for RV:LV was 1.54 with odds ratio of 2.50 (1.85, 3.45) and area under the curve 0.6 (0.66, 0.70). Conclusion: Artifical intelligence-derived RV:LV measurements ≥1.5 on initial CTPA had strong associations with in-hospital clinical deterioration and advanced interventions in a large PERT database. This study points to the potential of capitalizing on immediately available CTPA RV:LV measurements for gauging PE severity and risk stratification.

Digital Mammography, Tomosynthesis, and Contrast-Enhanced Mammography: Intraindividual Comparison of Mean Glandular Dose for Screening Examinations.

Background: Contrast-enhanced mammography (CEM) is growing in clinical use due to its increased sensitivity and specificity compared to full-field digital mammography (FFDM) and/or digital breast tomosynthesis (DBT), particularly in patients with dense breasts. Objective: To perform an intraindividual comparison of MGD between FFDM, DBT, a combination protocol using both FFDM and DBT (combined FFDM-DBT), and CEM, in patients undergoing breast cancer screening. Methods: This retrospective study included 389 women (median age, 57.4 years) at elevated risk of breast cancer who, as part of participation in an earlier prospective clinical trial, underwent breast cancer screening by combined FFDM-DBT and CEM between February 2019 and April 2021. A total of 764 breasts (383 left, 381 right) were evaluated. One craniocaudal (CC) view and one mediolateral oblique (MLO) view were evaluated per breast for each of FFDM, DBT, and CEM. MGD values were extracted from DICOM metadata. BI-RADS breast density categories were extracted from clinical radiology reports. Data were summarized descriptively, including determination of corresponding effective doses. Results: Breast density category was A in no patients, B in 44 patients (88 breasts), C in 306 patients (599 breasts), and D in 39 patients (77 breasts). Median MGD per breast (CC and MLO views combined) was 4.07 mGy for FFDM alone, 4.97 mGy for DBT alone, 9.38 mGy for combined FFDM-DBT, 3.96 mGy for low-energy CEM, 1.90 mGy for high-energy CEM, and 5.87 for CEM overall. Corresponding effective dose values were 0.49 mSv, 0.60 mSv, 1.13 mSv, 0.48 mSv, 0.23 mSv, and 0.70 mSv, respectively. Median MGD for density categories B, C, and D was 4.01 mGy, 4.22 mGy, and 2.70 mGy for FFDM; 5.93 mGy, 4.93 mGy, and 3.17 mGy for DBT; and 5.90 mGy, 6.02 mGy, and 4.52 mGy for CEM. Conclusion: In this intraindividual comparative study of screening examinations, MGD per breast was higher for CEM than for FFDM or DBT alone. However, these differences were small, and MGD was lower for CEM than for combined FFDM-DBT. Clinical Impact: These findings are relevant to ongoing considerations of the role of CEM in breast cancer screening pathways.

Characterization of adrenal glands on computed tomography with a 3D V-Net-based model

ObjectivesTo evaluate the performance of a 3D V-Net-based segmentation model of adrenal lesions in characterizing adrenal glands as normal or abnormal.MethodsA total of 1086 CT image series with focal adrenal lesions were retrospectively collected, annotated, and used for the training of the adrenal lesion segmentation model. The dice similarity coefficient (DSC) of the test set was used to evaluate the segmentation performance. The other cohort, consisting of 959 patients with pathologically confirmed adrenal lesions (external validation dataset 1), was included for validation of the classification performance of this model. Then, another consecutive cohort of patients with a history of malignancy (N = 479) was used for validation in the screening population (external validation dataset 2). Parameters of sensitivity, accuracy, etc., were used, and the performance of the model was compared to the radiology report in these validation scenes.ResultsThe DSC of the test set of the segmentation model was 0.900 (0.810–0.965) (median (interquartile range)). The model showed sensitivities and accuracies of 99.7%, 98.3% and 87.2%, 62.2% in external validation datasets 1 and 2, respectively. It showed no significant difference comparing to radiology reports in external validation datasets 1 and lesion-containing groups of external validation datasets 2 (p = 1.000 and p > 0.05, respectively).ConclusionThe 3D V-Net-based segmentation model of adrenal lesions can be used for the binary classification of adrenal glands.Critical relevance statementA 3D V-Net-based segmentation model of adrenal lesions can be used for the detection of abnormalities of adrenal glands, with a high accuracy in the pre-surgical scene as well as a high sensitivity in the screening scene.Key PointsAdrenal lesions may be prone to inter-observer variability in routine diagnostic workflow.The study developed a 3D V-Net-based segmentation model of adrenal lesions with DSC 0.900 in the test set.The model showed high sensitivity and accuracy of abnormalities detection in different scenes.Graphical

Leveraging Artificial Intelligence as a Safety Net for Incidentally Identified Lung Nodules at a Tertiary Center.

Artificial intelligence (AI)-powered platforms may be used to ensure that clinically significant lung nodules receive appropriate management. We studied the impact of a commercially available AI natural language processing tool on detection of clinically significant indeterminate pulmonary nodules (IPNs) based on radiology reports and provision of guideline-consistent care. All computed tomography (CT) scans performed at a single tertiary care center in the outpatient or emergency room setting between 20-Feb-2024 and 20-March-2024 were processed by the AI natural language processing algorithm. CT radiology reports mentioning a lung nodule or focal indeterminate lesion were flagged. All flagged reports were reviewed by a lung nodule expert two weeks after nodule identification. IPNs were classified as "appropriately followed" if follow-up imaging, referral to a nodule clinic, or other guideline-consistent care was ordered. IPNs were classified as "not appropriately followed" if no acknowledgement of the reported nodule was documented in the electronic health record within two weeks of being flagged. The AI software processed 76,507 unique radiology reports, identified 2,585 CT scans with chest imaging, and found 389 IPNs. Review determined that 272 (70%) nodules were appropriately followed while 117 (30%) were not appropriately followed. Of the 117 nodules without documented follow-up, 67 (57%) were > 8mm and 24 (20.5%) were > 15mm. IPNs that would not have received follow-up in the absence of the AI software generated 43 additional clinical appointments and 3 procedures. At a large tertiary care center, 30% of clinically significant incidental pulmonary nodules that would have otherwise been missed were brought to the attention of lung nodule clinicians by an AI software, allowing for initiation of appropriate follow-up.

Scaphoid Fat Stripe Sign: Is It a Reliable Radiological Sign of Scaphoid Fracture in Children?

Objectives: The scaphoid fat pad stripe (SFS) is a radiological sign first described in 1975 as a line of relative lucency lying parallel to the lateral border of the scaphoid, with slight convexity toward it, and it is optimally demonstrated on postero-anterior and oblique views with ulnar deviation of the carpus. The obliteration or displacement of this line is commonly present in acute fractures of the scaphoid, radial styloid process, and proximal first metacarpus. The aim of this observational study is to investigate the supportive value of the fat stripe sign (SFS) in the diagnosis of scaphoid fractures in the pediatric population. Methods: This is a monocentric, retrospective study of all patients referred to the Pediatric Traumatology Unit of a tertiary hospital from the Emergency Department with clinical suspicion of scaphoid fracture without visible fracture in the initial X-ray. Radiological reports for CT and MRIs were recorded, and the initial X-rays were blindly reviewed by a pediatric orthopedic fellowship-accredited surgeon for the presence of an abnormal scaphoid fat pad stripe sign and the presence of a fracture line in the initial X-rays. Results: The results of the blind interpretation of the initial X-rays for the fat stripe sign showed 86% sensitivity and 58% specificity, with the negative predictive value reaching 92%. Conclusions: The scaphoid fat stripe sign can be used as an adjacent in the diagnosis of an occult scaphoid fracture in children or adolescents. Its high negative predictive value, if confirmed in larger studies, can be an element used to exclude scaphoid fracture and consequently avoid unnecessary immobilizations and health costs.

Automated Incidental Findings Notification Through the Electronic Health Record Utilizing Dictation Macros.

The objective of this study is to implement an actionable incidental findings (AIFs) communication workflow integrated into the electronic health record (EHR) using dictation macros to improve the quality of radiology reports and facilitate delivery of findings to clinicians. The workflow was implemented across an academic multi-hospital health system and used by over 100 radiologists from 12 divisions. Standardized macros were created for different organ systems including the thyroid, lungs, liver, pancreas, spleen, kidney, female reproductive, and others, designed based on the ACR Novel Quality Measure Set. All macros contained special codes enabling automated notification of clinicians in Epic EHR and unique codes to allow for tracking. When notified, clinicians can fast track ordering of follow-up imaging exams. All alerts were monitored by radiology operations who ensured messages were acknowledged within 73h. From September 2023 to March 2024, 12,919 AIFs alerts were filed for 10,766 patients. Median age was 65years, and 63.6% were female and 36.4% were male. Most alerts were submitted for outpatients (73.5%), and a majority originated from CT exams (57.3%) followed by radiographs (12.2%) and ultrasound (11.5%). Number of submissions per radiologist ranged from 0 to 930 with a median of 62. Median time to alert acknowledgment was 8.1h, and 93.9% were acknowledged within 73h. Follow-up orders were placed for 62.3% of patients. A standardized AIFs communication workflow utilizing dictation macros can help facilitate delivery of findings and follow-up recommendations to clinicians.

Ovarian cancer staging and follow-up: updated guidelines from the European Society of Urogenital Radiology female pelvic imaging working group.

To provide up-to-date European Society of Urogenital Radiology (ESUR) guidelines for staging and follow-up of patients with ovarian cancer (OC). Twenty-one experts, members of the female pelvis imaging ESUR subcommittee from 19 institutions, replied to 2 rounds of questionnaires regarding imaging techniques and structured reporting used for pre-treatment evaluation of OC patients. The results of the survey were presented to the other authors during the group's annual meeting. The lexicon was aligned with the Society of American Radiology (SAR)-ESUR lexicon; a first draft was circulated, and then comments and suggestions from the other authors were incorporated. Evaluation of disease extent at diagnosis should be performed by chest, abdominal, and pelvic CT. The radiological report should map the disease with specific mention of sites that may preclude optimal cytoreductive surgery. For suspected recurrence, CT and [18F]FDG PET-CT are both valid options. MRI can be considered in experienced centres, as an alternative to CT, considering the high costs and the need for higher expertise in reporting. CT is the imaging modality of choice for preoperative evaluation and follow-up in OC patients. A structured radiological report, including specific mention of sites that may preclude optimal debulking, is of value for patient management. Question Guidelines were last published for ovarian cancer (OC) imaging in 2010; here, guidance on imaging techniques and reporting, incorporating advances in the field, are provided. Findings Structured reports should map out sites of disease, highlighting sites that limit cytoreduction. For suspected recurrence, CT and 18FDG PET-CT are options, and MRI can be considered. Clinical relevance Imaging evaluation of OC patients at initial diagnosis (mainly based on CT), using a structured report that considers surgical needs is valuable in treatment selection and planning.

AUTOMATED OPPORTUNISTIC CT-SCREENING OF ABDOMINAL AORTIC ANEURYSMS USING ARTIFICIAL INTELLIGENCE: PROSPECTS AND CHALLENGES (LITERATURE REVIEW)

Highlights Non-contrast computed tomography (CT) scan is a promising modality for opportunistic screening of abdominal aortic aneurysm (AAA).Automation of opportunistic screening of AAA according to CT data is a promising use of artificial intelligence (AI) technologies.The development of AI algorithms for opportunistic screening of AAA based on CT data is currently limited due to the high labor costs in preparing datasets for AI training and testing. AnnotationAbdominal aortic aneurysm is a cardiovascular disease characterized by a latent progression and adverse prognosis. Timely diagnosis reduces surgical risks and postoperative complications. Diagnostic imaging methods used to detect and evaluate this disease, particularly in targeted and opportunistic screening, are reviewed. The prospects of automation using artificial intelligence technologies for opportunistic screening are explored, moreover, they have already proven to be effective tools for optimizing radiology reports in several fields. This review highlights, however, relatively poor development of artificial intelligence algorithms for opportunistic screening of abdominal aortic aneurysms on native non-contrast abdominal CT studies. Possible reasons for this phenomenon and potential ways of development of this subject area are investigated.

Do reporting guidelines improve the efficiency and quality of radiological case reports writing? A study protocol

Background Radiological case reports summarize the imaging characteristics of individual cases to identify patterns and valuable lessons. Their writing can be guided by the CARE reporting guidelines or the CARE-radiology reporting guidelines. However, the impact of following these guidelines, or not following them, on the quality and efficiency of radiological case reports writing remain unclear. Objective To examine whether following the CARE reporting guidelines or the CARE-radiology reporting guidelines, or not following any reporting guidelines, affects the quality and writing efficiency of radiological case reports. Protocol The purpose of this protocol is to outline the background and objectives of this study, as well as the design of a cluster randomized controlled trial (cRCT). We will design a three-arm, multicentre cRCT, targeting radiologists or radiology students from different medical institutions. The intervention include training on the CARE-radiology checklist; the second intervention involves training on the CARE checklist. The control group will follow standard procedures without any intervention. After the intervention, participants will be required to write a case report within a period of 10 days. The primary outcome measures will be the completeness and clarity of the case reports. Discussion This study aims to enhance the adherence of researchers to reporting guidelines for case reports through rigorous design. The results will be further explained and interpreted in the final trial outcomes. Registration This study has been registered with the Chinese Clinical Trial Registry, with the registration number: ChiCTR2400089973.

Outcome of follow-up computed tomography of suspected occult scaphoid fracture after normal radiography.

To evaluate the rate of missed scaphoid fractures on follow-up computed tomography (CT) for suspected occult scaphoid fracture after normal radiography with residual radial-sided wrist pain. In a retrospective analysis, wrist CT during a five-year period was analyzed. The CT examinations and radiological reports were re-evaluated. Available clinical findings and radiologic follow-up performed during a period of a minimum of three years served as outcome reference. In total, 178 examinations had been performed on 174 patients for suspect scaphoid fracture, 67 men and 107 women, showing 15 and 6 scaphoid fractures, respectively; a statistically significant sex difference (p = 0.0024). In 157 examinations, no scaphoid fracture was detected on CT, instead 29 other wrist or carpal bone fractures were found. On follow-up, no missed scaphoid fractures were found. Before CT, 124 of the 157 patients had been treated with a cast. After CT, 35 patients continued with cast treatment for a median of 14days. CT appears to be a reliable method for evaluating suspect scaphoid fracture as part of a diagnosis-treatment regimen including pain immobilization with a plaster cast.

Implementation of O-RADS Ultrasound Reporting System: A Quality Improvement Initiative.

Objectives: To determine the feasibility of implementing Ovarian-Adnexal Reporting & Data System (O-RADS) ultrasound (US) for reporting of adnexal masses at our institution, with a specific goal of increasing the use of O-RADS from a baseline of <5% to at least 75% over a 16-month period. Methods: A prospective interrupted time series quality improvement study was undertaken over a 16-month period. Plan, do, study, act cycles included: (1) Engagement of interested parties, (2) Targeted educational sessions, (3) Development of reporting templates, (4) Weekly audit-feedback. Inter-reader variability assessment was performed on 70% of O-RADS risk-category 2 to 5. The primary outcome was the reporting of an O-RADS risk category. Results: A total of 635 female pelvic US were performed at our centre between July 2022 and April 2023. An O-RADS risk category was provided on the final radiology report by the radiologist for 489/635 (77%) US. From November 2022 to April 2023, the weekly rate of O-RADS risk category reporting reached 88%. The O-RADS score was concordant between readers for 83/103 (81%) of US reports with kappa score of 0.69 corresponding to good agreement. Conclusions: The reporting of O-RADS risk category increased from <5% to 88% over a 16-month period with a high level of agreement among readers in assigning O-RADS risk category. Implementation of a standardizing reporting ultrasound system at a tertiary cancer centre is feasible with rapid learning and uptake curves.

Editorial Comment: Focused and Structured Radiology Reporting With Multidisciplinary Approach.

Editorial Comment: Focused and Structured Radiology Reporting With Multidisciplinary Approach.

Current Status of CT Imaging Before Common Transcatheter Interventions for Structural Heart Disease.

Background: Over the past decade, several trials and observational studies have validated the use of minimally invasive cardiac interventions as viable treatment options for various cardiac diseases. Transcatheter techniques for severe aortic valve stenosis have rapidly emerged as alternatives to surgical aortic valve replacement in certain patient populations. Additionally, non-surgical treatment options have expanded for conditions affecting other cardiac valves, such as the mitral valve. These emerging minimally invasive interventions complement already well-established endovascular techniques for, among others, atrial septal defect closure, left atrial appendage occlusion and pulmonary vein isolation in patients with atrial fibrillation. Given their non-surgical nature and lack of direct visualisation of the targeted anatomy, these procedures heavily rely on precise pre-procedural radiological imaging for optimal patient selection and procedural success. Method: This paper is based on the expert opinion of the authors and an exhaustive literature research. Results: This manuscript reviews the most commonly employed minimally invasive cardiac interventions, highlighting the essential pre-procedural imaging information and key aspects that must be included in radiological reports to mitigate potential complications. Conclusion: Accurate pre-procedural imaging is crucial for ensuring safe and effective minimally invasive cardiac interventions, underscoring the importance of the radiologist in the pre-procedural work-up of these patients.

Multilingual feasibility of GPT-4o for automated Voice-to-Text CT and MRI report transcription.

Large language models (LLMs) promise to streamline radiology reporting. With the release of OpenAI's GPT-4o (Generative Pre-trained Transformers-4 omni), which processes not only text but also speech, multimodal LLMs might now also be used as medical speech recognition software for radiology reporting in multiple languages. This proof-of-concept study investigates the feasibility of using GPT-4o for automated voice-to-text transcription of radiology reports in English and German. Three readers with varying levels of experience each dictated 100 synthetic radiology reports in both languages using GPT-4o via the ChatGPT iOS mobile application. Reports included CT and MRI scans of various anatomical regions. Evaluation metrics included error type, severity, and correction time. BERTScore and ROUGE metrics were calculated to assess semantic similarity and n-gram overlap between dictated and original reports. No significant differences in correction time between languages were found, but differences were observed between readers based on experience. Error rates were similar for both languages, with most errors being minor (92.68%, n=114/123 German; 94.74%, n=90/95 English) and technical (27.04%, n=43/159 German; 35.65%, n=41/115 English) or typographical (23.9%, n=38/159 German; 27.83%, n=32/115 English). BERTScore metrics were significantly higher for German, while ROUGE metrics showed no significant differences between languages. This study demonstrates the potential of GPT-4o for multilingual transcription of radiology reports, effectively handling both English and German with minimal errors and high semantic understanding. Future research should compare GPT-4o with current radiology dictation tools, assessing performance, cost-effectiveness, and multilingual capabilities across diverse speaker populations.

Generating synthetic clinical text with local large language models to identify misdiagnosed limb fractures in radiology reports.

Large language models (LLMs) demonstrate impressive capabilities in generating human-like content and have much potential to improve the performance and efficiency of healthcare. An important application of LLMs is to generate synthetic clinical reports that could alleviate the burden of annotating and collecting real-world data in training AI models. Meanwhile, there could be concerns and limitations in using commercial LLMs to handle sensitive clinical data. In this study, we examined the use of open-source LLMs as an alternative to generate synthetic radiology reports to supplement real-world annotated data. We found LLMs hosted locally can achieve similar performance compared to ChatGPT and GPT-4 in augmenting training data for the downstream report classification task of identifying misdiagnosed fractures. We also examined the predictive value of using synthetic reports alone for training downstream models, where our best setting achieved more than 90% of the performance using real-world data. Overall, our findings show that open-source, local LLMs can be a favourable option for creating synthetic clinical reports for downstream tasks.

Privacy-ensuring Open-weights Large Language Models Are Competitive with Closed-weights GPT-4o in Extracting Chest Radiography Findings from Free-Text Reports.

Background Large-scale secondary use of clinical databases requires automated tools for retrospective extraction of structured content from free-text radiology reports. Purpose To share data and insights on the application of privacy-preserving open-weights large language models (LLMs) for reporting content extraction with comparison to standard rule-based systems and the closed-weights LLMs from OpenAI. Materials and Methods In this retrospective exploratory study conducted between May 2024 and September 2024, zero-shot prompting of 17 open-weights LLMs was preformed. These LLMs with model weights released under open licenses were compared with rule-based annotation and with OpenAI's GPT-4o, GPT-4o-mini, GPT-4-turbo, and GPT-3.5-turbo on a manually annotated public English chest radiography dataset (Indiana University, 3927 patients and reports). An annotated nonpublic German chest radiography dataset (18 500 reports, 16 844 patients [10 340 male; mean age, 62.6 years ± 21.5 {SD}]) was used to compare local fine-tuning of all open-weights LLMs via low-rank adaptation and 4-bit quantization to bidirectional encoder representations from transformers (BERT) with different subsets of reports (from 10 to 14 580). Nonoverlapping 95% CIs of macro-averaged F1 scores were defined as relevant differences. Results For the English reports, the highest zero-shot macro-averaged F1 score was observed for GPT-4o (92.4% [95% CI: 87.9, 95.9]); GPT-4o outperformed the rule-based CheXpert [Stanford University] (73.1% [95% CI: 65.1, 79.7]) but was comparable in performance to several open-weights LLMs (top three: Mistral-Large [Mistral AI], 92.6% [95% CI: 88.2, 96.0]; Llama-3.1-70b [Meta AI], 92.2% [95% CI: 87.1, 95.8]; and Llama-3.1-405b [Meta AI]: 90.3% [95% CI: 84.6, 94.5]). For the German reports, Mistral-Large (91.6% [95% CI: 90.5, 92.7]) had the highest zero-shot macro-averaged F1 score among the six other open-weights LLMs and outperformed the rule-based annotation (74.8% [95% CI: 73.3, 76.1]). Using 1000 reports for fine-tuning, all LLMs (top three: Mistral-Large, 94.3% [95% CI: 93.5, 95.2]; OpenBioLLM-70b [Saama]: 93.9% [95% CI: 92.9, 94.8]; and Mixtral-8×22b [Mistral AI]: 93.8% [95% CI: 92.8, 94.7]) achieved significantly higher macro-averaged F1 score than did BERT (86.7% [95% CI: 85.0, 88.3]); however, the differences were not relevant when 2000 or more reports were used for fine-tuning. Conclusion LLMs have the potential to outperform rule-based systems for zero-shot "out-of-the-box" structuring of report databases, with privacy-ensuring open-weights LLMs being competitive with closed-weights GPT-4o. Additionally, the open-weights LLM outperformed BERT when moderate numbers of reports were used for fine-tuning. Published under a CC BY 4.0 license. Supplemental material is available for this article. See also the editorial by Gee and Yao in this issue.

Recalibrated cross-modal alignment network for radiology report generation with weakly supervised contrastive learning

Recalibrated cross-modal alignment network for radiology report generation with weakly supervised contrastive learning

SumGPT: A Novel Multimodal Framework for Radiology Report Summarization to Improve Clinical Performance

SumGPT: A Novel Multimodal Framework for Radiology Report Summarization to Improve Clinical Performance