Radiology Reports Research Articles

Can AI generate diagnostic reports for radiologist approval on CXR images? Amulti-reader and multi-case observer performance study.

Accurately detecting a variety of lung abnormalities from heterogenous chest X-ray (CXR) images and writing radiology reports is often difficult and time-consuming. To access the utility of a novel artificial intelligence (AI) system (MOM-ClaSeg) in enhancing the accuracy and efficiency of radiologists in detecting heterogenous lung abnormalities through a multi-reader and multi-case (MRMC) observer performance study. Over 36,000 CXR images were retrospectively collected from 12 hospitals over 4 months and used as the experiment group and the control group. In the control group, a double reading method is used in which two radiologists interpret CXR to generate a final report, while in the experiment group, one radiologist generates the final reports based on AI-generated reports. Compared with double reading, the diagnostic accuracy and sensitivity of single reading with AI increases significantly by 1.49% and 10.95%, respectively (P < 0.001), while the difference in specificity is small (0.22%) and without statistical significance (P = 0.255). Additionally, the average image reading and diagnostic time in the experimental group is reduced by 54.70% (P < 0.001). This MRMC study demonstrates that MOM-ClaSeg can potentially serve as the first reader to generate the initial diagnostic reports, with a radiologist only reviewing and making minor modifications (if needed) to arrive at the final decision. It also shows that single reading with AI can achieve a higher diagnostic accuracy and efficiency than double reading.

Assessment of the Diagnostic Performance of a Commercially Available Artificial Intelligence Algorithm for Risk Stratification of Thyroid Nodules on Ultrasound.

Background: Thyroid nodules are challenging to accurately characterize on ultrasound (US), though the emergence of risk stratification systems and more recently artificial intelligence (AI) algorithms has improved nodule classification. The purpose of this study was to evaluate the performance of a recent Food and Drug Administration (FDA)-cleared AI tool for detection of malignancy in thyroid nodules on US. Methods: One year of consecutive thyroid US with ≥1 nodule from Duke University Hospital and its affiliate community hospital (649 nodules from 347 patients) were retrospectively evaluated. Included nodules had ground truth diagnoses by surgical pathology, fine needle aspiration (FNA), or three-year follow-up US showing stability. An FDA-cleared AI tool (Koios DS Thyroid) analyzed each nodule to generate (i) American College of Radiology Thyroid Imaging Reporting and Data System (ACR TI-RADS) descriptors, scores, and follow-up recommendations and (ii) an AI-adapter score to further adjust risk assessments and recommendations. Four groups were then compared: (i) Koios with AI-adapter, (ii) Koios without AI-adapter, (iii) clinical radiology report, and (iv) radiology report combined with AI-adapter. Performance of the final recommendations (FNA or no FNA) was determined based on ground truth, and comparison between the four groups was made using sensitivity, specificity, and receiver-operating-curve analysis. Results: Of 649 nodules, 32 were malignant and 617 were benign. Performance of Koios with AI-adapter enabled was similar to radiologists (area under the curve [AUC] 0.70 for both, [CI 0.60-0.81] and [0.60-0.79], respectively). Koios with AI-adapter had improved specificity compared to radiologists (0.63 [CI: 0.59-0.67] versus 0.43 [CI: 0.38-0.48]) but decreased sensitivity (0.69 [CI: 0.50-0.83) versus 0.81 [CI: 0.61, 0.92]). Highest performance was seen when the radiology interpretation was combined with the AI-adapter (AUC 0.76, [CI: 0.67-0.85]). Combined with the AI-adapter, radiologist specificity improved from 0.43 ([CI: 0.38-0.48]) to 0.53 ([CI: 0.49-0.58]) (McNemar's test p < 0.001), resulting in 17% fewer FNA recommendations, with unchanged sensitivity (0.81, p = 1). Conclusion: Koios DS demonstrated standalone performance similar to radiologists, though with lower sensitivity and higher specificity. Performance was best when radiologist interpretations were combined with the AI-adapter component, with improved specificity and reduced unnecessary FNA recommendations.

Differential diagnosis of congenital ventricular septal defect and atrial septal defect in children using deep learning–based analysis of chest radiographs

BackgroundChildren with atrial septal defect (ASD) and ventricular septal defect (VSD) are frequently examined for respiratory symptoms, even when the underlying disease is not found. Chest radiographs often serve as the primary imaging modality. It is crucial to differentiate between ASD and VSD due to their distinct treatment.PurposeTo assess whether deep learning analysis of chest radiographs can more effectively differentiate between ASD and VSD in children.MethodsIn this retrospective study, chest radiographs and corresponding radiology reports from 1,194 patients were analyzed. The cases were categorized into a training set and a validation set, comprising 480 cases of ASD and 480 cases of VSD, and a test set with 115 cases of ASD and 119 cases of VSD. Four deep learning network models—ResNet-CBAM, InceptionV3, EfficientNet, and ViT—were developed for training, and a fivefold cross-validation method was employed to optimize the models. Receiver operating characteristic (ROC) curve analyses were conducted to assess the performance of each model. The most effective algorithm was compared with the interpretations provided by two radiologists on 234 images from the test group.ResultsThe average accuracy, sensitivity, and specificity of the four deep learning models in the differential diagnosis of VSD and ASD were higher than 70%. The AUC values of ResNet-CBAM, IncepetionV3, EfficientNet, and ViT were 0.87, 0.91, 0.90, and 0.66, respectively. Statistical analysis showed that the differential diagnosis efficiency of InceptionV3 was the highest, reaching 87% classification accuracy. The accuracy of InceptionV3 in the differential diagnosis of VSD and ASD was higher than that of the radiologists.ConclusionsDeep learning methods such as IncepetionV3 based on chest radiographs in the study showed good performance for differential diagnosis of congenital VSD and ASD, which may be able to assist radiologists in diagnosis, education, and training, and reduce missed diagnosis and misdiagnosis.

PROLONGED SURVIVAL IN IDH-WILDTYPE GLIOBLASTOMA: CASE REPORTS, LITERATURE REVIEW AND CURRENT PERSPECTIVES OF MOLECULAR BEHAVIOR

Abstract AIMS Glioblastoma remains one of the most aggressive primary brain tumours, with a median survival of around 12 months despite advances in treatment. However, rare cases of long-term survival challenge this prognosis. The evaluation of age, tumour location, surgical approach and strategies, adjuvant therapy, and molecular behavior provides valuable prognostic and predictive information in patients with glioblastoma. METHOD We present a case report of two patients diagnosed with wild-type glioblastoma who have survived over 10 years post-primary surgery. A review of epidemiological data, preoperative and postoperative radiological reports, surgical approach, and histology reports were studied. Additionally, we review the current literature to explore current nuances in High-Grade Glioma Wild Type prognosis and potential future directions. Three major medical database engines, PubMed (Medline), Scopus (ELSEVIER), and Cochrane were used to conduct a thorough literature search. RESULTS Both patients underwent maximal safe resection followed by the Stupp Protocol, comprising adjuvant chemotherapy with temozolomide and radiotherapy. Our case report highlights two exceptional cases of prolonged survival in Glioblastoma patients, both under 51 years-old at the moment of diagnosis, early diagnosis with isolated lobar lesions, no compromise of deep white matter, and achieving maximal safe resection and functional independence. Molecular analysis revealed specific genetic in common, including IDH Wild Type, p53 mutation, retained ATRX expression, TERT mutation, and MGMT promoter methylation, consistent with a favourable prognosis. Moreover, patients without recurrence often exhibit distinct molecular profiles, such as MGMT promoter methylation, suggesting a potential subtype associated with prolonged survival. CONCLUSION Our case reports and literature review highlight the possibility of prolonged survival in Glioblastoma patients. The age, tumor location at the time of diagnosis, comprehensive treatment approaches, may contribute to improved outcomes. Molecular profiling, particularly assessing genetic mutations and MGMT promoter methylation, holds promise for identifying patients with a higher likelihood of prolonged survival.

Automatic structuring of radiology reports with on-premise open-source large language models.

Structured reporting enhances comparability, readability, and content detail. Large language models (LLMs) could convert free text into structured data without disrupting radiologists' reporting workflow. This study evaluated an on-premise, privacy-preserving LLM for automatically structuring free-text radiology reports. We developed an approach to controlling the LLM output, ensuring the validity and completeness of structured reports produced by a locally hosted Llama-2-70B-chat model. A dataset with de-identified narrative chest radiograph (CXR) reports was compiled retrospectively. It included 202 English reports from a publicly available MIMIC-CXR dataset and 197 German reports from our university hospital. Senior radiologist prepared a detailed, fully structured reporting template with 48 question-answer pairs. All reports were independently structured by the LLM and two human readers. Bayesian inference (Markov chain Monte Carlo sampling) was used to estimate the distributions of Matthews correlation coefficient (MCC), with [-0.05, 0.05] as the region of practical equivalence (ROPE). The LLM generated valid structured reports in all cases, achieving an average MCC of 0.75 (94% HDI: 0.70-0.80) and F1 score of 0.70 (0.70-0.80) for English, and 0.66 (0.62-0.70) and 0.68 (0.64-0.72) for German reports, respectively. The MCC differences between LLM and humans were within ROPE for both languages: 0.01 (-0.05 to 0.07), 0.01 (-0.05 to 0.07) for English, and -0.01 (-0.07 to 0.05), 0.00 (-0.06 to 0.06) for German, indicating approximately comparable performance. Locally hosted, open-source LLMs can automatically structure free-text radiology reports with approximately human accuracy. However, the understanding of semantics varied across languages and imaging findings. Question Why has structured reporting not been widely adopted in radiology despite clear benefits and how can we improve this? Findings A locally hosted large language model successfully structured narrative reports, showing variation between languages and findings. Critical relevance Structured reporting provides many benefits, but its integration into the clinical routine is limited. Automating the extraction of structured information from radiology reports enables the capture of structured data while allowing the radiologist to maintain their reporting workflow.

Boosting Deep Learning for Interpretable Brain MRI Lesion Detection through the Integration of Radiology Report Information.

Purpose To guide the attention of a deep learning (DL) model toward MRI characteristics of brain lesions by incorporating radiology report-derived textual features to achieve interpretable lesion detection. Materials and Methods In this retrospective study, 35 282 brain MRI scans (January 2018 to June 2023) and corresponding radiology reports from center 1 were used for training, validation, and internal testing. A total of 2655 brain MRI scans (January 2022 to December 2022) from centers 2-5 were reserved for external testing. Textual features were extracted from radiology reports to guide a DL model (ReportGuidedNet) focusing on lesion characteristics. Another DL model (PlainNet) without textual features was developed for comparative analysis. Both models identified 15 conditions, including 14 diseases and normal brains. Performance of each model was assessed by calculating macro-averaged area under the receiver operating characteristic curve (ma-AUC) and micro-averaged AUC (mi-AUC). Attention maps, which visualized model attention, were assessed with a five-point Likert scale. Results ReportGuidedNet outperformed PlainNet for all diagnoses on both internal (ma-AUC, 0.93 [95% CI: 0.91, 0.95] vs 0.85 [95% CI: 0.81, 0.88]; mi-AUC, 0.93 [95% CI: 0.90, 0.95] vs 0.89 [95% CI: 0.83, 0.92]) and external (ma-AUC, 0.91 [95% CI: 0.88, 0.93] vs 0.75 [95% CI: 0.72, 0.79]; mi-AUC, 0.90 [95% CI: 0.87, 0.92] vs 0.76 [95% CI: 0.72, 0.80]) testing sets. The performance difference between internal and external testing sets was smaller for ReportGuidedNet than for PlainNet (Δma-AUC, 0.03 vs 0.10; Δmi-AUC, 0.02 vs 0.13). The Likert scale score of ReportGuidedNet was higher than that of PlainNet (mean ± SD: 2.50 ± 1.09 vs 1.32 ± 1.20; P < .001). Conclusion The integration of radiology report textual features improved the ability of the DL model to detect brain lesions, thereby enhancing interpretability and generalizability. Keywords: Deep Learning, Computer-aided Diagnosis, Knowledge-driven Model, Radiology Report, Brain MRI Supplemental material is available for this article. Published under a CC BY 4.0 license.

Implementation of an AI Algorithm in Clinical Practice to Reduce Missed Incidental Pulmonary Embolisms on Chest CT and Its Impact on Short-Term Survival.

A substantial number of incidental pulmonary embolisms (iPEs) in computed tomography scans are missed by radiologists in their daily routine. This study analyzes the radiological reports of iPE cases before and after implementation of an artificial intelligence (AI) algorithm for iPE detection. Furthermore, we investigate the anatomic distribution patterns within missed iPE cases and mortality within a 90-day follow-up in patients before and after AI use. This institutional review board-approved observational single-center study included 5298 chest computed tomography scans performed for reasons other than suspected pulmonary embolism (PE). We compared 2 cohorts: cohort 1, consisting of 1964 patients whose original radiology reports were generated before the implementation of an AI algorithm, and cohort 2, consisting of 3334 patients whose scans were analyzed after the implementation of an Food and Drug Administration-approved and CE-certified AI algorithm for iPE detection (Aidoc Medical, Tel Aviv, Israel). For both cohorts, any discrepancies between the original radiology reports and the AI results were reviewed by 2 thoracic imaging subspecialized radiologists. In the original radiology report and in case of discrepancies with the AI algorithm, the expert review served as reference standard. Sensitivity, specificity, prevalence, negative predictive value (NPV), and positive predictive value (PPV) were calculated. The rates of missed iPEs in both cohorts were compared statistically using STATA (Version 17.1). Kaplan-Meier curves and Cox proportional hazards models were used for survival analysis. In cohort 1 (mean age 70.6 years, 48% female [n = 944], 52% male [n = 1020]), the prevalence of confirmed iPE was 2.2% (n = 42), and the AI detected 61 suspicious iPEs, resulting in a sensitivity of 95%, a specificity of 99%, a PPV of 69%, and an NPV of 99%. Radiologists missed 50% of iPE cases in cohort 1. In cohort 2 (mean age 69 years, 47% female [n = 1567], 53% male [n = 1767]), the prevalence of confirmed iPEs was 1.7% (56/3334), with AI detecting 59 suspicious cases (sensitivity 90%, specificity 99%, PPV 95%, NPV 99%). The rate of missed iPEs by radiologists dropped to 7.1% after AI implementation, showing a significant improvement (P < 0.001). Most overlooked iPEs (61%) were in the right lower lobe. The survival analysis showed no significantly decreased 90-day mortality rate, with a hazards ratio of 0.95 (95% confidence interval, 0.45-1.96; P = 0.88). The implementation of an AI algorithm significantly reduced the rate of missed iPEs from 50% to 7.1%, thereby enhancing diagnostic accuracy. Despite this improvement, the 90-day mortality rate remained unchanged. These findings highlight the AI tool's potential to assist radiologists in accurately identifying iPEs, although its implementation does not significantly affect short-term survival. Notably, most missed iPEs were located in the right lower lobe, suggesting that radiologists should pay particular attention to this area during evaluations.

Risk-benefit analysis of a multi-site radiographer comment model for emergency departments.

Critical/urgent X-ray findings are not always communicated in an appropriate time frame to ED physicians. The practice of radiographers alerting referrers to clinically significant image findings (verbally, via image flags or written comment) is noted internationally but risk assessment data is unavailable in the literature. A hybrid radiographer comment and alert model was piloted in New South Wales and a risk-benefit assessment conducted for timely and safe communication of abnormal X-ray appearances to ED physicians. Radiographer comments (n = 1102) were provided to five New South Wales EDs by 69 radiographers for a period of 3-12 months. Site auditors classified comments as true positive (TP), false positive (FP) or indeterminate (ID) with respect to the radiology report. FP comments were investigated with ED referrers and a low-medium-high-risk assessment was provided by two independent reviewers. A total of 42 FP (3.9%; 95% confidence interval [CI] 2.9-5.3) comments were analysed for any adverse outcomes. Risk assessments demonstrated 37 low, 5 low-moderate and no high-risk cases. A total of 282 direct or potential patient benefits were identified (26.4%; 95% CI 23.8-29.1). A total of 42 radiology report discrepancies were incidentally found: (3.9%; 95% CI 2.9-5.3). Audit results demonstrated areas where the radiographer comment could mitigate risk. The provision of radiographer alerts with a written comment for ED was found to be low risk to patients in the pilot study. Radiographers communicating directly with the emergency team when abnormal image appearances are detected can reduce diagnostic error and improve patient safety and health outcomes.

Text mining approach for feature extraction and cartilage disease grade classification using knee MRI radiology reports

MRI radiology reporting processes can be improved by exploiting structured and semantically labelled data that can be fed to artificial intelligence (AI) tools. AI-based tools assisting radiology reporting can help to automatically individuate cartilage grading in textual magnetic resonance imaging (MRI) reports, thus supporting clinicians' decisions regarding medical imaging utilisation, diagnosis and treatment. In this study, we extracted information (clinical findings, observations, anatomical regions, etc.) and classified knee cartilage degradation from medical reports utilising transfer-learning techniques applied to the Bidirectional Encoder Representations from Transformers (BERT) model and its variants, pre-trained on an Italian-language corpus. To realise this objective, we used a dataset of 750 MRI knee reports written by three radiologists who contributed to a manual annotation process to perform text classification (TC) and named entity recognition (NER) tasks. The dataset was obtained from an internal database of the IRCCS SYNLAB SDN. Seventy percent of the dataset was used for training, 10% was used for validation and 20% was used for testing. The best-performing configurations for NER and TC tasks were based on the pre-trained BERT model. The macro F1-scores obtained with the NER and TC models are 0.89 and 0.81, respectively. The accuracies calculated on the test set for both tasks are 0.96 and 0.99, respectively.

7949 Unraveling a Unique Presentation of Paget’s Disease of Bone

Abstract Disclosure: S. Puri: None. L. Bovee: None. R. Narla: None. Background: Paget's Disease of Bone (PDB) is a disorder characterized by rapid bone destruction and repair, which results in distortion of bone architecture. PDB occurs most commonly in adults over 55 and of European descent. PDB presents with both lytic and sclerotic lesions, predominantly in the axial skeleton. Serum alkaline phosphatase (ALP) levels and bone turnover markers (BTMs) are typically elevated in PDB. Clinical Case: A 53-year-old African-American woman with a history of breast cancer, PTSD, depression, and obesity (BMI 37) presented with worsening back pain over the past six months. Her pain would start at the center of her back and radiate to her left lower extremity. She endorsed rapid weight loss and night sweats and had lumbar bony tenderness on exam, prompting a spinal MRI. Spinal MRI was significant for diffuse sclerosis of the L3 vertebral body only. The radiology report provided a broad differential for this finding, including metastases, lymphoma, infection, PDB, and hemangioma. Due to progression of the patient’s back pain and concern for possible malignancy, a whole-body SPECT was completed which showed isolated tracer uptake into the L3 vertebral body extending into the pedicles and spinous process. Similar findings were noted on bone scans dating back to 2001. Per the radiology report, the presence of sclerosis with minimal expansion is most suggestive of PDB. Endocrinology was consulted for guidance on treatment and surveillance imaging for this patient. The endocrinology team discussed the case with radiology, who confirmed that the patient’s imaging findings had classic features of PDB. Bone specific ALP (B-ALP) and procollagen type 1 amino-terminal propeptide (P1NP) as bone turnover markers (BTMs) were within normal limits. A DEXA scan is pending to evaluate for concurrent osteoporosis. Clinical Lessons: PDB was favored as the most likely diagnosis primarily due to the sclerotic appearance of the patient’s vertebral lesion on imaging, radicular symptoms, and exam finding of bony point tenderness. Other diagnoses, such as metastatic disease, were considered and were thought to be unlikely given that the vertebral lesion is monostotic and stable. This case highlights the necessity of carefully evaluating radiographic findings and clinical history in the diagnosis of PDB. In this case, the patient was from a demographic group unlikely to have PDB and, unexpectedly, did not have elevated BTMs. It is important to consider PDB on the differential diagnosis in cases such as this because PDB can present in myriad ways and in a variety of patient populations. Presentation: 6/2/2024

Uterine Sarcoma or Degenerating Fibroid? Validating the New Consensus Magnetic Resonance Imaging Algorithm for Evaluating Atypical Uterine Masses.

The aim of the study is to assess the validity of a recently published consensus magnetic resonance imaging (MRI) diagnostic algorithm for differentiating degenerating leiomyomas from uterine sarcomas and other atypical appearing uterine malignancies. Atypical uterine masses on pelvic MRI were identified using a radiology report search engine and teaching files with the keywords "atypical leiomyoma," "atypical fibroid," and "sarcoma." All cases were pathology-proven. Two radiologists blinded to clinical, surgical, and pathologic reports retrospectively and independently reviewed 40 pelvic MRI examinations dated 1/2007-9/2022 to determine whether the masses appeared benign or malignant, using the 2022 consensus atypical uterine mass flow chart. Imaging features assessed included intermediate/high signal intensity (SI) at T2-weighted imaging, high diffusion weighted imaging SI (equal or higher SI than endometrium or lymph nodes on high b value imaging), apparent diffusion coefficient (ADC) value ≤0.905 × 10-3 mm2/s, peritoneal metastases, and abnormal lymph nodes. Among the 40 atypical uterine mass cases reviewed, 24 masses were benign (22 leiomyomas, 1 adenomyoma, and 1 borderline ovarian tumor) and 16 masses were malignant (6 leiomyosarcomas, 6 carcinosarcomas, 2 endometrial stromal sarcomas, 1 high-grade adenosarcoma, and 1 low-grade uterine sarcoma). Sensitivity, specificity, positive predictive value, and negative predictive value of whether a mass was benign or malignant were 75%, 95.8%, 92.3%, and 85% for reader 1, and 81.2%, 91.7%, 86.7%, and 88% for reader 2, respectively. Interrater agreement was strong, with a kappa statistic of 0.89. When excluding nonleiomyosarcoma uterine malignancies, sensitivity and negative predictive value improved to 100%. The new consensus pelvic MRI algorithm for evaluating atypical uterine masses has good specificity, sensitivity, positive predictive value, and negative predictive value for determining malignancy, particularly for uterine sarcomas that are leiomyosarcomas. However, if ADC value is near but not below 0.905 × 10-3 mm2/s, the mass may still be malignant, especially if a b value lower than 1000 is used. If the atypical uterine mass is predominantly endometrial, morphological features on T2 and postgadolinium sequences should guide suspicion, as some atypical appearing nonleiomyosarcoma uterine malignancies may have an ADC value greater than 0.905 × 10-3 mm2/s.

Incidental findings on MRI for the evaluation of endometriosis: prevalence and clinical significance.

This study aimed to analyze the prevalence and clinical significance of incidental findings on MRI for endometriosis. Differences between patients with and without evidence of deep infiltrating endometriosis on MRI were to be examined. This was a retrospective, descriptive cross-sectional single-center study. All patients who received a pelvic MRI for endometriosis between April 2021 and February 2023 were included. The presence and frequency of incidental findings were noted after review of all MR images and radiology reports. The potential clinical significance of the findings was analyzed. Differences in the frequency of incidental findings between patients with and without evidence of deep infiltrating endometriosis on MRI were evaluated, utilizing the Chi-square test, Fisher's exact test and Mann-Whitney U-test. 303 consecutive patients (mean age, 33.4 years ± 8.3) were evaluated. Incidental findings were noted in 299/303 (98.7%) patients. Most frequently, ossification of the hip acetabular rim and degenerative changes of the lumbar spine were noted. In 25/303 (8.3%) patients, incidental findings had high clinical significance. For specific incidental findings, significantly higher prevalences were found in patients with than in patients without evidence of deep infiltrating endometriosis on MRI (hip acetabular rim ossification, p = 0.041; annulus fibrosus fissures, p = 0.006; gallstones, p = 0.042). Incidental findings are very common on pelvic MRI for endometriosis. The detection of incidental findings can lead to the diagnosis of relevant diseases and thus enable early treatment. On the other hand, many incidental findings have no, only minor, or uncertain consequences.

Modeling disagreement in automatic data labeling for semi-supervised learning in Clinical Natural Language Processing.

Computational models providing accurate estimates of their uncertainty are crucial for risk management associated with decision-making in healthcare contexts. This is especially true since many state-of-the-art systems are trained using the data which have been labeled automatically (self-supervised mode) and tend to overfit. In this study, we investigate the quality of uncertainty estimates from a range of current state-of-the-art predictive models applied to the problem of observation detection in radiology reports. This problem remains understudied for Natural Language Processing in the healthcare domain. We demonstrate that Gaussian Processes (GPs) provide superior performance in quantifying the risks of three uncertainty labels based on the negative log predictive probability (NLPP) evaluation metric and mean maximum predicted confidence levels (MMPCL), whilst retaining strong predictive performance. Our conclusions highlight the utility of probabilistic models applied to "noisy" labels and that similar methods could provide utility for Natural Language Processing (NLP) based automated labeling tasks.

Streamlining Radiology Reporting: A Hands-Free Approach With Voice to Text and Generative Artificial Intelligence

ObjectiveIn radiology, report generation is critical for patient management, though can be time-consuming and labor-intensive. This study aims to investigate the feasibility of a hands-free radiology reporting system, leveraging voice-to-text technology and generative AI to improve time efficiency, report standardization, and reduce radiologist workload. Methods100 chest and musculoskeletal radiographs reports were generated using both traditional methods and an automated process using generative AI. To use generative AI, a radiologist dictated their findings into a voice-to-text software (Dragon), creating a raw text transcript. GPT-4 was then used to develop a report based on our institution’s template. Completion time for each traditional report and voice-to-text transcript was recorded. All reports were then evaluated for comprehensiveness, clarity, factual accuracy, and conciseness on a 5-point Likert scale. ResultsThe AI-driven system demonstrated considerable proficiency in transcribing and generating radiology reports, albeit with occasional inaccuracies. While it achieved a high level of precision in standard scenarios, its performance varied in more complex cases. Notably, the system significantly reduced the time taken to generate reports by 38% (p < 0.0001), signifying a potential increase in overall radiological workflow efficiency. ConclusionThis study highlights the potential of voice-to-text and generative AI integration in radiology reporting. Increased time efficiency, even in standard scenarios, can allow radiologists to focus more on complex cases and patient care. However, further development and refinement are required to address comprehensiveness and occasional inaccuracies. Ongoing advancements in AI and speech recognition technologies suggest a positive outlook for future implementations in medical reporting.

Enhancing Aortic Aneurysm Surveillance: Transformer Natural Language Processing for Flagging and Measuring in Radiology Reports

Incidental findings of aortic aneurysms (AAs) often go unreported, and established patients are frequently lost to follow-up. Natural language processing (NLP) offers a promising solution to address these issues. While rule-based NLP methods have shown some success, recent advancements in transformer-based large language models (LLMs) remain underutilized. This study has three aims: (1) to evaluate the effectiveness of our innovative transformer-based NLP pipeline regarding AA detection; (2) to detail the clinical impact by quantifying the number of patients who could benefit from such technology; and (3) to use this information to help coordinate appointments with patients, ensuring proper monitoring and management. 3229 radiology reports were divided into three batches with varying class balance. Each entry was processed through our innovative NLP pipeline, where it was fragmented using regular expression (regex) functions to isolate relevant textual segments. These segments were subsequently processed through our "question and find" (Q&F) function, powered by Google's BERT, a well-established transformer LLM. This Q&F function extracted aortic diameter measurements, flagging measurements that exceeded a predefined threshold. Following detection, we conducted comprehensive chart reviews and contacted primary care providers (PCPs) and patients to categorize aneurysms as "known" or "incidental." We also assessed whether patients with known aneurysms were adhering to regular yearly screenings and coordinated follow-up appointments. Evaluation of the three batches showed high F1 scores: 99.4% (95% CI [98.5-100]), 96.7% (95% CI [95.0-98.2]), and 98.9% (95% CI [98.0-99.6]). Overall measurement accuracy was 98.9% (95% CI [97.6-100]), 99.6% (95% CI [99.3-99.9]), and 98.1% (95% CI [96.8-99.4]). Compared to manual chart reviews, the NLP system demonstrated superior accuracy and fewer errors: 12 vs. 22 (p=0.084), 47 vs. 98 (p=0.000021), and 31 vs. 53 (p=0.015). Of the 412 patients investigated, 58 (14.1%) involved incidental findings, 54 patients (15.3%) were lost to follow-up, 39 patients (55.7%) were successfully contacted, and 37 follow-up appointments (12.1%) were successfully coordinated. The high-performance metrics from our study demonstrate that transformer-based NLP can enhance aortic aneurysm surveillance. Our subsequent comprehensive patient profiling highlighted the need for such a system as a safety net within the electronic medical record (EMR), systematically reviewing radiology reports to detect incidental findings and patients lost to follow-up. This ensures appropriate referrals and monitoring, improving patient outcomes and healthcare efficiency through timely clinical interventions.

MedSyn: Text-Guided Anatomy-Aware Synthesis of High-Fidelity 3-D CT Images.

This paper introduces an innovative methodology for producing high-quality 3D lung CT images guided by textual information. While diffusion-based generative models are increasingly used in medical imaging, current state-of-the-art approaches are limited to low-resolution outputs and underutilize radiology reports' abundant information. The radiology reports can enhance the generation process by providing additional guidance and offering fine-grained control over the synthesis of images. Nevertheless, expanding text-guided generation to high-resolution 3D images poses significant memory and anatomical detail-preserving challenges. Addressing the memory issue, we introduce a hierarchical scheme that uses a modified UNet architecture. We start by synthesizing low-resolution images conditioned on the text, serving as a foundation for subsequent generators for complete volumetric data. To ensure the anatomical plausibility of the generated samples, we provide further guidance by generating vascular, airway, and lobular segmentation masks in conjunction with the CT images. The model demonstrates the capability to use textual input and segmentation tasks to generate synthesized images. Algorithmic comparative assessments and blind evaluations conducted by 10 board-certified radiologists indicate that our approach exhibits superior performance compared to the most advanced models based on GAN and diffusion techniques, especially in accurately retaining crucial anatomical features such as fissure lines and airways. This innovation introduces novel possibilities. This study focuses on two main objectives: (1) the development of a method for creating images based on textual prompts and anatomical components, and (2) the capability to generate new images conditioning on anatomical elements. The advancements in image generation can be applied to enhance numerous downstream tasks.

Analysis of ChatGPT in the Triage of Common Spinal Complaints

ChatGPT is a natural language processing chatbot with a significant prevalence in modern media with a clear application in the medical triage workflow. ChatGPT has shown significant capacity for understanding clinical vignettes, radiology reports, and even passing the American Board of Neurological Surgery board examination. There has never been an evaluation of the chatbot in triage and diagnosing spinal vignettes common to primary and urgent care practice. Fifteen clinical scenarios were created to mimic spinal complaints common to primary and urgent care scenarios. GPT-4 was instructed to assess the situation as if it was in a primary care office and determine diagnosis, imaging recommendations, and if emergency room (ER) or operative referral was necessary. Answers were recorded and the results compared to those of attending and resident respondents. GPT-4 provided the most likely diagnosis in each scenario. Additionally, it recommended reasonable clinical management of each scenario which would fall within standard practice guidelines. ChatGPT tended toward over-referral to the ER; however, this was not significant. GPT-4 was noninferior in all categories when compared to respondents. ChatGPT is a powerful tool for primary triage of spinal issues. It can rapidly and accurately evaluate clinical scenarios and provide clear diagnostic reasoning. GPT-4 is not designed for medical use and will provide a disclaimer as such. It did tend toward over-referring patients to the ER. With specific training, it is likely that artificial intelligence and natural language processing chatbots will become widely used in primary triage of spinal issues.

Comparing Commercial and Open-Source Large Language Models for Labeling Chest Radiograph Reports.

Background Rapid advances in large language models (LLMs) have led to the development of numerous commercial and open-source models. While recent publications have explored OpenAI's GPT-4 to extract information of interest from radiology reports, there has not been a real-world comparison of GPT-4 to leading open-source models. Purpose To compare different leading open-source LLMs to GPT-4 on the task of extracting relevant findings from chest radiograph reports. Materials and Methods Two independent datasets of free-text radiology reports from chest radiograph examinations were used in this retrospective study performed between February 2, 2024, and February 14, 2024. The first dataset consisted of reports from the ImaGenome dataset, providing reference standard annotations from the MIMIC-CXR database acquired between 2011 and 2016. The second dataset consisted of randomly selected reports created at the Massachusetts General Hospital between July 2019 and July 2021. In both datasets, the commercial models GPT-3.5 Turbo and GPT-4 were compared with open-source models that included Mistral-7B and Mixtral-8 × 7B (Mistral AI), Llama 2-13B and Llama 2-70B (Meta), and Qwen1.5-72B (Alibaba Group), as well as CheXbert and CheXpert-labeler (Stanford ML Group), in their ability to accurately label the presence of multiple findings in radiograph text reports using zero-shot and few-shot prompting. The McNemar test was used to compare F1 scores between models. Results On the ImaGenome dataset (n = 450), the open-source model with the highest score, Llama 2-70B, achieved micro F1 scores of 0.97 and 0.97 for zero-shot and few-shot prompting, respectively, compared with the GPT-4 F1 scores of 0.98 and 0.98 (P > .99 and < .001 for superiority of GPT-4). On the institutional dataset (n = 500), the open-source model with the highest score, an ensemble model, achieved micro F1 scores of 0.96 and 0.97 for zero-shot and few-shot prompting, respectively, compared with the GPT-4 F1 scores of 0.98 and 0.97 (P < .001 and > .99 for superiority of GPT-4). Conclusion Although GPT-4 was superior to open-source models in zero-shot report labeling, few-shot prompting with a small number of example reports closely matched the performance of GPT-4. The benefit of few-shot prompting varied across datasets and models. © RSNA, 2024 Supplemental material is available for this article.

Leveraging artificial intelligence for timely detection and referral of incidental pulmonary nodules at a safety-net hospital.

422 Background: Incidental pulmonary nodules (IPNs) are commonly reported through a multitude of radiologic exams. A small number of these IPNs can harbor lung cancer (LC). To aide LC detection rates, hospitals must identify and validate the completion of appropriate follow-up regarding IPNs in a timely manner. The problem with timely referral and completion of work up is found to be amplified in safety-net settings with limited resources. John Peter Smith (JPS) hospital, a safety-net hospital, has developed a method to identify, track, and refer patients presenting with IPNs. Methods: JPS has created an algorithm to classify all IPNs by means of Thynk Health, an Artificial Intelligence (AI) platform that uses Natural Language Processing (NLP) to identify and track incidental findings and lung cancer screening patients (LCS). The algorithm created a scoring system by combining Fleischner Society (2017) and the American College of Radiology (ACR) LCS (Lung-RADS) guideline recommendations. The scoring system was designed to mimic Lung-RADS established by the ACR (1, 2, 3, 4A, and 4B). The assigned score creates a risk stratification to assist providers in the patient’s care plan. Using NLP, the project queried radiologic exams performed at JPS searching for key terminology relating to IPNs. Findings were then reviewed by radiology staff for validity and assigned a score based on the algorithm. Program exclusions, when identified in the radiology report, included: patient &lt; 35 years, patient has known cancer or under surveillance of oncology team, abnormalities favoring infectious process, LCS patients and when nodule size is not reported in the radiologist’s dictation. Results: A current state analysis was completed from 3/25/24 to 4/25/24. The AI platform analyzed 5,276 CT exams (2049 abdomen, 1347 head/neck, and 1880 chest) involving partial or complete imaging of the lungs. A total of 727 (13.8%) IPNs were flagged for review. After excluding exams falling outside of the set program parameters, a total of 178 (3.4%) IPNs were identified. 21 (11.8%) of the IPNs identified received a score of 4A or 4B. Patients found to have a score of 4A (11, 6.2%) and 4B (10, 5.6%) underwent additional review. Conclusions: Our team has created a unique system that can be used as a model for other large health systems. By leveraging technology, we are able to successfully identify key factors associated with IPNs to help ensure patients receive the recommended care in a timely manner. Results of JPS incidental pulmonary nodule program: first monthly analysis. JPS Pulmonary Nodule Score 4A 4B Number of Patients (% of IPNs) 11 (6.2%) 10 (5.6%) Median Age (years) 70 68.5 JPS Assigned PCP 1 7 (63.6%) 7 (70%) Median Smoking History (PY) 2 6.4 24.5 Average Nodule Size (mm) 11.5 34.3 Pulmonary Referral 3/11 (27.3%) 7/10 (70%) Positive for Cancer 0 3/4 (75%) 1 Primary care provider. 2 Pack years.

Evaluating the Performance and Bias of Natural Language Processing Tools in Labeling Chest Radiograph Reports.

Background Natural language processing (NLP) is commonly used to annotate radiology datasets for training deep learning (DL) models. However, the accuracy and potential biases of these NLP methods have not been thoroughly investigated, particularly across different demographic groups. Purpose To evaluate the accuracy and demographic bias of four NLP radiology report labeling tools on two chest radiograph datasets. Materials and Methods This retrospective study, performed between April 2022 and April 2024, evaluated chest radiograph report labeling using four NLP tools (CheXpert [rule-based], RadReportAnnotator [RRA; DL-based], OpenAI's GPT-4 [DL-based], cTAKES [hybrid]) on a subset of the Medical Information Mart for Intensive Care (MIMIC) chest radiograph dataset balanced for representation of age, sex, and race and ethnicity (n = 692) and the entire Indiana University (IU) chest radiograph dataset (n = 3665). Three board-certified radiologists annotated the chest radiograph reports for 14 thoracic disease labels. NLP tool performance was evaluated using several metrics, including accuracy and error rate. Bias was evaluated by comparing performance between demographic subgroups using the Pearson χ2 test. Results The IU dataset included 3665 patients (mean age, 49.7 years ± 17 [SD]; 1963 female), while the MIMIC dataset included 692 patients (mean age, 54.1 years ± 23.1; 357 female). All four NLP tools demonstrated high accuracy across findings in the IU and MIMIC datasets, as follows: CheXpert (92.6% [47 516 of 51 310], 90.2% [8742 of 9688]), RRA (82.9% [19 746 of 23 829], 92.2% [2870 of 3114]), GPT-4 (94.3% [45 586 of 48 342], 91.6% [6721 of 7336]), and cTAKES (84.7% [43 436 of 51 310], 88.7% [8597 of 9688]). RRA and cTAKES had higher accuracy (P < .001) on the MIMIC dataset, while CheXpert and GPT-4 had higher accuracy on the IU dataset. Differences (P < .001) in error rates were observed across age groups for all NLP tools except RRA on the MIMIC dataset, with the highest error rates for CheXpert, RRA, and cTAKES in patients older than 80 years (mean, 15.8% ± 5.0) and the highest error rate for GPT-4 in patients 60-80 years of age (8.3%). Conclusion Although commonly used NLP tools for chest radiograph report annotation are accurate when evaluating reports in aggregate, demographic subanalyses showed significant bias, with poorer performance in older patients. © RSNA, 2024 Supplemental material is available for this article. See also the editorial by Cai in this issue.