Findings In Radiology Reports Research Articles

Use of GPT-4 With Single-Shot Learning to Identify Incidental Findings in Radiology Reports.

Use of GPT-4 With Single-Shot Learning to Identify Incidental Findings in Radiology Reports.

Potential Use Cases for ChatGPT in Radiology Reporting.

Large language models (LLMs) such as ChatGPT are advanced artificial intelligence (AI) models that are designed to process and understand human language. LLMs have the potential to improve radiology reporting and patient engagement by automating generation of the clinical history and impression of a radiology report, creating layperson reports, and providing patients pertinent questions and answers about findings in radiology reports. However, LLMs are errors-prone, and human oversight is needed to reduce the risk of patient harm.

From Softmax to Nucleusmax: A Novel Sparse Language Model for Chinese Radiology Report Summarization

The Chinese radiology report summarization is a crucial component in smart healthcare that employs language models to summarize key findings in radiology reports and communicate these findings to physicians. However, most language models for radiology report summarization utilize a softmax transformation in their output layer, leading to dense alignments and strictly positive output probabilities. This density is inefficient, reducing model interpretability and giving probability mass to many unrealistic outputs. To tackle this issue, we propose a novel approach named nucleusmax. Nucleusmax is able to mitigate dense outputs and improve model interpretability by truncating the unreliable tail of the probability distribution. In addition, we incorporate nucleusmax with a copy mechanism, a useful technique to avoid professional errors in the generated diagnostic opinions. To further promote the research of radiology report summarization, we also have created a Chinese radiology report summarization dataset, which is freely available. Experimental results showed via both automatic and human evaluation that the proposed approach substantially improves the sparsity and overall quality of outputs over competitive softmax models, producing radiology summaries that approach the quality of those authored by physicians. In general, our work demonstrates the feasibility and prospect of the language model to the domain of radiology and smart healthcare.

A Natural Language Processing and Machine Learning Approach to Identification of Incidental Radiology Findings in Trauma Patients Discharged from the Emergency Department.

Patients undergoing diagnostic imaging studies in the emergency department (ED) commonly have incidental findings, which may represent unrecognized serious medical conditions, including cancer. Recognition of incidental findings frequently relies on manual review of textual radiology reports and can be overlooked in a busy clinical environment. Our study aimed to develop and validate a supervised machine learning model using natural language processing to automate the recognition of incidental findings in radiology reports of patients discharged from the ED. We performed a retrospective analysis of computed tomography (CT) reports from trauma patients discharged home across an integrated health system in 2019. Two independent annotators manually labeled CT reports for the presence of an incidental finding as a reference standard. We used regular expressions to derive and validate a random forest model using open-source and machine learning software. Final model performance was assessed across different ED types. The study CT reports were divided into derivation (690 reports) and validation (282 reports) sets, with a prevalence of incidental findings of 22.3%, and 22.7%, respectively. The random forest model had an area under the curve of 0.88 (95% confidence interval [CI], 0.84 to 0.92) on the derivation set and 0.92 (95% CI, 0.88 to 0.96) on the validation set. The final model was found to have a sensitivity of 92.2%, a specificity of 79.4%, and a negative predictive value of 97.2%. Similarly, strong model performance was found when stratified to a dedicated trauma center, high-volume, and low-volume community EDs. Machine learning and natural language processing can classify incidental findings in CT reports of ED patients with high sensitivity and high negative predictive value across a broad range of ED settings. These findings suggest the utility of natural language processing in automating the review of free-text reports to identify incidental findings and may facilitate interventions to improve timely follow-up.

Performance of a Chest Radiography AI Algorithm for Detection of Missed or Mislabeled Findings: A Multicenter Study.

Purpose: We assessed whether a CXR AI algorithm was able to detect missed or mislabeled chest radiograph (CXR) findings in radiology reports. Methods: We queried a multi-institutional radiology reports search database of 13 million reports to identify all CXR reports with addendums from 1999–2021. Of the 3469 CXR reports with an addendum, a thoracic radiologist excluded reports where addenda were created for typographic errors, wrong report template, missing sections, or uninterpreted signoffs. The remaining reports contained addenda (279 patients) with errors related to side-discrepancies or missed findings such as pulmonary nodules, consolidation, pleural effusions, pneumothorax, and rib fractures. All CXRs were processed with an AI algorithm. Descriptive statistics were performed to determine the sensitivity, specificity, and accuracy of the AI in detecting missed or mislabeled findings. Results: The AI had high sensitivity (96%), specificity (100%), and accuracy (96%) for detecting all missed and mislabeled CXR findings. The corresponding finding-specific statistics for the AI were nodules (96%, 100%, 96%), pneumothorax (84%, 100%, 85%), pleural effusion (100%, 17%, 67%), consolidation (98%, 100%, 98%), and rib fractures (87%, 100%, 94%). Conclusions: The CXR AI could accurately detect mislabeled and missed findings. Clinical Relevance: The CXR AI can reduce the frequency of errors in detection and side-labeling of radiographic findings.

Quality Management of Pulmonary Nodule Radiology Reports Based on Natural Language Processing.

To investigate the feasibility of automated follow-up recommendations based on findings in radiology reports, this paper proposed a Natural Language Processing model specific for Pulmonary Nodule Radiology Reports. Unstructured findings used to describe pulmonary nodules in 48,091 radiology reports were processed in this study. We established an NLP model to extract information entities from findings of radiology reports, using deep learning and conditional random-field algorithms. Subsequently, we constructed a knowledge graph comprising 168 entities and four relationships, based on the export recommendations of the internationally renowned Fleischner Society for pulmonary nodules. These were employed in combination with rule templates to automatically generate follow-up recommendations. The automatically generated recommendations were then compared to the impression part of the reports to evaluate the matching rate of proper follow ups in the current situation. The NLP model identified eight types of entities with a recognition accuracy of up to 94.22%. A total of 43,898 out of 48,091 clinical reports were judged to contain appropriate follow-up recommendations, corresponding to the matching rate of 91.28%. The results show that NLP can be used on Chinese radiology reports to extract structured information at the content level, thereby realizing the prompt and intelligent follow-up suggestion generation or post-quality management of follow-up recommendations.

Causal Associations Among Diseases and Imaging Findings in Radiology Reports.

This study explored the ability to identify causal relationships between diseases and imaging findings from their co-occurrences in radiology reports. A natural language processing (NLP) system with negative-expression filtering detected positive mentions of 16,912 disorders, interventions, and imaging findings in 1,702,462 consecutive radiology reports; the 55,564 causal relations defined by the Radiology Gamuts Ontology (RGO) served as reference standard. Conditions were considered to co-occur if they were present in reports from the same patient. The ϕ and κ statistics both achieved AUC0.70, P<0.001 in identifying causal relationships from pairwise co-occurrence data. Analysis of radiology reports can identify a large proportion of known causal associations among diseases and imaging findings. Automated approaches hold promise to identify causal relationships among diseases and imaging findings from their co-occurrence in text-based radiology reports.

Factors Affecting Adherence to Recommendations for Additional Imaging of Incidental Findings in Radiology Reports

To determine the rate at which recommendations for additional imaging (RAIs) of incidental findings on CT are adhered to at a tertiary-care medical center and what factors influence adherence. We used a radiology clinical informatics tool (mPower, Nuance Communications Inc, Burlington, Massachusetts) to identify RAIs in reports from all CT examinations performed at a tertiary-care medical center during a 6-month period. For those studies in which the RAI was for incidental findings, we reviewed the patients' charts to determine if there was appropriate follow-up of the lesion in question. The overall rate of adherence to RAIs was 39.1%, and in patients with a same-institution primary care provider (PCP), 56.8% (P < .0001). Adherence was higher in studies ordered in the outpatient setting (P < .0001) and in patients with a same-institution PCP (P < .0001). Among patients with a same-institution PCP, adherence was highest for outpatients (66.7%), followed by patients seen in the emergency department (46.0%) and inpatients (36.0%). Among outpatients, adherence was highest with PCPs (67%) followed by internal medicine subspecialties (50%) and surgery (38%). The rate of adherence to recommendations for additional imaging of incidental findings was 39.1% in this study and higher for patients with a same-institution PCP, studies ordered in the outpatient setting, and in studies ordered by PCPs.

Detection of unexpected findings in radiology reports: A comparative study of machine learning approaches

This study explores machine learning methods for the detection of unexpected findings in Spanish radiology reports. Regarding radiological reports, unexpected findings are the set of radiological signs identified at a certain imaging modality exam which meet two characteristics: they are not apparently related with the a priori expected results of the radiological exam and involve a clinical emergency or urgency situation that must be reported shortly to the prescribing physician or another medical specialist as well as to the patient in order to preserve life and/or prevent dangerous occurrences. Several traditional machine learning and deep learning classification algorithms are evaluated and compared. To carry out the task we use 5947 anonymous radiology reports from HT médica. Experimental results suggest that the performance of the Convolutional Neural Networks models are better than traditional machine learning. The best F1 score for the identification of an unexpected finding was 90%. Finally, we also perform an error analysis which will guide us to achieve better results in the future.

Assessment of actionable findings in radiology reports

PurposeThe American College of Radiology (ACR) Actionable Reporting Work Group defined three categories of imaging findings that require additional, nonroutine communication with the referring physician because of their urgency or unexpectedness. The objective of this study was to determine the prevalence of actionable findings in radiology reports, and to assess how well radiologists agree on the categorisation of actionable findings. MethodFrom 124,909 consecutive radiology reports stored in the electronic health record system of a large university hospital, 1000 reports were randomly selected. Two radiologists independently annotated all actionable findings according to the three categories of urgency defined by the ACR Work Group. Annotation differences were resolved in a consensus meeting and a final category was established for each report. Interannotator agreement was measured by accuracy and the kappa coefficient. ResultsThe prevalence of the three categories of actionable findings together was 32.5 %. Of all reports, 10.9 % were from patients seen in the emergency department. Prevalence of actionable findings for these patients (45.9 %) was considerably higher than for patients in routine clinical care (30.9 %). Interannotator agreement scores on the categorisation of actionable findings were 0.812 for accuracy and 0.616 for kappa coefficient. ConclusionsThe prevalence of actionable findings in radiology reports is high. The interannotator agreement scores are moderate, indicating that categorisation of actionable findings is a difficult task. To avoid unneeded increase in the workload of radiologists, in particular in routine practice, clinical context may need to be considered in deciding whether a finding is actionable.

Framework for Extracting Critical Findings in Radiology Reports.

Critical results reporting guidelines demand that certain critical findings are communicated to the responsible provider within a specific period of time. In this paper, we discuss a generic report processing pipeline to extract critical findings within the dictated report to allow for automation of quality and compliance oversight using a production dataset containing 1,210,858 radiology exams. Algorithm accuracy on an annotated dataset having 327 sentences was 91.4% (95% CI 87.6-94.2%). Our results show that most critical findings are diagnosed on CT and MR exams and that intracranial hemorrhage and fluid collection are the most prevalent at our institution. 1.6% of the exams were found to have at least one of the ten critical findings we focused on. This methodology can enable detailed analysis of critical results reporting for research, workflow management, compliance, and quality assurance.

Management Driven Structured Reporting in Ovarian Cancer

AbstractSince majority (80%) of ovarian cancer patients present at an advanced stage, imaging performed on these patients have numerous findings. The combination of multiple findings on imaging, complexity of anatomical structures which are involved in ovarian cancer, and the need to perceive certain subtle imaging features which would impact management often makes it challenging to systematically review images of these patients. Similarly, it is difficult to effectively communicate these findings in radiology reports. Structured reporting that is geared toward clinical decision-making has been an area of recognized need. An understanding of the review areas, which aid clinical decision-making in a multidisciplinary team setting at our institution led us to the proposed structured reporting template for ovarian cancer. Through this review, the authors would like to share this reporting template with examples.

Automate incidental findings in radiology reports using natural language processing and machine learning to identify and classify lung nodules.

49 Background: Pulmonary nodule incidental findings challenge providers to balance resource efficiency and high clinical quality. Incidental findings tend to be under evaluated with studies reporting appropriate follow-up rates as low as 29%. The efficient identification of patients with high risk nodules is foundational to ensuring appropriate follow-up and requires the clinical reading and classification of radiology reports. We tested the feasibility of automating this process with natural language processing (NLP) and machine learning (ML). Methods: In cooperation with Sarah Cannon, the Cancer Institute of HCA Healthcare, we conducted a series of experiments on 8,879 free-text, narrative CT radiology reports. A representative sample of health system ED, IP, and OP reports dated from Dec 2015 - April 2017 were divided into a development set for model training and validation, and a test set to evaluate model performance. A “Nodule Model” was trained to detect the reported presence of a pulmonary nodule and a rules-based “Size Model” was developed to extract the size of the nodule in mms. Reports were bucketed into three prediction groups: ≥ 6 mm, &lt;6 mm, and no size indicated. Nodules were placed in a queue for follow-up if the nodule was predicted ≥ 6 mm, or if the nodule had no size indicated and the report contained the word “mass.” The Fleischner Society Guidelines and clinical review informed these definitions. Results: Precision and recall metrics were calculated for multiple model thresholds. A threshold was selected based on the validation set calculations and a success criterion of 90% queue precision was selected to minimize false positives. On the test dataset, the F1 measure of the entire pipeline was 72.9%, recall was 60.3%, and queue precision was 90.2%, exceeding success criteria. Conclusions: The experiments demonstrate the feasibility of technology to automate the detection and classification of pulmonary nodule incidental findings in radiology reports. This approach promises to improve healthcare quality by increasing the rate of appropriate lung nodule incidental finding follow-up and treatment without excessive labor or risking overutilization.

Interactive NLP in Clinical Care: Identifying Incidental Findings in Radiology Reports.

Despite advances in natural language processing (NLP), extracting information from clinical text is expensive. Interactive tools that are capable of easing the construction, review, and revision of NLP models can reduce this cost and improve the utility of clinical reports for clinical and secondary use. We present the design and implementation of an interactive NLP tool for identifying incidental findings in radiology reports, along with a user study evaluating the performance and usability of the tool. Expert reviewers provided gold standard annotations for 130 patient encounters (694 reports) at sentence, section, and report levels. We performed a user study with 15 physicians to evaluate the accuracy and usability of our tool. Participants reviewed encounters split into intervention (with predictions) and control conditions (no predictions). We measured changes in model performance, the time spent, and the number of user actions needed. The System Usability Scale (SUS) and an open-ended questionnaire were used to assess usability. Starting from bootstrapped models trained on 6 patient encounters, we observed an average increase in F1 score from 0.31 to 0.75 for reports, from 0.32 to 0.68 for sections, and from 0.22 to 0.60 for sentences on a held-out test data set, over an hour-long study session. We found that tool helped significantly reduce the time spent in reviewing encounters (134.30 vs. 148.44 seconds in intervention and control, respectively), while maintaining overall quality of labels as measured against the gold standard. The tool was well received by the study participants with a very good overall SUS score of 78.67. The user study demonstrated successful use of the tool by physicians for identifying incidental findings. These results support the viability of adopting interactive NLP tools in clinical care settings for a wider range of clinical applications.

Identifying incidental findings from radiology reports of trauma patients: An evaluation of automated feature representation methods

BackgroundRadiologic imaging of trauma patients often uncovers findings that are unrelated to the trauma. These are termed as incidental findings and identifying them in radiology examination reports is necessary for appropriate follow-up. We developed and evaluated an automated pipeline to identify incidental findings at sentence and section levels in radiology reports of trauma patients. MethodsWe created an annotated dataset of 4,181 reports and investigated automated feature representations including traditional word and clinical concept (such as SNOMED CT) representations, as well as word and concept embeddings. We evaluated these representations by using them with traditional classifiers such as logistic regression and with deep learning methods such as convolutional neural networks (CNNs). ResultsThe best performance was observed using word embeddings with CNNs with F1 scores of 0.66 and 0.52 at section and sentence levels respectively. The F1 score was statistically significantly higher for sections compared to sentences (Wilcoxon; Z < 0.001, p < 0.05). Compared to using words alone, the addition of SNOMED CT concepts did not improve performance. At the sentence level, the F1 score improved significantly from 0.46 to 0.52 when using pre-trained embeddings (Wilcoxon; Z < 0.001, p < 0.05). ConclusionThe results show that the best performance was achieved by using embeddings with CNNs at both sentence and section levels. This provides evidence that such a pipeline is capable of accurately identifying incidental findings in radiology reports in an automated manner.

Automating incidental findings in radiology reports using natural language processing and machine learning to identify and classify pulmonary nodules.

e18093 Background: Pulmonary nodule incidental findings challenge providers to balance resource efficiency and high clinical quality. Incidental findings tend to be undertreated with studies reporting appropriate follow-up rates as low as 29%. Ensuring appropriate follow-up on all incidental findings is labor-intensive; requires the clinical reading and classification of radiology reports to identify high-risk lung nodules. We tested the feasibility of automating this process with natural language processing (NLP) and machine learning (ML). Methods: In cooperation with Sarah Cannon Research Institute (SCRI), we conducted a series of data science experiments utilizing NLP and ML computing techniques on 8,879 free-text, narrative CT (computerized tomography) radiology reports. Reports used were dated from Dec 8, 2015 - April 23, 2017, came from SCRI-affiliated Emergency Department, Inpatient, and Outpatient facilities and were a representative, random sample of the patient populations. Reports were divided into a development set for model training and validation, and a test set to evaluate model performance. Two models were developed - a “Nodule Model” was trained to detect the reported presence of a pulmonary nodule and a rules-based “Sizing Model” was developed to extract the size of the nodule in millimeters. Reports were bucketed into three prediction groups: &gt; = 6 mm, &lt; 6 mm, and no size indicated. Nodules were considered positives and placed in a queue for follow-up if the nodule was predicted &gt; = 6 mm, or if the nodule had no size indicated and the radiology report contained the word “mass.” The Fleischner Society Guidelines and clinical review informed these definitions. Results: Precision and recall metrics were calculated for multiple model thresholds. A threshold was selected based on the validation set calculations and a success criterion of 90% queue precision was selected to minimize false positives. On the test dataset, the F1 measure of the entire pipeline (lung nodule classification model and size extraction model) was 72.9%, recall was 60.3%, and queue precision was 90.2%, exceeding success criteria. Conclusions: The experiments demonstrate the feasibility of NLP and ML technology to automate the detection and classification of pulmonary nodule incidental findings in radiology reports. This approach promises to improve healthcare quality by increasing the rate of appropriate lung nodule incidental finding follow-up and treatment without excessive labor or risking overutilization.

Process improvement for follow-up radiology report recommendations of lung nodules

In the modern healthcare system, there are still wide gaps of communication of imaging results to physician and patient stakeholders and tracking of whether follow-up has occurred. Patients are also...

Automatic Disease Annotation From Radiology Reports Using Artificial Intelligence Implemented by a Recurrent Neural Network

Radiology reports are rich resources for biomedical researchers. Before utilization of radiology reports, experts must manually review these reports to identify the categories. In fact, automatically categorizing electronic medical record (EMR) text with key annotation is difficult because it has a free-text format. To address these problems, we developed an automated system for disease annotation. Reports of musculoskeletal radiography examinations performed from January 1, 2016, through December 31, 2016, were exported from the database of Hanyang University Medical Center. After sentences not written in English and sentences containing typos were excluded, 3032 sentences were included. We built a system that uses a recurrent neural network (RNN) to automatically identify fracture and nonfracture cases as a preliminary study. We trained and tested the system using orthopedic surgeon-classified reports. We evaluated the system for the number of layers in the following two ways: the word error rate of the output sentences and performance as a binary classifier using standard evaluation metrics including accuracy, precision, recall, and F1 score. The word error rate using Levenshtein distance showed the best performance in the three-layer model at 1.03%. The three-layer model also showed the highest overall performance with the highest precision (0.967), recall (0.967), accuracy (0.982), and F1 score (0.967). Our results indicate that the RNN-based system has the ability to classify important findings in radiology reports with a high F1 score. We expect that our system can be used in cohort construction such as for retrospective studies because it is efficient for analyzing a large amount of data.

Structural imaging findings on non-enhanced computed tomography are severely underreported in the primary care diagnostic work-up of subjective cognitive decline

PurposeThe purpose of this study was to investigate how structural imaging findings of medial temporal lobe atrophy (MTA), posterior cortical atrophy (PCA), global cortical atrophy (GCA), white matter changes (WMC), and Evans’ index/width of lateral ventricles (EI/WLV) are reported in the primary care diagnostic work-up of patients with subjective cognitive decline or mild cognitive impairment.MethodsWe included 197 patients referred to a non-enhanced computed tomography (NECT) as part of the diagnostic work-up. We compared the frequencies of reported findings in radiology reports written by neuroradiologists and general radiologists with actual pathological findings in a second view done by a single neuroradiologist using the MTA, PCA, GCA, WMC, and EI/WLV visual rating scales. Structural findings were also compared to cognitive tests.ResultsWe found that MTA and PCA were clearly underreported by both neuroradiologists and general radiologists. The presence of GCA and WMC was also underreported among general radiologists. Only MTA showed a clear association with cognitive test results.ConclusionsWe believe that the use of visual rating scales should be put into clinical practice to increase the yield of clinical NECT exams in the investigation of cognitive impairment. Special emphasis should be put on reporting MTA.

Natural Language-based Machine Learning Models for the Annotation of Clinical Radiology Reports.

Purpose To compare different methods for generating features from radiology reports and to develop a method to automatically identify findings in these reports. Materials and Methods In this study, 96 303 head computed tomography (CT) reports were obtained. The linguistic complexity of these reports was compared with that of alternative corpora. Head CT reports were preprocessed, and machine-analyzable features were constructed by using bag-of-words (BOW), word embedding, and Latent Dirichlet allocation-based approaches. Ultimately, 1004 head CT reports were manually labeled for findings of interest by physicians, and a subset of these were deemed critical findings. Lasso logistic regression was used to train models for physician-assigned labels on 602 of 1004 head CT reports (60%) using the constructed features, and the performance of these models was validated on a held-out 402 of 1004 reports (40%). Models were scored by area under the receiver operating characteristic curve (AUC), and aggregate AUC statistics were reported for (a) all labels, (b) critical labels, and (c) the presence of any critical finding in a report. Sensitivity, specificity, accuracy, and F1 score were reported for the best performing model's (a) predictions of all labels and (b) identification of reports containing critical findings. Results The best-performing model (BOW with unigrams, bigrams, and trigrams plus average word embeddings vector) had a held-out AUC of 0.966 for identifying the presence of any critical head CT finding and an average 0.957 AUC across all head CT findings. Sensitivity and specificity for identifying the presence of any critical finding were 92.59% (175 of 189) and 89.67% (191 of 213), respectively. Average sensitivity and specificity across all findings were 90.25% (1898 of 2103) and 91.72% (18 351 of 20 007), respectively. Simpler BOW methods achieved results competitive with those of more sophisticated approaches, with an average AUC for presence of any critical finding of 0.951 for unigram BOW versus 0.966 for the best-performing model. The Yule I of the head CT corpus was 34, markedly lower than that of the Reuters corpus (at 103) or I2B2 discharge summaries (at 271), indicating lower linguistic complexity. Conclusion Automated methods can be used to identify findings in radiology reports. The success of this approach benefits from the standardized language of these reports. With this method, a large labeled corpus can be generated for applications such as deep learning. © RSNA, 2018 Online supplemental material is available for this article.