Free-response Receiver Operating Characteristic Research Articles

A Deep Learning Framework for the Detection and Quantification of Reticular Pseudodrusen and Drusen on Optical Coherence Tomography.

The purpose of this study was to develop and validate a deep learning (DL) framework for the detection and quantification of reticular pseudodrusen (RPD) and drusen on optical coherence tomography (OCT) scans. A DL framework was developed consisting of a classification model and an out-of-distribution (OOD) detection model for the identification of ungradable scans; a classification model to identify scans with drusen or RPD; and an image segmentation model to independently segment lesions as RPD or drusen. Data were obtained from 1284 participants in the UK Biobank (UKBB) with a self-reported diagnosis of age-related macular degeneration (AMD) and 250 UKBB controls. Drusen and RPD were manually delineated by five retina specialists. The main outcome measures were sensitivity, specificity, area under the receiver operating characteristic (ROC) curve (AUC), kappa, accuracy, intraclass correlation coefficient (ICC), and free-response receiver operating characteristic (FROC) curves. The classification models performed strongly at their respective tasks (0.95, 0.93, and 0.99 AUC, respectively, for the ungradable scans classifier, the OOD model, and the drusen and RPD classification models). The mean ICC for the drusen and RPD area versus graders was 0.74 and 0.61, respectively, compared with 0.69 and 0.68 for intergrader agreement. FROC curves showed that the model's sensitivity was close to human performance. The models achieved high classification and segmentation performance, similar to human performance. Application of this robust framework will further our understanding of RPD as a separate entity from drusen in both research and clinical settings.

Diagnostic Performance in Low- and High-Contrast Tasks of an Image-Based Denoising Algorithm Applied to Radiation Dose-Reduced Multiphase Abdominal CT Examinations.

BACKGROUND. Anatomic redundancy between phases can be used to achieve denoising of multiphase CT examinations. A limitation of iterative reconstruction (IR) techniques is that they generally require use of CT projection data. A frequency-split multi-band-filtration algorithm applies denoising to the multiphase CT images themselves. This method does not require knowledge of the acquisition process or integration into the reconstruction system of the scanner, and it can be implemented as a supplement to commercially available IR algorithms. OBJECTIVE. The purpose of the present study is to compare radiologists' performance for low-contrast and high-contrast diagnostic tasks (i.e., tasks for which differences in CT attenuation between the imaging target and its anatomic background are subtle or large, respectively) evaluated on multiphase abdominal CT between routine-dose images and radiation dose-reduced images processed by a frequency-split multiband-filtration denoising algorithm. METHODS. This retrospective single-center study included 47 patients who underwent multiphase contrast-enhanced CT for known or suspected liver metastases (a low-contrast task) and 45 patients who underwent multiphase contrast-enhanced CT for pancreatic cancer staging (a high-contrast task). Radiation dose-reduced images corresponding to dose reduction of 50% or more were created using a validated noise insertion technique and then underwent denoising using the frequency-split multi-band-filtration algorithm. Images were independently evaluated in multiple sessions by different groups of abdominal radiologists for each task (three readers in the low-contrast arm and four readers in the high-contrast arm). The noninferiority of denoised radiation dose-reduced images to routine-dose images was assessed using the jackknife alternative free-response ROC (JAFROC) figure-of-merit (FOM; limit of noninferiority, -0.10) for liver metastases detection and using the Cohen kappa statistic and reader confidence scores (100-point scale) for pancreatic cancer vascular invasion. RESULTS. For liver metastases detection, the JAFROC FOM for denoised radiation dose-reduced images was 0.644 (95% CI, 0.510-0.778), and that for routine-dose images was 0.668 (95% CI, 0.543-0.792; estimated difference, -0.024 [95% CI, -0.084 to 0.037]). Intraobserver agreement for pancreatic cancer vascular invasion was substantial to near perfect when the two image sets were compared (κ = 0.53-1.00); the 95% CIs of all differences in confidence scores between image sets contained zero. CONCLUSION. Multiphase contrast-enhanced abdominal CT images with a radiation dose reduction of 50% or greater that undergo denoising by a frequency-split multiband-filtration algorithm yield performance similar to that of routine-dose images for detection of liver metastases and vascular staging of pancreatic cancer. CLINICAL IMPACT. The image-based denoising algorithm facilitates radiation dose reduction of multiphase examinations for both low- and high-contrast diagnostic tasks without requiring manufacturer-specific hardware or software.

Multiarterial Phase Acquisition in Gadoxetic Acid-Enhanced Liver MRI for the Detection of Hypervascular Hepatocellular Carcinoma in High-Risk Patients: Comparison of Compressed Sensing Versus View Sharing Techniques.

The aim of this study was to compare compressed sensing (CS) and view sharing (VS) techniques for single breath-hold multiarterial phase imaging with respect to image quality and focal liver observation detectability during gadoxetic acid-enhanced magnetic resonance imaging in patients at high risk for hepatocellular carcinoma (HCC). A total of 385 patients who underwent gadoxetic acid-enhanced magnetic resonance imaging, including triple arterial phases using either CS (n = 224) or VS (n = 161) techniques, were retrospectively included. Among them, 117 patients had 171 focal liver observations (median diameter, 1.3 cm), which were classified according to Liver Imaging Reporting and Data System version 2018. The acquisition rate of optimally timed late arterial phase (LAP) was assessed, and image quality, including respiratory motion artifact and observation conspicuity, was rated on a 4-point scale by 3 radiologists. The Mann-Whitney U test and nonparametric test for repeated measures data were used for image quality and observation conspicuity analysis. The jackknife alternative free-response receiver operating characteristics method was used to compare the observation detectability between the 2 techniques. The CS technique showed significantly higher acquisition rate of optimally timed LAP without transient severe motion (82.1% [184/224] vs 71.4% [115/161]; P = 0.013) than the VS technique. The CS technique also demonstrated significantly improved overall image quality (3.42 ± 0.70 vs 2.97 ± 0.61; P < 0.001) compared with the VS technique. Regarding the detection of hyperenhancing observations, there was no significant difference between the figure of merits of CS and VS techniques (0.660 vs 0.665; P = 0.890). However, the CS technique showed a higher detection rate in Liver Imaging Reporting and Data System M (LR-M, probably or definitely malignant but not HCC specific) observations than the VS technique (100.0% [9/9] vs 44.4% [8/18]; P = 0.009). The CS technique tended to provide optimally timed LAP without transient severe motion and demonstrated greater detection rate of LR-M observations than the VS technique in patients at high risk of HCC.

Detection of distant metastases and distant second primary cancers in head and neck squamous cell carcinoma: comparison of [18F]FDG PET/MRI and [18F]FDG PET/CT

PurposeThis prospective study aimed to compare the diagnostic performance of [18]FDG PET/MRI and PET/CT for the detection of distant metastases and distant second primary cancers in patients with head and neck squamous cell carcinoma (HNSCC).MethodsA total of 103 [18F]FDG PET/MRI examinations immediately followed by PET/CT were obtained in 82 consecutive patients for staging of primary HNSCC (n = 38), suspected loco-regional recurrence/follow-up (n = 41) or unknown primary HNSCC (n = 3). Histology and follow-up > 2 years formed the standard of reference. Blinded readers evaluated the anonymized PET/MRI and PET/CT examinations separately using a 5-point Likert score. Statistical analysis included: receiver operating characteristic (ROC) analysis, jackknife alternative free-response ROC (JAFROC) and region-of-interest (ROI)-based ROC to account for data clustering and sensitivity/specificity/accuracy comparisons for a score ≥ 3.ResultsDistant metastases and distant second primary cancers were present in 23/103 (22%) examinations in 16/82 (19.5%) patients, and they were more common in the post-treatment group (11/41, 27%) than in the primary HNSCC group (3/38, 8%), p = 0.039. The area under the curve (AUC) per patient/examination/lesion was 0.947 [0.927–1]/0.965 [0.917–1]/0.957 [0.928–0.987] for PET/MRI and 0.975 [0.950–1]/0.968 [0.920–1]/0.944 [0.910–0.979] for PET/CT, respectively (p > 0.05). The diagnostic performance of PET/MRI and PET/CT was similar according to JAFROC (p = 0.919) and ROI-based ROC analysis (p = 0.574). Sensitivity/specificity/accuracy for PET/MRI and PET/CT for a score ≥ 3 was 94%/88%/89% and 94%/91%/91% per patient, 96%/90%/91% and 96%/93%/93% per examination and 95%/85%/90% and 90%/86%/88% per lesion, respectively, p > 0.05.ConclusionsIn HNSCC patients, PET/MRI and PET/CT had a high and similar diagnostic performance for detecting distant metastases and distant second primary cancers.

Rethinking Annotation Granularity for Overcoming Shortcuts in Deep Learning-based Radiograph Diagnosis: A Multicenter Study.

To evaluate the ability of fine-grained annotations to overcome shortcut learning in deep learning (DL)-based diagnosis using chest radiographs. Two DL models were developed using radiograph-level annotations (disease present: yes or no) and fine-grained lesion-level annotations (lesion bounding boxes), respectively named CheXNet and CheXDet. A total of 34 501 chest radiographs obtained from January 2005 to September 2019 were retrospectively collected and annotated regarding cardiomegaly, pleural effusion, mass, nodule, pneumonia, pneumothorax, tuberculosis, fracture, and aortic calcification. The internal classification performance and lesion localization performance of the models were compared on a testing set (n = 2922); external classification performance was compared on National Institutes of Health (NIH) Google (n = 4376) and PadChest (n = 24 536) datasets; and external lesion localization performance was compared on the NIH ChestX-ray14 dataset (n = 880). The models were also compared with radiologist performance on a subset of the internal testing set (n = 496). Performance was evaluated using receiver operating characteristic (ROC) curve analysis. Given sufficient training data, both models performed similarly to radiologists. CheXDet achieved significant improvement for external classification, such as classifying fracture on NIH Google (CheXDet area under the ROC curve [AUC], 0.67; CheXNet AUC, 0.51; P < .001) and PadChest (CheXDet AUC, 0.78; CheXNet AUC, 0.55; P < .001). CheXDet achieved higher lesion detection performance than CheXNet for most abnormalities on all datasets, such as detecting pneumothorax on the internal set (CheXDet jackknife alternative free-response ROC [JAFROC] figure of merit [FOM], 0.87; CheXNet JAFROC FOM, 0.13; P < .001) and NIH ChestX-ray14 (CheXDet JAFROC FOM, 0.55; CheXNet JAFROC FOM, 0.04; P < .001). Fine-grained annotations overcame shortcut learning and enabled DL models to identify correct lesion patterns, improving the generalizability of the models.Keywords: Computer-aided Diagnosis, Conventional Radiography, Convolutional Neural Network (CNN), Deep Learning Algorithms, Machine Learning Algorithms, Localization Supplemental material is available for this article © RSNA, 2022.

Fracture R-CNN: An anchor-efficient anti-interference framework for skull fracture detection in CT images.

Skull fracture, as a common traumatic brain injury, can lead to multiple complications including bleeding, leaking of cerebrospinal fluid, infection, and seizures. Automatic skull fracture detection (SFD) is of great importance, especially in emergencymedicine. Existing algorithms for SFD, developed based on hand-crafted features, suffer from low detection accuracy due to poor generalizability to unseen samples. Deploying deep detectors designed for natural images like Faster Region-based Convolutional Neural Network (R-CNN) for SFD can be helpful but are of high redundancy and with nonnegligible false detections due to the cranial suture and skull base interference. Therefore, we, for the first time, propose an anchor-efficient anti-interference deep learning framework named Fracture R-CNN for accurate SFD with low computationalcost. The proposed Fracture R-CNN is developed by incorporating the prior knowledge utilized in clinical diagnosis into the original Faster R-CNN. Specifically, based on the distributions of skull fractures, we first propose an adaptive anchoring region proposal network (AA-RPN) to generate proposals for diverse-scale fractures with low computational complexity. Then, based on the prior knowledge that cranial sutures exist in the junctions of bones and usually contain sclerotic margins, we design an anti-interference head (A-Head) network to eliminate the cranial suture interference for better SFD detection. In addition, to further enhance the anti-interference ability of the proposed A-Head, a difficulty-balanced weighted loss function is proposed to emphasize more on distinguishing the interference areas from the skull base and the cranial sutures duringtraining. Experimental results demonstrate that the proposed Fracture R-CNN outperforms the current state-of-the-art (SOTA) deep detectors for SFD with a higher recall and fewer false detections. Compared to Faster R-CNN, the proposed Fracture R-CNN improves the average precision (AP) by 11.74% and the free-response receiver operating characteristic (FROC) score by 11.08%. Through validating on various backbones, we further demonstrate the architecture independence of Fracture R-CNN, making it extendable to other detectionapplications. As the customized deep learning-based framework for SFD, Fracture R-CNN can effectively overcome the unique challenges in SFD with less computational cost, leading to a better detection performance compared to the SOTA deep detectors. Moreover, we believe the prior knowledge explored for Fracture R-CNN would shed new light on future deep learning approaches for SFD.

OA08 Development of an automated deep learning-based system for distinguishing between ‘systemic sclerosis' and ‘normal' capillaries

Abstract Background/Aims Nailfold capillaroscopy is a key investigation in differentiating between primary and systemic sclerosis (SSc)-related Raynaud’s phenomenon (RP). However, interpreting capillaroscopy images (i.e. distinguishing ‘normal’ capillaries of primary RP [PRP] from abnormal capillaries of SSc) can be challenging. Our objectives were: (1) to develop a complete deep learning-based system to identify and classify capillary apices, and compute useful parameters such as width, shape score and vessel density; and (2) use the measured parameters to train and test a logistic regressor to distinguish between subjects with SSc and those with ‘normal’ capillary appearance. Methods 911 mosaic images were included for image-level feature extraction. 456 mosaic images (104 healthy control [HC], 83 PRP, 269 SSc) were used for training and the remaining 455 mosaic images (104 HC, 83 PRP and 268 SSc) were used for validation and testing the deep learning model. For subject-level classification 133 subjects (105 [79%] female, median age [range] 57 [19-80]) were used; 87 (25 HC, 14 PRP, and 48 SSc) were used for training and the remaining 46 (12 HC, 3 PRP, 31 SSc) for testing the logistic regressor. Images from patients with PRP and healthy subjects were grouped for all cases. The deep learning system uses a UNet architecture for capillary detection and vessel segmentation (density and shape score estimation). A ResNet-34 framework is used for capillary classification (distal vs. non-distal) and computing vessel widths. Area under Receiver Operating Characteristic (ROC) curves (AUC) measures the classification accuracy when assigning individuals to SSc and HC/PRP groups. Repeatability experiments were performed with different random seeds to validate the reproducibility of the results. Results For apex detection, the sensitivity and average false positive count from the Free-response Receiver Operating Characteristic (FROC) curve were 0.91 and 127.6, respectively. Apex classification accuracy (distal apices) was 93.8%, with an AUC of 0.92 and sensitivity of 0.9603. Overall AUC for subject level classification (HC/PRP vs SSc) for the current approach was 0.9739 ± 0.0207. Conclusion We have developed a completely automated nailfold capillary detection and analysis system based on deep learning. Apex detection, distal classification, and subsequent computation of distal apex measurements from nailfold capillaroscopy images are all performed with no human input. We have combined image-level features and have built a subject-level classifier model which successfully discriminates between SSc and HC/PRP with an improved AUC (i.e. improved sensitivity and specificity) compared to our previous work. We have performed the repeatability experiments with different random seeds and the performance of system is consistent with all the seeds. Disclosure P. Gurunath Bharathi: None. M. Berks: None. A. Herrick: None. G. Dinsdale: None. A. Murray: None. J. Manning: None. S. Wilkinson: None. C. Taylor: None.

A novel regression method for the analysis of multireader multicase-free-response receiver operating characteristics studies.

In diagnostic radiology, the multireader multicase (MRMC) design and the free-response receiver operating characteristics (FROC) method are often used in combination. The cross-correlated data generated by the MRMC-FROC study leads to difficulties in the corresponding analysis, and the need to include covariates in the model further complicates the subsequent analysis. In this paper, we propose a regression approach based on three new measures with good interpretability. The correlation structure of the original test results is taken directly into account in the estimation procedure. The proposed method also allows the inclusion of continuous or discrete covariates. Consistent and asymptotically normal estimators are derived for the new measures. Simulation studies are conducted to evaluate the performance of the proposed approach. The simulation results show that the regression approach performs well under a wide range of scenarios. We also apply the proposed regression approach to a diagnostic study of computer-aided diagnosis in lung cancer.

Deep learning-based image reconstruction of 40-keV virtual monoenergetic images of dual-energy CT for the assessment of hypoenhancing hepatic metastasis.

To evaluate the diagnostic value of deep learning model (DLM) reconstructed dual-energy CT (DECT) low-keV virtual monoenergetic imaging (VMI) for assessing hypoenhancing hepatic metastases. This retrospective study included 131 patients who underwent contrast-enhanced DECT (80-kVp and 150-kVp with a tin filter) in the portal venous phase for hepatic metastasis surveillance. Linearly blended images simulating 100-kVp images (100-kVp), standard 40-keV VMI images (40-keV VMI), and post-processed 40-keV VMI using a vendor-agnostic DLM (i.e., DLM 40-keV VMI) were reconstructed. Lesion conspicuity and diagnostic acceptability were assessed by three independent reviewers and compared using the Wilcoxon signed-rank test. The contrast-to-noise ratios (CNRs) were also measured placing ROIs in metastatic lesions and liver parenchyma. The detection performance of hepatic metastases was assessed by using a jackknife alternative free-response ROC method. The consensus by two independent radiologists was used as the reference standard. DLM 40-keV VMI, compared to 40-keV VMI and 100-kVp, showed a higher lesion-to-liver CNR (8.25 ± 3.23 vs. 6.05 ± 2.38 vs. 5.99 ± 2.00), better lesion conspicuity (4.3 (4.0-4.7) vs. 3.7 (3.7-4.0) vs. 3.7 (3.3-4.0)), and better diagnostic acceptability (4.3 (4.0-4.3) vs. 3.0 (2.7-3.3) vs. 4.0 (4.0-4.3)) (p < 0.001 for all). For lesion detection (246 hepatic metastases in 68 patients), the figure of merit was significantly higher with DLM 40-keV VMI than with 40-keV VMI (0.852 vs. 0.822, p = 0.012), whereas no significant difference existed between DLM 40-keV VMI and 100-kVp (0.852 vs. 0.842, p = 0.31). DLM 40-keV VMI provided better image quality and comparable diagnostic performance for detecting hypoenhancing hepatic metastases compared to linearly blended images. • DLM 40-keV VMI provides a superior image quality compared with 40-keV or 100-kVp for assessing hypoenhancing hepatic metastasis. • DLM 40-keV VMI has the highest CNR and lesion conspicuity score for hypoenhancing hepatic metastasis due to noise reduction and structural preservation. • DLM 40-keV VMI provides higher lesion detectability than standard 40-keV VMI (p = 0.012).

ProstAttention-Net: A deep attention model for prostate cancer segmentation by aggressiveness in MRI scans

Multiparametric magnetic resonance imaging (mp-MRI) has shown excellent results in the detection of prostate cancer (PCa). However, characterizing prostate lesions aggressiveness in mp-MRI sequences is impossible in clinical practice, and biopsy remains the reference to determine the Gleason score (GS). In this work, we propose a novel end-to-end multi-class network that jointly segments the prostate gland and cancer lesions with GS group grading. After encoding the information on a latent space, the network is separated in two branches: 1) the first branch performs prostate segmentation 2) the second branch uses this zonal prior as an attention gate for the detection and grading of prostate lesions. The model was trained and validated with a 5-fold cross-validation on a heterogeneous series of 219 MRI exams acquired on three different scanners prior prostatectomy. In the free-response receiver operating characteristics (FROC) analysis for clinically significant lesions (defined as GS>6) detection, our model achieves 69.0%±14.5% sensitivity at 2.9 false positive per patient on the whole prostate and 70.8%±14.4% sensitivity at 1.5 false positive when considering the peripheral zone (PZ) only. Regarding the automatic GS group grading, Cohen's quadratic weighted kappa coefficient (κ) is 0.418±0.138, which is the best reported lesion-wise kappa for GS segmentation to our knowledge. The model has encouraging generalization capacities with κ=0.120±0.092 on the PROSTATEx-2 public dataset and achieves state-of-the-art performance for the segmentation of the whole prostate gland with a Dice of 0.875±0.013. Finally, we show that ProstAttention-Net improves performance in comparison to reference segmentation models, including U-Net, DeepLabv3+ and E-Net. The proposed attention mechanism is also shown to outperform Attention U-Net.

Human Observer Net: A Platform Tool for Human Observer Studies of Image Data.

Background Current software applications for human observer studies of images lack flexibility in study design, platform independence, multicenter use, and assessment methods and are not open source, limiting accessibility and expandability. Purpose To develop a user-friendly software platform that enables efficient human observer studies in medical imaging with flexibility of study design. Materials and Methods Software for human observer imaging studies was designed as an open-source web application to facilitate access, platform-independent usability, and multicenter studies. Different interfaces for study creation, participation, and management of results were implemented. The software was evaluated in human observer experiments between May 2019 and March 2021, in which duration of observer responses was tracked. Fourteen radiologists evaluated and graded software usability using the 100-point system usability scale. The application was tested in Chrome, Firefox, Safari, and Edge browsers. Results Software function was designed to allow visual grading analysis (VGA), multiple-alternative forced-choice (m-AFC), receiver operating characteristic (ROC), localization ROC, free-response ROC, and customized designs. The mean duration of reader responses per image or per image set was 6.2 seconds ± 4.8 (standard deviation), 5.8 seconds ± 4.7, 8.7 seconds ± 5.7, and 6.0 seconds ± 4.5 in four-AFC with 160 image quartets per reader, four-AFC with 640 image quartets per reader, localization ROC, and experimental studies, respectively. The mean system usability scale score was 83 ± 11 (out of 100). The documented code and a demonstration of the application are available online (https://github.com/genskeu/HON, https://hondemo.pythonanywhere.com/). Conclusion A user-friendly and efficient open-source application was developed for human reader experiments that enables study design versatility, as well as platform-independent and multicenter usability. © RSNA, 2022 Online supplemental material is available for this article. See also the editorial by Thompson in this issue.

Abbreviated Gadoxetic Acid-Enhanced MRI for the Detection of Liver Metastases in Patients With Potentially Resectable Pancreatic Ductal Adenocarcinoma.

Gadoxetic acid-enhanced magnetic resonance imaging (MRI) is useful in detecting liver metastases from pancreatic ductal adenocarcinoma (PDAC). However, the long examination time limits its utility in the initial workup of patients with PDAC. To evaluate the incremental value of an abbreviated gadoxetic acid-enhanced MRI for the detection of liver metastases in patients with PDAC. Retrospective. Patients (N=130) with potentially resectable PDAC (women, 58 [44.6%]). 1.5 T and 3T; gradient dual-echo T1-weighted (in-phase and opposed-phase), fat-suppressed fast spin-echo T2-weighted, single-shot echo-planar diffusion-weighted, and three-dimensional fat-suppressed T1-weighted gradient-echo dynamic contrast-enhanced and hepatobiliary phase sequences, as well as contrast-enhanced computed tomography (CECT). Three radiologists independently reviewed three different image sets to detect liver metastases: set 1, CECT alone; set 2, CECT and abbreviated MRI comprising fat-suppressed T2-weighted, diffusion-weighted, and hepatobiliary phase images; and set 3, CECT and standard gadoxetic acid-enhanced MRI. Figure of merit (FOM) was compared using the jackknife alternative free-response receiver operating characteristics, and other per-lesion and per-patient diagnostic parameters for each image set were compared using McNemar's and Fisher's test. P < 0.05 was considered statistically significant. A total of 43 liver metastases were identified in 13 patients. Reader-averaged FOM to detect liver metastases were significantly higher for sets 2 (0.884) and 3 (0.886) than for set 1 (0.609), while they were comparable between sets 2 and 3 (P=0.96). The mean per-patient sensitivities, negative predictive values, and accuracies were significantly higher for sets 2 and 3 than for set 1, while those between sets 2 and 3 were not significantly different (not applicable, P > 0.99, and P > 0.99, respectively). Gadoxetic acid-enhanced MRI combined with CECT had higher diagnostic performance than CECT alone for the detection of liver metastases in patients with PDAC. The incremental values were comparable for the abbreviated MRI and standard MRI. 3 TECHNICAL EFFICACY: Stage 2.

An Accurate and Explainable Deep Learning System Improves Interobserver Agreement in the Interpretation of Chest Radiograph

Interpretation of chest radiographs (CXR) is a difficult but essential task for detecting thoracic abnormalities. Recent artificial intelligence (AI) algorithms have achieved radiologist-level performance on various medical classification tasks. However, only a few studies addressed the localization of abnormal findings from CXR scans, which is essential in explaining the image-level classification to radiologists. Additionally, the actual impact of AI algorithms on the diagnostic performance of radiologists in clinical practice remains relatively unclear. To bridge these gaps, we developed an explainable deep learning system called VinDr-CXR that can classify a CXR scan into multiple thoracic diseases and, at the same time, localize most types of critical findings on the image. VinDr-CXR was trained on 51,485 CXR scans with radiologist-provided bounding box annotations. It demonstrated a comparable performance to experienced radiologists in classifying 6 common thoracic diseases on a retrospective validation set of 3,000 CXR scans, with a mean area under the receiver operating characteristic curve (AUROC) of 0.967 (95% confidence interval [CI]: 0.958–0.975). The VinDr-CXR was also externally validated in independent patient cohorts and showed its robustness. For the localization task with 14 types of lesions, our free-response receiver operating characteristic (FROC) analysis showed that the VinDr-CXR achieved a sensitivity of 80.2% at the rate of 1.0 false-positive lesion identified per scan. A prospective study was also conducted to measure the clinical impact of the VinDr-CXR in assisting six experienced radiologists. The results indicated that the proposed system, when used as a diagnosis supporting tool, significantly improved the agreement between radiologists themselves with an increase of 1.5% in mean Fleiss’ Kappa. We also observed that, after the radiologists consulted VinDr-CXR’s suggestions, the agreement between each of them and the system was remarkably increased by 3.3% in mean Cohen’s Kappa. Altogether, our results highlight the potentials of the proposed deep learning system as an effective assistant to radiologists in clinical practice. Part of the dataset used for developing the VinDr-CXR system has been made publicly available at https://physionet.org/content/vindr-cxr/1.0.0/.

Bone Suppression on Chest Radiographs for Pulmonary Nodule Detection: Comparison between a Generative Adversarial Network and Dual-Energy Subtraction

ObjectiveTo compare the effects of bone suppression imaging using deep learning (BSp-DL) based on a generative adversarial network (GAN) and bone subtraction imaging using a dual energy technique (BSt-DE) on radiologists’ performance for pulmonary nodule detection on chest radiographs (CXRs).Materials and MethodsA total of 111 adults, including 49 patients with 83 pulmonary nodules, who underwent both CXR using the dual energy technique and chest CT, were enrolled. Using CT as a reference, two independent radiologists evaluated CXR images for the presence or absence of pulmonary nodules in three reading sessions (standard CXR, BSt-DE CXR, and BSp-DL CXR). Person-wise and nodule-wise performances were assessed using receiver-operating characteristic (ROC) and alternative free-response ROC (AFROC) curve analyses, respectively. Subgroup analyses based on nodule size, location, and the presence of overlapping bones were performed.ResultsBSt-DE with an area under the AFROC curve (AUAFROC) of 0.996 and 0.976 for readers 1 and 2, respectively, and BSp-DL with AUAFROC of 0.981 and 0.958, respectively, showed better nodule-wise performance than standard CXR (AUAFROC of 0.907 and 0.808, respectively; p ≤ 0.005). In the person-wise analysis, BSp-DL with an area under the ROC curve (AUROC) of 0.984 and 0.931 for readers 1 and 2, respectively, showed better performance than standard CXR (AUROC of 0.915 and 0.798, respectively; p ≤ 0.011) and comparable performance to BSt-DE (AUROC of 0.988 and 0.974; p ≥ 0.064). BSt-DE and BSp-DL were superior to standard CXR for detecting nodules overlapping with bones (p < 0.017) or in the upper/middle lung zone (p < 0.017). BSt-DE was superior (p < 0.017) to BSp-DL in detecting peripheral and sub-centimeter nodules.ConclusionBSp-DL (GAN-based bone suppression) showed comparable performance to BSt-DE and can improve radiologists’ performance in detecting pulmonary nodules on CXRs. Nevertheless, for better delineation of small and peripheral nodules, further technical improvements are required.

Diagnostic effect of artificial intelligence solution for referable thoracic abnormalities on chest radiography: a multicenter respiratory outpatient diagnostic cohort study

ObjectivesWe aimed to evaluate a commercial artificial intelligence (AI) solution on a multicenter cohort of chest radiographs and to compare physicians' ability to detect and localize referable thoracic abnormalities with and without AI assistance.MethodsIn this retrospective diagnostic cohort study, we investigated 6,006 consecutive patients who underwent both chest radiography and CT. We evaluated a commercially available AI solution intended to facilitate the detection of three chest abnormalities (nodule/masses, consolidation, and pneumothorax) against a reference standard to measure its diagnostic performance. Moreover, twelve physicians, including thoracic radiologists, board-certified radiologists, radiology residents, and pulmonologists, assessed a dataset of 230 randomly sampled chest radiographic images. The images were reviewed twice per physician, with and without AI, with a 4-week washout period. We measured the impact of AI assistance on observer's AUC, sensitivity, specificity, and the area under the alternative free-response ROC (AUAFROC).ResultsIn the entire set (n = 6,006), the AI solution showed average sensitivity, specificity, and AUC of 0.885, 0.723, and 0.867, respectively. In the test dataset (n = 230), the average AUC and AUAFROC across observers significantly increased with AI assistance (from 0.861 to 0.886; p = 0.003 and from 0.797 to 0.822; p = 0.003, respectively).ConclusionsThe diagnostic performance of the AI solution was found to be acceptable for the images from respiratory outpatient clinics. The diagnostic performance of physicians marginally improved with the use of AI solutions. Further evaluation of AI assistance for chest radiographs using a prospective design is required to prove the efficacy of AI assistance.Key Points• AI assistance for chest radiographs marginally improved physicians’ performance in detecting and localizing referable thoracic abnormalities on chest radiographs.• The detection or localization of referable thoracic abnormalities by pulmonologists and radiology residents improved with the use of AI assistance.

Rib fracture detection system based on deep learning

Rib fracture detection is time-consuming and demanding work for radiologists. This study aimed to introduce a novel rib fracture detection system based on deep learning which can help radiologists to diagnose rib fractures in chest computer tomography (CT) images conveniently and accurately. A total of 1707 patients were included in this study from a single center. We developed a novel rib fracture detection system on chest CT using a three-step algorithm. According to the examination time, 1507, 100 and 100 patients were allocated to the training set, the validation set and the testing set, respectively. Free Response ROC analysis was performed to evaluate the sensitivity and false positivity of the deep learning algorithm. Precision, recall, F1-score, negative predictive value (NPV) and detection and diagnosis were selected as evaluation metrics to compare the diagnostic efficiency of this system with radiologists. The radiologist-only study was used as a benchmark and the radiologist-model collaboration study was evaluated to assess the model’s clinical applicability. A total of 50,170,399 blocks (fracture blocks, 91,574; normal blocks, 50,078,825) were labelled for training. The F1-score of the Rib Fracture Detection System was 0.890 and the precision, recall and NPV values were 0.869, 0.913 and 0.969, respectively. By interacting with this detection system, the F1-score of the junior and the experienced radiologists had improved from 0.796 to 0.925 and 0.889 to 0.970, respectively; the recall scores had increased from 0.693 to 0.920 and 0.853 to 0.972, respectively. On average, the diagnosis time of radiologist assisted with this detection system was reduced by 65.3 s. The constructed Rib Fracture Detection System has a comparable performance with the experienced radiologist and is readily available to automatically detect rib fracture in the clinical setting with high efficacy, which could reduce diagnosis time and radiologists’ workload in the clinical practice.

Image quality in liver CT: low-dose deep learning vs standard-dose model-based iterative reconstructions

To compare the overall image quality and detectability of significant (malignant and pre-malignant) liver lesions of low-dose liver CT (LDCT, 33.3% dose) using deep learning denoising (DLD) to standard-dose CT (SDCT, 100% dose) using model-based iterative reconstruction (MBIR). In this retrospective study, CT images of 80 patients with hepatic focal lesions were included. For noninferiority analysis of overall image quality, a margin of - 0.5 points (scored in a 5-point scale) for the difference between scan protocols was pre-defined. Other quantitative or qualitative image quality assessments were performed. Additionally, detectability of significant liver lesions was compared, with 64 pairs of CT, using the jackknife alternative free-response ROC analysis, with noninferior margin defined by the lower limit of 95% confidence interval (CI) of the difference of figure-of-merit less than - 0.1. The mean overall image quality scores with LDCT and SDCT were 3.77 ± 0.38 and 3.94 ± 0.34, respectively, demonstrating a difference of - 0.17 (95% CI: - 0.21 to - 0.12), which did not cross the predefined noninferiority margin of - 0.5. Furthermore, LDCT showed significantly superior quantitative results of liver lesion contrast to noise ratio (p < 0.05). However, although LDCT scored higher than the average score in qualitative image quality assessments, they were significantly lower than those of SDCT (p < 0.05). Figure-of-merit for lesion detection was 0.859 for LDCT and 0.878 for SDCT, showing noninferiority (difference: - 0.019, 95% CI: - 0.058 to 0.021). LDCT using DLD with 67% radiation dose reduction showed non-inferior overall image quality and lesion detectability, compared to SDCT. • Low-dose liver CT using deep learning denoising (DLD), at 67% dose reduction, provided non-inferior overall image quality compared to standard-dose CT using model-based iterative reconstruction (MBIR). • Low-dose CT using DLD showed significantly less noise and higher CNR lesion to liver than standard-dose CT using MBIR and demonstrated at least average image quality score among all readers, albeit with lower scores than standard-dose CT using MBIR. • Low-dose liver CT showed noninferior detectability for malignant and pre-malignant liver lesions, compared to standard-dose CT.

Deep learning\u2013assisted prostate cancer detection on bi-parametric MRI: minimum training data size requirements and effect of prior knowledge

ObjectivesTo assess Prostate Imaging Reporting and Data System (PI-RADS)–trained deep learning (DL) algorithm performance and to investigate the effect of data size and prior knowledge on the detection of clinically significant prostate cancer (csPCa) in biopsy-naïve men with a suspicion of PCa.MethodsMulti-institution data included 2734 consecutive biopsy-naïve men with elevated PSA levels (≥ 3 ng/mL) that underwent multi-parametric MRI (mpMRI). mpMRI exams were prospectively reported using PI-RADS v2 by expert radiologists. A DL framework was designed and trained on center 1 data (n = 1952) to predict PI-RADS ≥ 4 (n = 1092) lesions from bi-parametric MRI (bpMRI). Experiments included varying the number of cases and the use of automatic zonal segmentation as a DL prior. Independent center 2 cases (n = 296) that included pathology outcome (systematic and MRI targeted biopsy) were used to compute performance for radiologists and DL. The performance of detecting PI-RADS 4–5 and Gleason > 6 lesions was assessed on 782 unseen cases (486 center 1, 296 center 2) using free-response ROC (FROC) and ROC analysis.ResultsThe DL sensitivity for detecting PI-RADS ≥ 4 lesions was 87% (193/223, 95% CI: 82–91) at an average of 1 false positive (FP) per patient, and an AUC of 0.88 (95% CI: 0.84–0.91). The DL sensitivity for the detection of Gleason > 6 lesions was 85% (79/93, 95% CI: 77–83) @ 1 FP compared to 91% (85/93, 95% CI: 84–96) @ 0.3 FP for a consensus panel of expert radiologists. Data size and prior zonal knowledge significantly affected performance (4%, p<0.05).ConclusionPI-RADS-trained DL can accurately detect and localize Gleason > 6 lesions. DL could reach expert performance using substantially more than 2000 training cases, and DL zonal segmentation.Key Points• AI for prostate MRI analysis depends strongly on data size and prior zonal knowledge.• AI needs substantially more than 2000 training cases to achieve expert performance.

Comparison of two-view versus single-view digital breast tomosynthesis and 2D-mammography in breast cancer surveillance imaging.

PurposeLimited work has been performed for the implementation of digital breast tomosynthesis (DBT) in breast cancer surveillance imaging. The aim of this study was to investigate the differences between two different DBT implementations in breast cancer surveillance imaging, for patients with a personal history of breast cancer.MethodThe DBT implementations investigated were: (1) 2-view 2D digital mammography and 2-view DBT (2vDM&2vDBT) (2) 1-view (cranial-caudal) DM and 1-view (mediolateral-oblique) DBT (1vDM&1vDBT). Clinical performance of these two implementations was assessed retrospectively using observer studies with 118 sets of real patient images, from a single imaging centre, and six observers. Sensitivity, specificity and area under the curve (AUC) using the Jack-knife alternative free-response receiver operating characteristics (JAFROC) analysis were evaluated.ResultsResults suggest that the two DBT implementations are not significantly different in terms of sensitivity, specificity and AUC. When looking at the two main different lesion types, non-calcifications and calcifications, and two different density levels, no difference in the performance of the two DBT implementations was found.ConclusionsSince 1vDM&1vDBT exposes the patient to half the dose of 2vDM&2vDBT, it might be worth considering 1vDM&1vDBT in breast cancer surveillance imaging. However, larger studies are required to conclude on this matter.

Precise anatomical localization and classification of rib fractures on CT using a convolutional neural network

ObjectiveTo develop a convolutional neural network (CNN) model for the detection, precise anatomical localization (right 1-12th and left 1-12th) and classification (fresh, healing and old fractures) of rib fractures automatically, and to compare the performance with the experienced radiologists. Materials and methodsA total of 640 rib fracture patients with 340,501 annotations were retrospectively collected from three hospitals. They consisted of a classification training dataset (n = 482), a localization training dataset (n = 30), an internal testing dataset (n = 90) and an external testing dataset (n = 38). RetinaNet with rib localization postprocessing and the result merging technique were employed to structure the CNN model. ROC curve, free-response ROC curve, AUC, precision, recall, and F1-score were calculated to choose the better option between model I (training classification and localization data together) and model II (adding an additional classification model to model I). ResultsThe detection and classification performance of rib fractures was better in model II than in model I. The sensitivity of localization reached 97.11% and 94.87% on the right and left ribs, respectively. In the external dataset with different CT scanner and slice thickness, model II showed better diagnostic performance. Moreover, the CNN model was superior in diagnosing fresh and healing fractures to 5 radiologists and consumed shorter diagnosis time. ConclusionsOur CNN model was capable of detection, precise anatomical localization, and classification of rib fractures automatically.