Robustness of Deep Learning Algorithm to Varying Imaging Conditions in Detecting Low Contrast Objects In Computed Tomography Phantom Images: in Comparison to 12 Radiologists.

Dong Yul Oh,Won Chang,Yoon Jin Lee,Sungsoo Lee,Kyeorye Lee,Leonard Sunwoo,Hae Young Kim,Young Hoon Kim,Youngjune Kim

doi:10.3390/diagnostics11030410

Dong Yul Oh, Won Chang + Show 7 more

Open Access

https://doi.org/10.3390/diagnostics11030410

Copy DOI

Abstract

The performance of deep learning algorithm (DLA) to that of radiologists was compared in detecting low contrast objects in CT phantom images under various imaging conditions. For training, 10,000 images were created using American College of Radiology CT phantom as the background. In half of the images, objects of 3–20 mm size and 5–30 HU contrast difference were generated in random locations. Binary responses were used as the ground truth. For testing, 640 images of Catphan® phantom were used, half of which had objects of either 5 or 9 mm size with 10 HU contrast difference. Twelve radiologists evaluated the presence of objects on a five-point scale. The performances of the DLA and radiologists were compared across different imaging conditions in terms of area under receiver operating characteristics curve (AUC). Multi-reader multi-case AUC and Hanley and McNeil tests were used. We performed post-hoc analysis using bootstrapping and verified that the DLA is less affected by the changing imaging conditions. The AUC of DLA was consistently higher than those of the radiologists across different imaging conditions (p < 0.0001), and it was less affected by varying imaging conditions. The DLA outperformed the radiologists and showed more robust performance under varying imaging conditions.

Highlights

The increasing role of imaging in diagnostic processes, along with technological advances facilitating access to imaging, has resulted in an unprecedented amount of clinical workload for radiologists [1,2]
While it is well known that performance of radiologists is affected substantially by changes in imaging conditions, such as radiation dose, object size, or the reconstruction algorithm used [11,12,13,14], more research is demanded regarding whether and to what degree the performance of Deep learning (DL) techniques is affected by such variations in imaging conditions
We fill the existing knowledge gap in literature by confirming that DL algorithms can be robust to changing imaging conditions when detecting low-contrast objects

Summary

Introduction

The increasing role of imaging in diagnostic processes, along with technological advances facilitating access to imaging, has resulted in an unprecedented amount of clinical workload for radiologists [1,2] This has led to increasing interest in the medical society in developing techniques for automated imaging analysis that may improve the efficiency of radiological diagnosis [3]. An area of active research in DL-based imaging analysis has been the development of techniques for object detection on computed tomography (CT) These techniques have shown promising performance in previous studies [6,7,8,9], more research should be conducted on validating their robustness before they could be utilized in daily clinical practice. While it is well known that performance of radiologists is affected substantially by changes in imaging conditions, such as radiation dose, object size, or the reconstruction algorithm used [11,12,13,14], more research is demanded regarding whether and to what degree the performance of DL techniques is affected by such variations in imaging conditions

Methods

Results

Conclusion