Clinical evaluation of a deep-learning-based computer-aided detection system for the detection of pulmonary nodules in a large teaching hospital

Abstract

To evaluate a deep-learning-based computer-aided detection (DL-CAD) software system for pulmonary nodule detection on computed tomography (CT) images and assess its added value in the clinical practice of a large teaching hospital. A retrospective analysis was performed of 145 chest CT examinations by comparing the output of the DL-CAD software with a reference standard based on the consensus reading of three radiologists. For every nodule in each scan, the location, composition, and maximum diameter in the axial plane were recorded. The subgroup of chest CT examinations (n = 97) without any nodules was used to determine the negative predictive value at the given clinical sensitivity threshold setting. The radiologists found 91 nodules and the CAD system 130 nodules of which 80 were true positive. The measured sensitivity was 88% and the mean false-positive rate was 1.04 false positives/scan. The negative predictive value was 95%. For 23 nodules, there was a size discrepancy of which 19 (83%) were measured smaller by the radiologist. The agreement of nodule composition between the CAD results and the reference standard was 95%. The present study found a sensitivity of 88% and a false-positive rate of 1.04 false positives/scan, which match the vendor specification. Together with the measured negative predictive value of 95% the system performs very well; however, these rates are still not good enough to replace the radiologist, even for the specific task of nodule detection. Furthermore, a surprisingly high rate of overestimation of nodule size was observed, which can lead to too many follow-up examinations.