Comparing the number and relevance of false activations between 2 artificial intelligence computer-aided detection systems: the NOISE study

Abstract

Artificial intelligence has been shown to be effective in polyp detection, and multiple computer-aided detection (CADe) systems have been developed. False-positive (FP) activation emerged as a possible way to benchmark CADe performance in clinical practice. The aim of this study was to validate a previously developed classification of FPs comparing the performances of different brands of approved CADe systems. We compared 2 different consecutive video libraries (40 video per arm) collected at Humanitas Research Hospital with 2 different CADe system brands (CADe A and CADe B). For each video, the number of CADe false activations, cause, and time spent by the endoscopist to examine the area erroneously highlighted were reported. The FP activations were classified according to the previously developed classification of FPs (the NOISE classification) according to their cause and relevance. In CADe A 1021 FP activations were registered across the 40 videos (25.5 ± 12.2 FPs per colonoscopy), whereas in CADe B 1028 were identified (25.7 ± 13.2 FPs per colonoscopy; P= .53). Among them, 22.9 ± 9.9 (89.8% in CADe A) and 22.1 ± 10.0 (86.0% in CADe B) were because of artifacts from the bowel wall. Conversely, 2.6 ± 1.9 (10.2% in CADe A) and 3.5 ± 2.1 (14% in CADe B) were caused by bowel content (P= .45). Within CADe A each false activation required .2 ± .9 seconds, with 1.6 ± 1.0 FPs (6.3%) requiring additional time for endoscopic assessment. Comparable results were reported within CADe B with .2 ± .8 seconds spent per false activation and 1.8 ± 1.2 FPs per colonoscopy requiring additional inspection. The use of a standardized nomenclature provided comparable results with either of the 2 recently approved CADe systems. (Clinical trial registration number: NCT04399590.).