Free-response Receiver Operating Characteristic Curve Research Articles

Assessment of Two 3D MDCT Colonography Protocols for Observation of Colorectal Polyps

The objective of our study was to assess the value of two-way interpretation (i.e., from rectum to cecum and vice versa) compared with one-way interpretation (i.e., from rectum to cecum only) in terms of polyp detection and interpretation time on MDCT colonography. Fifty consecutive patients underwent both CT colonography and conventional colonoscopy. Three radiologists independently analyzed the CT colonographic examinations of each patient using a primary 3D method. All examinations were analyzed using two techniques: navigation from rectum to cecum only (one-way) and navigation from rectum to cecum and vice versa (two-way). Sensitivity and positive predictive value were calculated on both a per-polyp basis and a per-patient basis. Alternative free-response receiver operating characteristic (ROC) curve analysis was estimated, and image interpretation time was documented. One hundred fifty-five polyps were depicted in 45 patients by colonoscopy. The mean sensitivity of CT colonography for polyp detection with two-way (88.4%) was significantly superior to that with one-way (78.1%) (p < 0.01). The mean positive predictive value of each observer with one-way was 66.7%, whereas that with two-way was 65.8%. The mean area under the alternative free-response ROC curve (A(z) value) with two-way (0.827) was higher than that with one-way (0.816), but there was not a statistically significant difference. The average interpretation time of each observer with two-way (39 min) was statistically significantly longer than that with one-way (25 min) (p < 0.01). When using a primary 3D interpretation technique at CT colonography, complete 3D navigation from rectum to cecum and from cecum to rectum is mandatory to maximize polyp detection. The image interpretation time for two-way interpretation is statistically significantly longer than that with one-way interpretation.

Estimation and Comparison of CAD System Performance in Clinical Settings 1

Computer-aided detection (CAD) systems are frequently compared using free-response receiver operating characteristic (FROC) curves. While there are ample statistical methods for comparing FROC curves, when one is interested in comparing the outcomes of 2 CAD systems applied in a typical clinical setting, there is the additional matter of correctly determining the system operating point. This article shows how the effect of the sampling error on determining the correct CAD operating point can be captured. By incorporating this uncertainty, a method is presented that allows estimation of the probability with which a particular CAD system performs better than another on unseen data in a clinical setting. The distribution of possible clinical outcomes from 2 artificial CAD systems with different FROC curves is examined. The sampling error is captured by the distribution of possible system thresholds of the classifying machine that yields a specified sensitivity. After introducing a measure of superiority, the probability of one system being superior to the other can be determined. It is shown that for 2 typical mammography CAD systems, each trained on independent representative datasets of 100 cases, the FROC curves must be separated by 0.20 false positives per image in order to conclude that there is a 90% probability that one is better than the other in a clinical setting. Also, there is no apparent gain in increasing the size of the training set beyond 100 cases. CAD systems for mammography are modeled for illustrative purposes, but the method presented is applicable to any computer-aided detection system evaluated with FROC curves. The presented method is designed to construct confidence intervals around possible clinical outcomes and to assess the importance of training set size and separation between FROC curves of systems trained on different datasets.

Pulmonary Nodules on Multi–Detector Row CT Scans: Performance Comparison of Radiologists and Computer-aided Detection

To compare the performance of radiologists and of a computer-aided detection (CAD) algorithm for pulmonary nodule detection on thin-section thoracic computed tomographic (CT) scans. The study was approved by the institutional review board. The requirement of informed consent was waived. Twenty outpatients (age range, 15-91 years; mean, 64 years) were examined with chest CT (multi-detector row scanner, four detector rows, 1.25-mm section thickness, and 0.6-mm interval) for pulmonary nodules. Three radiologists independently analyzed CT scans, recorded the locus of each nodule candidate, and assigned each a confidence score. A CAD algorithm with parameters chosen by using cross validation was applied to the 20 scans. The reference standard was established by two experienced thoracic radiologists in consensus, with blind review of all nodule candidates and free search for additional nodules at a dedicated workstation for three-dimensional image analysis. True-positive (TP) and false-positive (FP) results and confidence levels were used to generate free-response receiver operating characteristic (ROC) plots. Double-reading performance was determined on the basis of TP detections by either reader. The 20 scans showed 195 noncalcified nodules with a diameter of 3 mm or more (reference reading). Area under the alternative free-response ROC curve was 0.54, 0.48, 0.55, and 0.36 for CAD and readers 1-3, respectively. Differences between reader 3 and CAD and between readers 2 and 3 were significant (P < .05); those between CAD and readers 1 and 2 were not significant. Mean sensitivity for individual readings was 50% (range, 41%-60%); double reading resulted in increase to 63% (range, 56%-67%). With CAD used at a threshold allowing only three FP detections per CT scan, mean sensitivity was increased to 76% (range, 73%-78%). CAD complemented individual readers by detecting additional nodules more effectively than did a second reader; CAD-reader weighted kappa values were significantly lower than reader-reader weighted kappa values (Wilcoxon rank sum test, P < .05). With CAD used at a level allowing only three FP detections per CT scan, sensitivity was substantially higher than with conventional double reading.

Lung cancer: performance of automated lung nodule detection applied to cancers missed in a CT screening program.

To evaluate the performance of a fully automated computerized method for the detection of lung nodules in computed tomographic (CT) scans in the identification of lung cancers that may be missed during visual interpretation. A database of 38 low-dose CT scans with 50 lung nodules was obtained from a lung cancer screening program. Thirty-eight of the nodules represented biopsy-confirmed lung cancers that had not been reported during initial clinical interpretation. A computer detection method that involved the use of gray-level thresholding techniques to identify three-dimensionally contiguous structures within the lungs was applied to the CT data. Computer-extracted volume was used to determine whether a structure became a nodule candidate. A rule-based scheme and a cascaded automated classifier were applied to the set of nodule candidates to distinguish actual nodules from areas of normal anatomy. Overall performance of the computer detection method was evaluated with free-response receiver operating characteristic (FROC) analysis. At a specific operating point on the FROC curve, the method achieved a sensitivity of 80% (40 of 50 nodules), with an average of 1.0 false-positive detection per section. Missed cancers were detected by the computerized method with a sensitivity of 84% (32 of 38 nodules) and a false-positive rate of 1.0 per section. With an automated lung nodule detection method, a large fraction (84%, 32 of 38) of missed cancers in a database of low-dose CT scans were detected correctly.

Impact of attenuation correction on the accuracy of FDG-PET in patients with abdominal tumors: a free-response ROC analysis.

The aim of this study was to evaluate image quality and lesion detectability with and without attenuation correction in patients with abdominal tumors, using a free-response receiver operating characteristic (FROC) methodology. Thirty-four patients with various abdominal tumors were evaluated (11 men, 23 women, median age 48 years). Whole-body emission scans were performed 68 min (35-102 min) after intravenous injection of 4.3 MBq/kg fluorine-18 fluorodeoxyglucose (FDG). Images were reconstructed using the OS-EM algorithm and corrected for attenuation either using postinjection singles transmission (n=27) or by calculation and body outline (n=7). Total scan duration did not exceed 70 min. Studies were read independently by four observers unaware of any clinical data. The uncorrected (UC) images were systematically read before the attenuation-corrected (AC) images. All studies were given an image quality score ranging from 1 (unreadable) to 5 (excellent). Each focus of increased activity was then localized and given a probability of malignancy using a five-point scale. The average image quality score was similar for both UC and AC images. At the time of the positron emission tomography (PET) scans, 127 lesions (63 liver metastases, 9 retroperitoneal lesions, 50 peritoneal or bowel lesions, and 5 pancreatic carcinomas) were revealed by pathological or correlative studies. The areas under the FROC curves were consistently greater for AC images (range 0.8663-0.8867) than for UC images (range 0.7774 -0.8613). Overall, the difference between the AC images and the UC images was significant (P=0.019). In particular, correction for attenuation increased the sensitivity regardless of the location of the lesions. In conclusion, correction for attenuation significantly improves the diagnostic accuracy of FDG-PET for abdominal staging of neoplasms, without impairing the image quality.