Proper ROC analysis and joint ROC analysis of the satisfaction of search effect in chest radiology

Abstract

The purpose of this study was to test whether the satisfaction of search (SOS) effect in chest radiology could be demonstrated with proper receiver operating characteristic (ROC) curves and with joint detection and localization ROC curves. Data from an earlier ROC study of SOS in chest radiology were analyzed with three proper ROC models and one ROC model for joint detection and localization. Fits of the models were compared on the basis of likelihood-ratio chi-squared statistics (G2). To examine further the validity of the SOS effect in chest radiology, the authors also replicated the earlier study with a new sample of readers, analyzing the new data with the same methods. The proper contaminated binormal model fit the data better than the other two proper ROC models. Contaminated binormal analysis of the earlier and the replication experiment demonstrated an SOS effect: a reduction in area under the ROC curve for detection of the native abnormalities with the addition of nodules. Similarly, joint ROC analysis producing curves that appropriately cross the chance line gave similar results. Preventing inappropriate chance line crossing reduces measurement error and provides more powerful statistical tests. Results of both experiments showed that the SOS effect in chest radiology can be demonstrated with ROC methods that avoid inappropriate crossing of the chance line.