FROC Curves Research Articles

A Systematic Review of Deep-Learning Methods for Intracranial Aneurysm Detection in CT Angiography.

Background: Subarachnoid hemorrhage resulting from cerebral aneurysm rupture is a significant cause of morbidity and mortality. Early identification of aneurysms on Computed Tomography Angiography (CTA), a frequently used modality for this purpose, is crucial, and artificial intelligence (AI)-based algorithms can improve the detection rate and minimize the intra- and inter-rater variability. Thus, a systematic review and meta-analysis were conducted to assess the diagnostic accuracy of deep-learning-based AI algorithms in detecting cerebral aneurysms using CTA. Methods: PubMed (MEDLINE), Embase, and the Cochrane Library were searched from January 2015 to July 2023. Eligibility criteria involved studies using fully automated and semi-automatic deep-learning algorithms for detecting cerebral aneurysms on the CTA modality. Eligible studies were assessed using the Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) guidelines and the Quality Assessment of Diagnostic Accuracy Studies 2 (QUADAS-2) tool. A diagnostic accuracy meta-analysis was conducted to estimate pooled lesion-level sensitivity, size-dependent lesion-level sensitivity, patient-level specificity, and the number of false positives per image. An enhanced FROC curve was utilized to facilitate comparisons between the studies. Results: Fifteen eligible studies were assessed. The findings indicated that the methods exhibited high pooled sensitivity (0.87, 95% confidence interval: 0.835 to 0.91) in detecting intracranial aneurysms at the lesion level. Patient-level sensitivity was not reported due to the lack of a unified patient-level sensitivity definition. Only five studies involved a control group (healthy subjects), whereas two provided information on detection specificity. Moreover, the analysis of size-dependent sensitivity reported in eight studies revealed that the average sensitivity for small aneurysms (<3 mm) was rather low (0.56). Conclusions: The studies included in the analysis exhibited a high level of accuracy in detecting intracranial aneurysms larger than 3 mm in size. Nonetheless, there is a notable gap that necessitates increased attention and research focus on the detection of smaller aneurysms, the use of a common test dataset, and an evaluation of a consistent set of performance metrics.

A Lower False Positive Pulmonary Nodule Detection Approach for Early Lung Cancer Screening.

Pulmonary nodule detection with low-dose computed tomography (LDCT) is indispensable in early lung cancer screening. Although existing methods have achieved excellent detection sensitivity, nodule detection still faces challenges such as nodule size variation and uneven distribution, as well as excessive nodule-like false positive candidates in the detection results. We propose a novel two-stage nodule detection (TSND) method. In the first stage, a multi-scale feature detection network (MSFD-Net) is designed to generate nodule candidates. This includes a proposed feature extraction network to learn the multi-scale feature representation of candidates. In the second stage, a candidate scoring network (CS-Net) is built to estimate the score of candidate patches to realize false positive reduction (FPR). Finally, we develop an end-to-end nodule computer-aided detection (CAD) system based on the proposed TSND for LDCT scans. Experimental results on the LUNA16 dataset show that our proposed TSND obtained an excellent average sensitivity of 90.59% at seven predefined false positives (FPs) points: 0.125, 0.25, 0.5, 1, 2, 4, and 8 FPs per scan on the FROC curve introduced in LUNA16. Moreover, comparative experiments indicate that our CS-Net can effectively suppress false positives and improve the detection performance of TSND.

Medical image augmentation for lesion detection using a texture-constrained multichannel progressive GAN

Lesion detectors based on deep learning can assist doctors in diagnosing diseases. However, the performance of current detectors is likely to be unsatisfactory due to the scarcity of training samples. Therefore, it is beneficial to use image generation to augment the training set of a detector. However, when the imaging texture of the medical image is relatively delicate, the synthesized image generated by an existing method may be too poor in quality to meet the training requirements of the detectors. In this regard, a medical image augmentation method, namely, a texture-constrained multichannel progressive generative adversarial network (TMP-GAN), is proposed in this work. TMP-GAN uses joint training of multiple channels to effectively avoid the typical shortcomings of the current generation methods. It also uses an adversarial learning-based texture discrimination loss to further improve the fidelity of the synthesized images. In addition, TMP-GAN employs a progressive generation mechanism to steadily improve the accuracy of the medical image synthesizer. Experiments on the publicly available dataset CBIS-DDMS and our pancreatic tumor dataset show that the precision/recall/F1-score of the detector trained on the TMP-GAN augmented dataset improves by 2.59%/2.70%/2.77% and 2.44%/2.06%/2.36%, respectively, compared to the optimal results of other data augmentation methods. The FROC curve of the detector is also better than the curve from the contrast-augmented trained dataset. Therefore, we believe the proposed TMP-GAN is a practical technique to efficiently implement lesion detection case studies.

Detection of microaneurysms in color fundus images based on local Fourier transform

Retinal microaneurysm (MA) is one of the early clinical signs of diabetic retinopathy (DR), which is essential for the early diagnosis of DR. Its shape is often shown as a red dot, which makes detection of MA challenging because of its tiny size. There are many effective MA automatic detection methods having been proposed, but these methods are hard to describe the overall shape of Mas. This paper proposes an automatic MA detection method based on local Fourier transform which splits the original image into small independent images and calculates their local Fourier transforms in isolation. After that, the object is not the whole image but a series of independent small images. We also extract a statistical feature to check for normality and design a new feature to represent the difference among every direction. After that, a group of candidate points are extracted from the images. Finally, some features are extracted from the candidate points, and the confidence scores of the candidate points are obtained by using random forest algorithm for classification. The proposed method obtained a FROC curve score of 0.847 on the E-Ophtha MA database, which is better than most existing methods. In the ROC database, although the FROC curve score is only 0.283, the AUC can reach 0.961, which is better than most methods. It can be seen that the proposed method can achieve a better result with low noise and large sample size.

Rapid measurement of the low contrast detectability of CT scanners.

Low contrast detectability (LCD) is a metric of fundamental importance in computed tomography (CT) imaging. In spite of this, its measurement is challenging in the context of nonlinear data processing. We introduce a new framework for objectively characterizing LCD with a single scan of a special-purpose phantom and automated analysis software. The output of the analysis software is a "machine LCD" metric which is more representative of LCD than contrast-noise ratio (CNR). It is not intended to replace human observer or model observer studies. Following preliminary simulations, we fabricated a phantom containing hundreds of low-contrast beads. These beads are acrylic spheres (1.6mm, net contrast ~10HU) suspended and randomly dispersed in a background matrix of nylon pellets and isoattenuating saline. The task was to search for and localize the beads. A modified matched filter was used to automatically scan the reconstruction and select candidate bead localizations of varying confidence. These were compared to bead locations as determined from a high-dose reference scan to produce free-response ROC curves. We compared iterative reconstruction (IR) and filtered backpropagation (FBP) at multiple dose levels between 40 and 240mAs. The scans at 60, 120, and 180mAs were performed three times each to estimate uncertainty. Experimental scans demonstrated the feasibility of our technique. Our metric for machine LCD was the area under the exponential transform of the FROC curve (AUC). AUC increased monotonically from 0.21 at 40mAs to 0.84 at 240mAs. The sample standard deviation of AUC was approximately 0.02. This measurement uncertainty in AUC corresponded to a change in tube current of 4% to 8%. Surprisingly, we found that AUCs for IR were slightly worse than AUCs for FBP. While the phantom was sufficient for these experiments, it contained small air bubbles and alternative fabrication methods will be necessary for widespread utilization. It is feasible to measure machine LCD using a search task on a phantom with hundreds of beads and to obtain tight error bars using only a single scan. Our method could facilitate routine quality assurance or possibly enable comparisons between different protocols and scanners.

Breast Mass Detection With Faster R-CNN: On the Feasibility of Learning From Noisy Annotations

In this work we study the impact of noise on the training of object detection networks for the medical domain, and how it can be mitigated by improving the training procedure. Annotating large medical datasets for training data-hungry deep learning models is expensive and time consuming. Leveraging information that is already collected in clinical practice, in the form of text reports, bookmarks or lesion measurements would substantially reduce this cost. Obtaining precise lesion bounding boxes through automatic mining procedures, however, is difficult. We provide here a quantitative evaluation of the effect of bounding box coordinate noise on the performance of Faster R-CNN object detection networks for breast mass detection. Varying degrees of noise are simulated by randomly modifying the bounding boxes: in our experiments, bounding boxes could be enlarged up to six times the original size. The noise is injected in the CBIS-DDSM collection, a well curated public mammography dataset for which accurate lesion location is available. We show how, due to an imperfect matching between the ground truth and the network bounding box proposals, the noise is propagated during training and reduces the ability of the network to correctly classify lesions from background. When using the standard Intersection over Union criterion, the area under the FROC curve decreases by up to 9%. A novel matching criterion is proposed to improve tolerance to noise.

Automatic detection and classification of rib fractures based on patients' CT images and clinical information via convolutional neural network.

To develop a convolutional neural network (CNN) model for the automatic detection and classification of rib fractures in actual clinical practice based on cross-modal data (clinical information and CT images). In this retrospective study, CT images and clinical information (age, sex and medical history) from 1020 participants were collected and divided into a single-centre training set (n = 760; age: 55.8 ± 13.4years; men: 500), a single-centre testing set (n = 134; age: 53.1 ± 14.3years; men: 90), and two independent multicentre testing sets from two different hospitals (n = 62, age: 57.97 ± 11.88, men: 41; n = 64, age: 57.40 ± 13.36, men: 35). A Faster Region-based CNN (Faster R-CNN) model was applied to integrate CT images and clinical information. Then, a result merging technique was used to convert 2D inferences into 3D lesion results. The diagnostic performance was assessed on the basis of the receiver operating characteristic (ROC) curve, free-response ROC (fROC) curve, precision, recall (sensitivity), F1-score, and diagnosis time. The classification performance was evaluated in terms of the area under the ROC curve (AUC), sensitivity, and specificity. The CNN model showed improved performance on fresh, healing, and old fractures and yielded good classification performance for all three categories when both clinical information and CT images were used compared to the use of CT images alone. Compared with experienced radiologists, the CNN model achieved higher sensitivity (mean sensitivity: 0.95 > 0.77, 0.89 > 0.61 and 0.80 > 0.55), comparable precision (mean precision: 0.91 > 0.87, 0.84 > 0.77, and 0.95 > 0.70), and a shorter diagnosis time (average reduction of 126.15s). A CNN model combining CT images and clinical information can automatically detect and classify rib fractures with good performance and feasibility in actual clinical practice. • The developed convolutional neural network (CNN) performed better in fresh, healing, and old fractures and yielded a good classification performance in three categories, if both (clinical information and CT images) were used compared to CT images alone. • The CNN model had a higher sensitivity and matched precision in three categoriesthan experienced radiologists with a shorter diagnosis time in actual clinical practice.

NF-RCNN: Heart localization and right ventricle wall motion abnormality detection in cardiac MRI

Convolutional neural networks (CNNs) are extensively used in cardiac image analysis. However, heart localization has become a prerequisite to these networks since it decreases the size of input images. Accordingly, recent CNNs benefit from deeper architectures in gaining abstract semantic information. In the present study, a deep learning-based method was developed for heart localization in cardiac MR images. Further, Network in Network (NIN) was used as the region proposal network (RPN) of the faster R-CNN, and then NIN Faster-RCNN (NF-RCNN) was proposed. NIN architecture is formed based on “MLPCONV” layer, a combination of convolutional network and multilayer perceptron (MLP). Therefore, it could deal with the complicated structures of MR images. Furthermore, two sets of cardiac MRI dataset were used to evaluate the network, and all the evaluation metrics indicated an absolute superiority of the proposed network over all related networks. In addition, FROC curve, precision-recall (PR) analysis, and mean localization error were employed to evaluate the proposed network. In brief, the results included an AUC value of 0.98 for FROC curve, a mean average precision of 0.96 for precision-recall curve, and a mean localization error of 6.17 mm. Moreover, a deep learning-based approach for the right ventricle wall motion analysis (WMA) was performed on the first dataset and the effect of the heart localization on this algorithm was studied. The results revealed that NF-RCNN increased the speed and decreased the required memory significantly.

A new optical density granulometry-based descriptor for the classification of prostate histological images using shallow and deep Gaussian processes

Background and objectiveProstate cancer is one of the most common male tumors. The increasing use of whole slide digital scanners has led to an enormous interest in the application of machine learning techniques to histopathological image classification. Here we introduce a novel family of morphological descriptors which, extracted in the appropriate image space and combined with shallow and deep Gaussian process based classifiers, improves early prostate cancer diagnosis. MethodWe decompose the acquired RGB image in its RGB and optical density hematoxylin and eosin components. Then, we define two novel granulometry-based descriptors which work in both, RGB and optical density, spaces but perform better when used on the latter. In this space they clearly encapsulate knowledge used by pathologists to identify cancer lesions. The obtained features become the inputs to shallow and deep Gaussian process classifiers which achieve an accurate prediction of cancer. ResultsWe have used a real and unique dataset. The dataset is composed of 60 Whole Slide Images. For a five fold cross validation, shallow and deep Gaussian Processes obtain area under ROC curve values higher than 0.98. They outperform current state of the art patch based shallow classifiers and are very competitive to the best performing deep learning method. Models were also compared on 17 Whole Slide test Images using the FROC curve. With the cost of one false positive, the best performing method, the one layer Gaussian process, identifies 83.87% (sensitivity) of all annotated cancer in the Whole Slide Image. This result corroborates the quality of the extracted features, no more than a layer is needed to achieve excellent generalization results. ConclusionTwo new descriptors to extract morphological features from histological images have been proposed. They collect very relevant information for cancer detection. From these descriptors, shallow and deep Gaussian Processes are capable of extracting the complex structure of prostate histological images. The new space/descriptor/classifier paradigm outperforms state-of-art shallow classifiers. Furthermore, despite being much simpler, it is competitive to state-of-art CNN architectures both on the proposed SICAPv1 database and on an external database.

P3.13-037 Deep Learning System for Lung Nodule Detection

Ever since its first success in large scale image recognition problem, deep convolutional neural networks (DCNNs) have shown their capabilities in solving many challenging visual perceptual tasks, such as image classification, segmentation, object detection. In some cases, DCNNs have already achieved near-human performance. In medical image analysis, DCNNs have also been successfully applied to lesion detection, segmentation, and diagnosis. The power of DCNN lies in its ability to learn a hierarchical representation of raw input data, without hand-crafted features. The focus of this study was to improve the performance of DCNN in automatic detection of pulmonary nodule on CT scan. In particular, nodules whose size is less than 4mm were considered. A total of 171 scans were collected at Zhongshan Hospital Fudan University, which was first process by the proposed DCNN system (12Sigma). The detection results were then carefully reviewed by an experienced physician, all false positives and missed nodules were manually labeled. To refine the system, all false positives and true nodules of sizes 4mm and under were selected. The data set was randomly split into training and test sets, where the training set consists of 90% (154 scans) and the test set consists of 10% of data (17 scans). Fig 1 shows the FROC curves on the test set before and after re-training. Overall, the detection sensitivity improved at all false positive levels, but the improvement was most significant at low FP rate region. For example, when FP is 0.5 per scan, the detection sensitivity increased from 0.36 to 0.49. This improvement suggests that, even with limited data, the deep neural network can learn from its mistakes and be easily tuned to be more sensitive to small nodules. We believe that when more data is collected, more significant improve in high FP rate region will be observed.

Due to potential concerns of bias and conflicts of interest, regulatory bodies should not do evaluation methodology research related to their regulatory missions.

Due to potential concerns of bias and conflicts of interest, regulatory bodies should not do evaluation methodology research related to their regulatory missions.

Quantitative comparison of clustered microcalcifications in for-presentation and for-processing mammograms in full-field digital mammography.

Mammograms acquired with full-field digital mammography (FFDM) systems are provided in both "for-processing'' and "for-presentation'' image formats. For-presentation images are traditionally intended for visual assessment by the radiologists. In this study, we investigate the feasibility of using for-presentation images in computerized analysis and diagnosis of microcalcification (MC) lesions. We make use of a set of 188 matched mammogram image pairs of MC lesions from 95 cases (biopsy proven), in which both for-presentation and for-processing images are provided for each lesion. We then analyze and characterize the MC lesions from for-presentation images and compare them with their counterparts in for-processing images. Specifically, we consider three important aspects in computer-aided diagnosis (CAD) of MC lesions. First, we quantify each MC lesion with a set of 10 image features of clustered MCs and 12 textural features of the lesion area. Second, we assess the detectability of individual MCs in each lesion from the for-presentation images by a commonly used difference-of-Gaussians (DoG) detector. Finally, we study the diagnostic accuracy in discriminating between benign and malignant MC lesions from the for-presentation images by a pretrained support vector machine (SVM) classifier. To accommodate the underlying background suppression and image enhancement in for-presentation images, a normalization procedure is applied. The quantitative image features of MC lesions from for-presentation images are highly consistent with that from for-processing images. The values of Pearson's correlation coefficient between features from the two formats range from 0.824 to 0.961 for the 10 MC image features, and from 0.871 to 0.963 for the 12 textural features. In detection of individual MCs, the FROC curve from for-presentation is similar to that from for-processing. In particular, at sensitivity level of 80%, the average number of false-positives (FPs) per image region is 9.55 for both for-presentation and for-processing images. Finally, for classifying MC lesions as malignant or benign, the area under the ROC curve is 0.769 in for-presentation, compared to 0.761 in for-processing (P = 0.436). The quantitative results demonstrate that MC lesions in for-presentation images are highly consistent with that in for-processing images in terms of image features, detectability of individual MCs, and classification accuracy between malignant and benign lesions. These results indicate that for-presentation images can be compatible with for-processing images for use in CAD algorithms for MC lesions.

Detection of masses in mammograms with adaption to breast density using genetic algorithm, phylogenetic trees, LBP and SVM

Breast cancer is the second commonest type of cancer in the world, and the commonest among women, corresponding to 22% of the new cases every year. This work presents a new computational methodology, which helps the specialists in the detection of breast masses based on the breast density. The proposed methodology is divided into stages with the objective of overcoming several difficulties associated with the detection of masses. In many of these stages, we brought contributions to the areas. The first stage is intended to detect the type of density of the breast, which can be either dense or non-dense. We proposed an adaptive algorithm capable of analyzing and image and telling if it is dense or non-dense. The first stage consists in the segmentation of the regions that look like masses. We propose a novel use of the micro-genetic algorithm to create a texture proximity mask and select the regions suspect of containing lesions. The next stage is the reduction of false positives, which were generated in the previous stage. To this end, we proposed two new approaches. The first reduction of false positives used DBSCAN and a proximity ranking of the textures extracted from the ROIs. In the second reduction of false positives, the resulting regions have their textures analyzed by the combination of Phylogenetic Trees, Local Binary Patterns and Support Vector Machines (SVM). A micro-genetic algorithm was used to choose the suspect regions that generate the best training models and maximize the classification of masses and non-masses used in the SVM. The best result produced a sensitivity of 92.99%, a rate of 0.15 false positives per image and an area under the FROC curve of 0.96 in the analysis of the non-dense breasts; and a sensitivity of 83.70%, a rate of 0.19 false positives per image and an area under the FROC curve of 0.85, in the analysis of the dense breasts.

A Comprehensive Computer-Aided Polyp Detection System for Colonoscopy Videos.

Computer-aided detection (CAD) can help colonoscopists reduce their polyp miss-rate, but existing CAD systems are handicapped by using either shape, texture, or temporal information for detecting polyps, achieving limited sensitivity and specificity. To overcome this limitation, our key contribution of this paper is to fuse all possible polyp features by exploiting the strengths of each feature while minimizing its weaknesses. Our new CAD system has two stages, where the first stage builds on the robustness of shape features to reliably generate a set of candidates with a high sensitivity, while the second stage utilizes the high discriminative power of the computationally expensive features to effectively reduce false positives. Specifically, we employ a unique edge classifier and an original voting scheme to capture geometric features of polyps in context and then harness the power of convolutional neural networks in a novel score fusion approach to extract and combine shape, color, texture, and temporal information of the candidates. Our experimental results based on FROC curves and a new analysis of polyp detection latency demonstrate a superiority over the state-of-the-art where our system yields a lower polyp detection latency and achieves a significantly higher sensitivity while generating dramatically fewer false positives. This performance improvement is attributed to our reliable candidate generation and effective false positive reduction methods.

Statistical atlas based exudate segmentation

Diabetic macular edema (DME) is characterized by hard exudates. In this article, we propose a novel statistical atlas based method for segmentation of such exudates. Any test fundus image is first warped on the atlas co-ordinate and then a distance map is obtained with the mean atlas image. This leaves behind the candidate lesions. Post-processing schemes are introduced for final segmentation of the exudate. Experiments with the publicly available HEI-MED data-set shows good performance of the method. A lesion localization fraction of 82.5% at 35% of non-lesion localization fraction on the FROC curve is obtained. The method is also compared to few most recent reference methods.

Automated Feature Set Selection and Its Application to MCC Identification in Digital Mammograms for Breast Cancer Detection

We propose a fully automated algorithm that is able to select a discriminative feature set from a training database via sequential forward selection (SFS), sequential backward selection (SBS), and F-score methods. We applied this scheme to microcalcifications cluster (MCC) detection in digital mammograms for early breast cancer detection. The system was able to select features fully automatically, regardless of the input training mammograms used. We tested the proposed scheme using a database of 111 clinical mammograms containing 1,050 microcalcifications (MCs). The accuracy of the system was examined via a free response receiver operating characteristic (fROC) curve of the test dataset. The system performance for MC identifications was Az = 0.9897, the sensitivity was 92%, and 0.65 false positives (FPs) were generated per image for MCC detection.

An approach for computer-aided detection of brain metastases in post-Gd T1-W MRI

PurposeTo develop an approach for computer-aided detection (CAD) of small brain metastases in post-Gd T1-weighted magnetic resonance imaging (MRI). MethodA set of unevenly spaced 3D spherical shell templates was optimized to localize brain metastatic lesions by cross-correlation analysis with MRI. Theoretical and simulation analyses of effects of lesion size and shape heterogeneity were performed to optimize the number and size of the templates and the cross-correlation thresholds. Also, effects of image factors of noise and intensity variation on the performance of the CAD system were investigated. A nodule enhancement strategy to improve sensitivity of the system and a set of criteria based upon the size, shape and brightness of lesions were used to reduce false positives. An optimal set of parameters from the FROC curves was selected from a training dataset, and then the system was evaluated on a testing dataset including 186 lesions from 2753 MRI slices. Reading results from two radiologists are also included. ResultsOverall, a 93.5% sensitivity with 0.024 of intra-cranial false positive rate (IC-FPR) was achieved in the testing dataset. Our investigation indicated that nodule enhancement was very effective in improving both sensitivity and specificity. The size and shape criteria reduced the IC-FPR from 0.075 to 0.021, and the brightness criterion decreases the extra-cranial FPR from 0.477 to 0.083 in the training dataset. Readings from the two radiologists had sensitivities of 60% and 67% in the training dataset and 70% and 80% in the testing dataset for the metastatic lesions <5 mm in diameter. ConclusionOur proposed CAD system has high sensitivity and fairly low FPR for detection of the small brain metastatic lesions in MRI compared to the previous work and readings of neuroradiologists. The potential of this method for assisting clinical decision- making warrants further evaluation and improvements.

Lung nodule detection in low-dose and thin-slice computed tomography

A computer-aided detection (CAD) system for the identification of small pulmonary nodules in low-dose and thin-slice CT scans has been developed. The automated procedure for selecting the nodule candidates is mainly based on a filter enhancing spherical-shaped objects. A neural approach based on the classification of each single voxel of a nodule candidate has been purposely developed and implemented to reduce the amount of false-positive findings per scan. The CAD system has been trained to be sensitive to small internal and sub-pleural pulmonary nodules collected in a database of low-dose and thin-slice CT scans. The system performance has been evaluated on a data set of 39 CT containing 75 internal and 27 sub-pleural nodules. The FROC curve obtained on this data set shows high values of sensitivity to lung nodules (80–85% range) at an acceptable level of false positive findings per patient (10–13 FP/scan).

TH-C-L100F-09: Operating Characteristics Predicted by Models of Location Specific Diagnostic Tasks

Operating characteristics summarize observer performance in diagnostic tasks. Examples are the ROC, the LROC, the FROC, and the AFROC. Models of observer performance differ in their abilities to predict operating characteristics. For example, ROC models can predict the ROC curve, but cannot predict LROC, FROC or AFROC curves. Location specific models are, in historical order, the FROCFIT model, Swensson's LROC model and the search model (SM). In principle any of these models can predict all of the characteristic curves noted above. The purpose of this work was to compare the ability of these models to predict operating characteristics. CAD data consisting of mark‐rating pairs on 450 images were used to determine the various operating characteristics. For example, the highest rating on each image determined the ROC curve; the lesions rated higher than the highest noise determined the ordinate of the LROC, etc. All models agreed with the observed ROC data. Swensson's model agreed with the observed LROC curves but departed mildly from the observed AFROC curve and strongly from the observed FROC curve. The FROCFIT model departed mildly from the AFROC curve and strongly from the FROC curve. The search model provided good fits to all operating characteristics. In conclusion this work provides additional validation of the search model method of analyzing free‐response data.

TH-C-L100F-07: FROC Curves Without Arbitrary Scoring of Marks: Perceptual Vs. Scored Analysis of CAD Data

A free‐response study generates data in the form of mark‐rating pairs. Any mark closer to a lesion than the acceptance radius (or closeness or overlap criterion) is scored as lesion localization and other marks are scored as non‐lesion localizations. A scored FROC curve is a plot of lesion localization fraction vs. non‐lesion localization fraction. Because of the inherently arbitrary nature of the acceptance radius choice, the FROC curve is also arbitrary. A method of statistically modeling the spatial distribution of the marks with respect to lesion centers is described. This model allows one to obtain statistical (i.e., maximum likelihood) estimates of the numbers of marks that were true lesion localizations as opposed to false localizations. These estimates can be used to plot a perceptual FROC curve that is totally independent of acceptance radius choice. The method was applied to several CAD datasets. It was found that the perceptual and scored curves were in good agreement in most cases, and minor deviations were attributed to some cases of questionable classifications of marks. CAD developers could use this method to evaluate their algorithms in a manner that makes inter‐comparisons more meaningful.