Receiver Operating Characteristic Method Research Articles

Validating and Improving Models for Vibratory Installation of Steel Sheet Piles with Field Observations

Vibratory driving is the most common installation technique for steel sheet pile walls. In practice, the assessment of the feasibility of this installation process is mainly based on rules of thumb, on numerical and empirical models or on experts opinions. In order to improve these prediction methods and formulas, 252 observations from the Dutch engineering practice have been compared with six different types of models. This comparison has been carried out applying the receiver operating characteristic (ROC) curve technique, which is new in geotechnical engineering. This paper introduces the ROC-curve technique to estimate mainly the quality of a model and to be able to optimize parameters and variables in the model. 252 field observations were used to re-examine prediction methods for the minimum required vibration force and to prove the ROC method works. The paper shows this technique is suitable for three purposes: (1) determining the quality of a model, (2) objectively comparing several models to each other, given certain assumptions and (3) for optimizing thresholds within a model. The model with added professionals’ experience proves to perform equally well as the numerical model Hypervib-I.

Quantifying the clinical relevance of a laboratory observer performance paradigm

Laboratory observer performance measurements, receiver operating characteristic (ROC) and free-response ROC (FROC) differ from actual clinical interpretations in several respects, which could compromise their clinical relevance. The objective of this study was to develop a method for quantifying the clinical relevance of a laboratory paradigm and apply it to compare the ROC and FROC paradigms in a nodule detection task. The original prospective interpretations of 80 digital chest radiographs were classified by the truth panel as correct (C=1) or incorrect (C=0), depending on correlation with additional imaging, and the average of C was interpreted as the clinical figure of merit. FROC data were acquired for 21 radiologists and ROC data were inferred using the highest ratings. The areas under the ROC and alternative FROC curves were used as laboratory figures of merit. Bootstrap analysis was conducted to estimate conventional agreement measures between laboratory and clinical figures of merit. Also computed was a pseudovalue-based image-level correctness measure of the laboratory interpretations, whose association with C as measured by the area (rAUC) under an appropriately defined relevance ROC curve, is as a measure of the clinical relevance of a laboratory paradigm. Low correlations (e.g. κ=0.244) and near chance level rAUC values (e.g. 0.598), attributable to differences between the clinical and laboratory paradigms, were observed. The absolute width of the confidence interval was 0.38 for the interparadigm differences of the conventional measures and 0.14 for the difference of the rAUCs. The rAUC measure was consistent with the traditional measures but was more sensitive to the differences in clinical relevance. A new relevance ROC method for quantifying the clinical relevance of a laboratory paradigm is proposed.

Linking the Tinnitus Questionnaire and the subjective Clinical Global Impression: Which differences are clinically important?

BackgroundDevelopment of new tinnitus treatments requires prospective placebo-controlled randomized trials to prove their efficacy. The Tinnitus Questionnaire (TQ) is a validated and commonly used instrument for assessment of tinnitus severity and has been used in many clinical studies. Defining the Minimal Clinically Important Difference (MCID) for TQ changes is an important step to a better interpretation of the clinical relevance of changes observed in clinical trials. In this study we aimed to estimate the minimum change of the TQ score that could be considered clinically relevant.Methods757 patients with chronic tinnitus were pooled from the TRI database and the RESET study. An anchor-based approach using the Clinical Global Impression (CGI) scale and distributional approaches were used to estimate MCID. Receiver Operating Characteristic (ROC) curves were calculated to define optimal TQ change cutoffs discriminating between minimally changed and unchanged subjects.ResultsThe relationship between TQ change scores and CGI ratings of change was good (r = 0.52, p < 0.05). Mean change scores associated with minimally better and minimally worse CGI categories were −6.65 and +2.72 respectively. According to the ROC method MCID for improvement was −5 points and for deterioration +1 points.ConclusionDistribution and anchor-based methods yielded comparable results in identifying MCIDs. ΔTQ scores of −5 and +1 points were identified as the minimal clinically relevant change for improvement and worsening respectively. The asymmetry of the MCIDs for improvement and worsening may be related to expectation effects.

Nonparametric signal detectability evaluation using an exponential transformation of the FROC curve

To develop an efficient nonparametric method for evaluation the detectability of signals at unknown locations in images, as a mean for image quality assessment. We use the free-response methodology that allows the image observer to mark and score all locations found as suspicious in an image, summarizing these results in a free-response operating characteristic (FROC) curve. However, unlike the relative (or receiver) operating characteristic (ROC), or the localization ROC (LROC), the FROC curve has an undefined, theoretically infinite, right side limit. Therefore area under the FROC curves cannot be directly used as an overall performance index, as the area under the curve is for ROC or LROC. We circumvent this drawback by using a transformation of the abscissa that leads to a finite integration range. By applying an exponential transformation we derive a nonparametric estimator for such a metric, and we study its properties by deriving analytical expressions for the mean and standard deviation in conditions of scores independence. A comparative study with other related nonparametric estimators for ROC, LROC, and alternative FROC (AFROC) method is presented. The new nonparametric estimator has sensitivity and scalability properties that make it particularly advantageous for signal detectability evaluation in phantom experiments using model observers.

Structure‐Based Virtual Screening of Glycogen Synthase Kinase 3β Inhibitors: Analysis of Scoring Functions Applied to Large True Actives and Decoy Sets

Comparative assessment of nine different scoring functions (OpenEye and Tripos implementation) applied to structure-based virtual screening based on rigid docking of the pregenerated conformations library of glycogen synthase kinase 3β (GSK-3β) inhibitors has been carried out. The functions studied belong to the following types: Gaussian (Chemgauss3, Shapegauss), empirical (Chemscore, OEChemscore, Piecewise Linear Potential, Screenscore), force field-based (D_score and G_score), and potential of mean force (PMF_score). Overall enrichment of the large true inhibitors set against the set of true non-inhibitors, Directory of Useful Decoys (DUD), cyclin-dependent kinase 2 subset, and NCI Diversity Set was evaluated by means of ROC (receiver operating characteristic) method. According to this analysis, scoring function Chemscore leads to the best enrichment of the inhibitors whereas the best early enrichment of the actives may be obtained with the help of Chemgauss3 function as estimated by BEDROC (Boltzmann-enhanced discrimination of ROC) metrics.

Prevalence of subclinical ketosis in dairy cattle in the Southwestern Iran and detection of cutoff point for NEFA and glucose concentrations for diagnosis of subclinical ketosis

Subclinical ketosis (SCK) is simply a condition marked by increased levels of circulating ketone bodies without the presence of the clinical signs of ketosis. Subclinical ketosis can cause economic losses through decreased milk production and association with preparturient diseases. Limited information is available regarding the prevalence of SCK in dairy herds in Southwestern Iran. The objectives of this study were (i) determination of the cutoff point of nonesterified fatty acids (NEFAs) and glucose concentrations for diagnosis of SCK using receiver operating characteristic (ROC) analysis, and (ii) determination of prevalence of subclinical ketosis in apparently healthy dairy cattle in Southwestern Iran. From October to December 2009, a total of 100 clinically healthy multiparous Holstein cows (3–8 years old) were randomly selected from 16 dairy herds around Kazerun, Fars Province, Iran. The cows had two–six lactations, with body weight ranging from 500 to 650 kg. Blood samples for each cow were taken at 2, 4 and 6 weeks post parturition and 3–4 h after the morning feeding. The optimal cutoff point was set, by the ROC method, to >0.26 mmol/L for NEFA, and <2.26 mmol/L for glucose with corresponding 82.54% sensitivity and 91.89% specificity for NEFA and 44.44% sensitivity and 78.38% specificity for glucose. Cows with BHB concentrations higher than 1200 μmol/L were classified as having SCK. In 2, 4 and 6 weeks post parturition 63%, 68% and 59% of the tested cows were subclinically ketotic. Overall, 97% of tested cows (97/100) were considered subclinically ketotic in at least one sample period. Thirty percent of tested cows (30/100) suffered from subclinical ketosis in all of the 2, 4 and 6 weeks postpartum. The results suggest that, a cut-off point of 0.26 mmol/L for NEFA concentrations can be used during early lactation for diagnosis of subclinical ketosis and making management decisions for prevention and treatment. Glucose cannot be a good criterion for diagnosis of SCK and it does not appear to be useful for monitoring subclinical ketosis.

The contrast medium volume to estimated glomerular filtration rate ratio as a predictor of contrast-induced nephropathy after primary percutaneous coronary intervention

Contrast-induced nephropathy (CIN) is a serious complication in percutaneous coronary intervention (PCI) patients, which may be related to the contrast dose used during cardiac catheterization. We prospectively investigated 277 consecutive consenting patients with acute ST-segment elevation myocardial infarction (STEMI) who were given primary PCI, and we calculated their ratio of volume of contrast media to estimated glomerular filtration rate (V/eGFR). Receiver-operator characteristic methods were used to identify the optimal sensitivity for the observed range of V/eGFR for CIN (i.e., within 48-72 h). The predictive value of V/eGFR for the risk of CIN was assessed using multivariable logistic regression. Twenty-five (9%) patients developed CIN. The baseline mean and median V/eGFR values were significantly greater among patients with CIN (mean 3.22 ± 1.53, median 2.97, and interquartile range 1.90-4.17) than among those without CIN (mean 1.80 ± 1.00, median 1.52, and interquartile range 1.12-2.21, P < 0.001). The receiver-operator characteristic curve analysis indicated that a V/eGFR ratio of 2.39 was a fair discriminator for CIN (C statistic 0.81). After adjusting for other known predictors of CIN, a V/eGFR ratio ≥ 2.39 remained significantly associated with CIN (odds ratio 4.24, 95% confidence interval 1.23-14.66, P < 0.05). A V/eGFR ratio ≥ 2.39 was a significant and independent predictor of CIN after primary PCI in patients with STEMI.

Application of ROC curve analysis to FAMACHA © evaluation of haemonchosis on two sheep farms in South Africa

Test sensitivity and specificity for the FAMACHA © clinical test for anaemia due to haemonchosis have previously been shown to be adequate in differentiating between heavily/less heavily infected sheep, but these properties give no objective guidance for setting the optimum threshold at which anthelmintic treatment should occur. The aim of this work was to use Receiver Operating Characteristic (ROC) curves to evaluate the diagnostic accuracy of FAMACHA © testing by estimating the area under the ROC curve, and to use two-graph ROC curves to decrease subjectivity in selecting treatment thresholds on two farms with contrasting management. Test diagnostic accuracy, and thus discriminating power as determined by the area under the ROC curves, ranged from “moderate to good” on the first farm, and from “moderate to high” on the second farm for haematocrit (the Gold Standard for the test) cut-offs of ≤22% and ≤19% on both farms respectively. Accuracy of classification between haematocrit cut-offs was not significantly different within farms, but did differ significantly between farms, with test accuracy being highest on the second farm at both haematocrit cut-offs ( p < 0.05). The results also showed the suitability of the two-graph ROC curve approach for discriminating not only between different levels of accuracy of evaluators, but also to give an indication of the so-called ROC cut point (i.e. the desired threshold level) at which animals should be treated for a given level of risk of loss. The approach appears to have the potential not only to validate the diagnostic accuracy of the test across the complete testing range (i.e. all FAMACHA © categories from 1 to 5), but also to compensate for such inaccuracy by allowing objective adjustment of the threshold treatment level according to the output of the two-graph ROC method.

Power Estimation for Multireader ROC Methods: An Updated and Unified Approach

We describe a step-by-step procedure for estimating power and sample size for planned multireader receiver operating characteristic (ROC) studies that will be analyzed using either the Dorfman-Berbaum-Metz (DBM) or Obuchowski-Rockette (OR) method. This procedure updates previous approaches by incorporating recent methodological developments and unifies the approaches by allowing inputs to be conjectured parameter values or outputs from either a DBM or OR pilot-study analysis. Power computations are described in a step-by-step procedure and the theoretical basis for the procedure is described. Updates include using the currently recommended denominator degrees of freedom, accounting for different pilot and planned study normal-to-abnormal case ratios, and a new method for computing the OR test-by-reader variance component. Using a real dataset we illustrate how to compute the power for two planned studies, one having the same normal-to-abnormal case ratio as the pilot study and the other having a different ratio. In a simulation study, we show that the proposed procedure gives mean power estimates close to the true power. Application of the updated procedure is straightforward. It is important that pilot data be comparable to the planned study with respect to the modalities, reader expertise, and case selection. Variability of the power estimates warrants further investigation.

Time‐dependent ROC analysis for a three‐class prognostic with application to kidney transplantation

The medical decision-making community has an extensive literature on the use of receiver operating characteristic (ROC) graphs for diagnostic testing. Heagertybiet al. have recently developed this ROC curve theory within the context of survival data (Biometrics 2000; 56:337-344). The time-dependent ROC method allows evaluating the accuracy of a marker to predict a time-dependent failure, whereas the classic methodology focuses on diagnosis. One limitation to this approach, however, is to analyse a single failure. In many medical situations, a marker can be useful to predict different competitive failures. For example in kidney transplantation, the terminal evolution can be a return to dialysis or the death of the patient. With this application in mind, our paper proposes an extension of the time-dependent ROC method for analysing the accuracy of a marker to predict two competitive events.

Development of a predictive model for long-term survival after lung transplantation and implications for the lung allocation score

Improving long-term survival after lung transplantation can be facilitated by identifying patient characteristics that are predictors of positive long-term outcomes. Validated survival modeling is important for guiding clinical decision-making, case-mix adjustment in comparative effectiveness research and refinement of the lung allocation system (LAS). We used the registry of the International Society for Heart and Lung Transplantation (ISHLT) to develop and validate a predictive model of 5-year survival after lung transplantation. A total of 18,072 eligible cases were randomly split into development and validation datasets. Pre-transplant recipient variables considered included age, gender, diagnosis, body mass index, serum creatinine, hemodynamic variables, pulmonary function variables, viral status and comorbidities. Predictors were considered in a stepwise approach with the Akaike Information Criteria (AIC). Time-dependent receiver operator characteristic (ROC) curves assessed predictive ability. A 1-year conditional model and three models for disease subgroups were considered. ROC methods were used to characterize the predictive potential of the LAS post-transplant model at 1 and 5 years. The baseline model included age, diagnosis, creatinine, bilirubin, oxygen requirement, cardiac output, Epstein-Barr virus status, transfusion history and diabetes history. Prediction of long-term survival was poor (area under the curve [AUC] = 0.582). Neither the 1-year conditional model (AUC = 0.573) nor models designed for separate diseases (AUC = 0.553 to 0.591) improved survival prediction. The predictive ability of the LAS post-transplant parameters was similar to that of our model (1-year AUC = 0.580 and 5-year AUC = 0.566). Models developed from pre-transplant characteristics poorly predict long-term survival. Models for separate diseases and 1-year conditional models did not improve prediction. Better databases and approaches to predict survival are needed to improve lung allocation.

Consistency of methods for analysing location-specific data

Although the receiver operating characteristic (ROC) method is the acknowledged gold-standard for imaging system assessment, it ignores localisation information and differentiation between multiple abnormalities per case. As the free-response ROC (FROC) method uses localisation information and more closely resembles the clinical reporting process, it is being increasingly used. A number of methods have been proposed to analyse the data that result from an FROC study: jackknife alternative FROC (JAFROC) and a variant termed JAFROC1, initial detection and candidate analysis (IDCA) and ROC analysis via the reduction of the multiple ratings on a case to a single rating. The focus of this paper was to compare JAFROC1, IDCA and the ROC analysis methods using a clinical FROC human data set. All methods agreed on the ordering of the modalities and all yielded statistically significant differences of the figures-of-merit, i.e. p < 0.05. Both IDCA and JAFROC1 yielded much smaller p-values than ROC. The results are consistent with a recent simulation-based validation study comparing these and other methods. In conclusion, IDCA or JAFROC1 analysis of FROC human data may be superior at detecting modality differences than ROC analysis.

Clinical relevance of the ROC and free-response paradigms for comparing imaging system efficacies

Observer performance studies are widely used to assess medical imaging systems. Unlike technical/engineering measurements observer performance include the entire imaging chain and the radiologist. However, the widely used receiver operating characteristic (ROC) method ignores lesion localisation information. The free-response ROC (FROC) method uses the location information to appropriately reward or penalise correct or incorrect localisations, respectively. This paper describes a method for improving the clinical relevance of FROC studies. The method consists of assigning appropriate risk values to the different lesions that may be present on a single image. A high-risk lesion is one that is critical to detect and act upon, and is assigned a higher risk value than a low-risk lesion, one that is relatively innocuous. Instead of simply counting the number of lesions that are detected, as is done in conventional FROC analysis, a risk-weighted count is used. This has the advantage of rewarding detections of high-risk lesions commensurately more than detections of lower risk lesions. Simulations were used to demonstrate that the new method, termed case-based analysis, results in a higher figure of merit for an expert who detects more high-risk lesions than a naive observer who detects more low-risk lesions, even though both detect the same total number of lesions. Conventional free-response analysis is unable to distinguish between the two types of observers. This paper also comments on the issue of clinical relevance of ROC analysis vs. FROC for tasks that involve lesion localisation.

Study protocol of effectiveness of a biopsychosocial multidisciplinary intervention in the evolution of non-speficic sub-acute low back pain in the working population: cluster randomised trial

BackgroundNon-specific low back pain is a common cause for consultation with the general practitioner, generating increased health and social costs. This study will analyse the effectiveness of a multidisciplinary intervention to reduce disability, severity of pain, anxiety and depression, to improve quality of life and to reduce the incidence of chronic low back pain in the working population with non-specific low back pain, compared to usual clinical care.Methods/DesignA Cluster randomised clinical trial will be conducted in 38 Primary Health Care Centres located in Barcelona, Spain and its surrounding areas. The centres are randomly allocated to the multidisciplinary intervention or to usual clinical care. Patients between 18 and 65 years old (n = 932; 466 per arm) and with a diagnostic of a non-specific sub-acute low back pain are included. Patients in the intervention group are receiving the recommendations of clinical practice guidelines, in addition to a biopsychosocial multidisciplinary intervention consisting of group educational sessions lasting a total of 10 hours. The main outcome is change in the score in the Roland Morris disability questionnaire at three months after onset of pain. Other outcomes are severity of pain, quality of life, duration of current non-specific low back pain episode, work sick leave and duration, Fear Avoidance Beliefs and Goldberg Questionnaires. Outcomes will be assessed at baseline, 3, 6 and 12 months. Analysis will be by intention to treat. The intervention effect will be assessed through the standard error of measurement and the effect-size. Responsiveness of each scale will be evaluated by standardised response mean and receiver-operating characteristic method. Recovery according to the patient will be used as an external criterion. A multilevel regression will be performed on repeated measures. The time until the current episode of low back pain takes to subside will be analysed by Cox regression.DiscussionWe hope to provide evidence of the effectiveness of the proposed biopsychosocial multidisciplinary intervention in avoiding the chronification of low back pain, and to reduce the duration of non-specific low back pain episodes. If the intervention is effective, it could be applied to Primary Health Care Centres.Trial RegistrationISRCTN21392091

Potential of 1H MR spectroscopic imaging to segregate patients who are likely to show malignancy of the peripheral zone of the prostate on biopsy

To evaluate the potential of MR spectroscopic imaging (MRSI) to segregate patients who, upon prostate biopsy, are more likely to show a malignancy in the peripheral zone (PZ) of the prostate gland. Before biopsy, MRSI was carried in 123 men with elevated prostate specific antigen level or an abnormal digital rectal examination. After the MRSI investigation, all patients underwent systematic transrectal ultrasound guided biopsy and were categorized using standard random number tables into the following two groups: (i) Group I, a 62 member training set; and (ii) Group II, a 61-member test set. The cutoff value for the [citrate/(choline+creatine)] ratio for patients in the training set was obtained using the receiver operating characteristic (ROC) curve method. This value was then applied to the test set of patients as well. The ROC method gave a cutoff value of 1.2 for the [Citrate/(Choline+Creatine)] ratio. When applied as a malignancy-predictor to the test group of patients (Group II), the ROC method generated the following results: sensitivity, 77%; specificity, 83%; negative predictive value, 93%; positive predictive value, 55% and accuracy, 82%. The results indicate that patients who are deemed as malignancy-positive in the PZ by MRSI using the ROC cutoff may be subjected to prostate biopsy to confirm the diagnosis of cancer.

Improving Performance of Computer-aided Detection Scheme by Combining Results From Two Machine Learning Classifiers

Global data-based and local instance-based machine-learning methods and classifiers have been widely used to optimize computer-aided detection and diagnosis (CAD) schemes to classify between true-positive and false-positive detections. In this study, the correlation between these two types of classifiers was investigated using a new independent testing data set, and the potential improvement of a CAD scheme's performance by combining the results of the two classifiers in detecting breast masses was assessed. The CAD scheme first used image filtering and a multilayer topographic region growth algorithm to detect and segment suspicious mass regions. The scheme then used an image feature-based classifier to classify these regions into true-positive and false-positive regions. Two classifiers were used in this study. One was a global data-based machine-learning classifier, an artificial neural network (ANN), and the other was a local instance-based machine-learning classifier, a k-nearest neighbor (KNN) algorithm. An independent image database including 400 mammographic examinations was used in this study. Of these, 200 were cancer cases and 200 were negative cases. The preoptimized CAD scheme was applied twice to the database using the two different classifiers. The correlation between the two sets of classification results was analyzed. Three sets of CAD performance results using the ANN, KNN, and average detection scores from both classifiers were assessed and compared using the free-response receiver-operating characteristic method. The results showed that the ANN achieved higher performance than the KNN algorithm, with a normalized area under the performance curve (AUC) of 0.891 versus 0.845. The correlation coefficients between the detection scores generated by the two classifiers were 0.436 and 0.161 for the true-positive and false-positive detections, respectively. The average detection scores of the two classifiers improved CAD performance and reliability by increasing the AUC to 0.912 and reducing the standard error of the estimated AUC by 14.4%. The detection sensitivity was also increased from 75.8% (ANN) and 65.9% (KNN) to 80.3% at a false-positive detection rate of 0.3 per image. This study demonstrates that two global data-based and local data-based machine-learning classifiers (ANN and KNN) generated low correlated detection results and that combining the detection scores of these two classifiers significantly improved overall CAD performance (P < .01) and reduced standard error in CAD performance assessment.

Transformation‐invariant and nonparametric monotone smooth estimation of ROC curves

When a new diagnostic test is developed, it is of interest to evaluate its accuracy in distinguishing diseased subjects from non-diseased subjects. The accuracy of the test is often evaluated by receiver operating characteristic (ROC) curves. Smooth ROC estimates are often preferable for continuous test results when the underlying ROC curves are in fact continuous. Nonparametric and parametric methods have been proposed by various authors to obtain smooth ROC curve estimates. However, there are certain drawbacks with the existing methods. Parametric methods need specific model assumptions. Nonparametric methods do not always satisfy the inherent properties of the ROC curves, such as monotonicity and transformation invariance. In this paper we propose a monotone spline approach to obtain smooth monotone ROC curves. Our method ensures important inherent properties of the underlying ROC curves, which include monotonicity, transformation invariance, and boundary constraints. We compare the finite sample performance of the newly proposed ROC method with other ROC smoothing methods in large-scale simulation studies. We illustrate our method through a real life example.

The use of receiver operating characteristic (ROC) analysis to assess the diagnostic value of serum amyloid A, haptoglobin and fibrinogen in traumatic reticuloperitonitis in cattle

Acute phase proteins (APPs) such as serum amyloid A (SAA), haptoglobin (Hp), and plasma fibrinogen (Fb) are diagnostic and prognostic biomarkers. This study determined the pattern of SAA in traumatic reticuloperitonitis (TRP) in cows, and compared the findings with Hp, and plasma Fb concentrations in the differential diagnosis of TRP using receiver operating characteristic (ROC) analysis. Blood samples were collected from 89 cows: 53 cases of TRP, 16 animals with other internal disorders and 20 healthy controls. SAA and Hp had a similar pattern and magnitude of changes in TRP. Using the ROC method, the optimal cut-off for the diagnosis of TRP was 68 microg/mL for SAA and 0.74 g/L for Hp, with 100% sensitivity and 86.1% specificity; either of these parameters may therefore be used as a sensitive and relatively specific tool for the differential diagnosis of TRP in cattle. The diagnostic accuracy of plasma Fb was significantly lower than either SAA or Hp (P<0.01). The combined use of these APPs (whilst aware of their conditional interdependence) may not provide any additional diagnostic information than can be obtained using either SAA or Hp alone.

Bivariate Random Effects Meta-Analysis of ROC Curves

Meta-analysis of receiver operating characteristic (ROC)-curve data is often done with fixed-effects models, which suffer many shortcomings. Some random-effects models have been proposed to execute a meta-analysis of ROC-curve data, but these models are not often used in practice. Straightforward modeling techniques for multivariate random-effects meta-analysis of ROC-curve data are needed. The 1st aim of this article is to present a practical method that addresses the drawbacks of the fixed-effects summary ROC (SROC) method of Littenberg and Moses. Sensitivities and specificities are analyzed simultaneously using a bivariate random-effects model. The 2nd aim is to show that other SROC curves can also be derived from the bivariate model through different characterizations of the estimated bivariate normal distribution. Thereby the authors show that the bivariate random-effects approach not only extends the SROC approach but also provides a unifying framework for other approaches. The authors bring the statistical meta-analysis of ROC-curve data back into a framework of relatively standard multivariate meta-analysis with random effects. The analyses were carried out using the software package SAS (Proc NLMIXED).

COMPARISON OF ROC AND LIKELIHOOD DECISION METHODS IN AUTOMATIC FINGERPRINT VERIFICATION

The biometric verification task is to determine whether or not an input and a template belong to the same individual. In the context of automatic fingerprint verification the task consists of three steps: feature extraction, where features (typically minutiae) are extracted from each fingerprint, scoring, where the degree of match between the two sets of features is determined, and decision, where the score is used to accept or reject the hypothesis that the input and template belong to the same individual. The paper focuses on the final decision step, which is a binary classification problem involving a single score variable. The commonly used decision method is to learn a score threshold from a labeled set of inputs and templates, by first determining the receiver operating characteristic (ROC) of the task. The ROC method works well when there is a well-registered fingerprint image. The paper shows that when there is uncertainty due to fingerprint quality, e.g. the input is a latent print or a partial print, the decision method can be improved by using the likelihood ratio of match/non match. The likelihood ratio is obtained by modeling the distributions of same finger and different finger scores using parametric distributions. The parametric forms considered are Gaussian and Gamma distributions whose parameters are learnt from labeled training samples. The performances of the likelihood and ROC methods are compared for varying numbers of minutiae points available for verification. Using either Gaussian or Gamma parametric distributions, the likelihood method has a lower error rate than the ROC method when few minutiae points are available. Likelihood and ROC methods converge to the same accuracy as more minutiae points are available.