Binormal Receiver Operating Characteristic Curve Research Articles

Developing Robust Flood Susceptibility Model with Small Numbers of Parameters in Highly Fertile Regions of Northwest Bangladesh for Sustainable Flood and Agriculture Management

The present study intends to improve the robustness of a flood susceptibility (FS) model with a small number of parameters in data-scarce areas, such as northwest Bangladesh, by employing machine learning-based sensitivity analysis and an analytical hierarchy process (AHP). In this study, the nine most relevant flood elements (such as distance from the river, rainfall, and drainage density) were chosen as flood conditioning variables for modeling. The FS model was produced using AHP technique. We used an empirical and binormal receiver operating characteristic (ROC) curves for validating the models. We performed Sensitivity analyses using a random forest (RF)-based mean Gini decline (MGD), mean decrease accuracy (MDA), and information gain ratio to find out the sensitive flood conditioning variables. After performing sensitivity analysis, the least sensitivity variables were eliminated. We re-ran the model with the rest of the parameters to enhance the model’s performance. Based on previous studies and the AHP weighting approach, the general soil type, rainfall, distance from river/canal (Dr), and land use/land cover (LULC) had higher factor weights of 0.22, 0.21, 0.19, and 0.15, respectively. The FS model without sensitivity and with sensitivity performed well in the present study. According to the RF-based sensitivity and information gain ratio, the most sensitive factors were rainfall, soil type, slope, and elevation, while curvature and drainage density were less sensitive parameters, which were excluded in re-running the FS model with just vital parameters. Using empirical and binormal ROC curves, the new FS model yields higher AUCs of 0.835 and 0.822, respectively. It is discovered that the predicted model’s robustness may be maintained or increased by removing less relevant factors. This study will aid decision-makers in developing flood management plans for the examined region.

Novel hybrid models to enhance the efficiency of groundwater potentiality model

The present study aimed to create novel hybrid models to produce groundwater potentiality models (GWP) in the Teesta River basin of Bangladesh. Six ensemble machine learning (EML) algorithms, such as random forest (RF), random subspace, dagging, bagging, naïve Bayes tree (NBT), and stacking, coupled with fuzzy logic (FL) models and a ROC-based weighting approach have been used for creating hybrid models integrated GWP. The GWP was then verified using both parametric and nonparametric receiver operating characteristic curves (ROC), such as the empirical ROC (eROC) and the binormal ROC curve (bROC). We conducted an RF-based sensitivity analysis to compute the relevancy of the conditioning variables for GWP modeling. The very high and high groundwater potential regions were predicted as 831–1200 km2 and 521–680 km2 areas based on six EML models. Based on the area under the curve of the ROC, the NBT (eROC: 0.892; bROC: 0.928) model outperforms rest of the models. Six GPMs were considered variables for the next step and turned into crisp fuzzy layers using the fuzzy membership function, and the ROC-based weighting approach. Subsequently four fuzzy logic operators were used to assimilate the crisp fuzzy layers, including AND, OR, GAMMA0.8, and GAMMA 0.9, as well as GAMMA0.9. Thus, we created four hybrid models using FL model. The results of the eROC and bROC curve showed that GAMMA 0.9 operator outperformed other fuzzy operators-based GPMs in terms of accuracy. According to the validation outcomes, four hybrid models outperformed six EML models in terms of performance. The present study will aid in enhancing the efficiency of GPMs in preparing viable planning for groundwater management.

Minimum distance estimation of the Lehmann receiver operating characteristic curve

A receiver operating characteristic (ROC) curve based on the Lehmann family of distributions has a very simple form and is an important alternative to the popular binormal ROC curve. In this paper, we propose to use the minimum distance method to estimate a one-dimensional parameter of this curve. We prove the consistency and asymptotic normality of estimators obtained by applying various measures of discrepancy. In a simulation study, we compare the accuracy of the estimators proposed with the accuracy of a known estimator based on a partial likelihood method and an estimator based on the estimation of the area under the curve. We also provide an illustrative example of an application to real data.

Biomarker assessment in ROC curve analysis using the length of the curve as an index of diagnostic accuracy: the binormal model framework

In receiver operating characteristic (ROC) curve analysis, the area under the curve (AUC) is undoubtedly the most widely used index of diagnostic accuracy for the assessment of the utility of a biomarker or for the comparison of competing biomarkers. Along with the AUC, the maximum of the Youden index, J, is often used both as an index of diagnostic accuracy and as a tool useful for the estimation of an optimal cutoff point that can be used for diagnostic purposes based on the biomarker under consideration. In this work, we study the utility of the length of the binormal model-based ROC curve (LoC) as an index of diagnostic accuracy for biomarker evaluation. Estimation procedures for LoC, described in this article, are based either on normality assumptions or on the same assumptions after a Box–Cox transformation to normality. Two simulation studies are considered. In the first, the estimation procedures for LoC are compared in terms of bias and root mean squared error, while in the second one, the performance of LoC is compared with approaches based on AUC and J, both for the case of the assessment of a single biomarker and for the comparison of two biomarkers, in a parametric framework. We provide an interpretation for the proposed index and illustrate with an application on biomarkers from a colorectal cancer study.

On the Binormal Predictive Receiver Operating Characteristic Curve for the Joint Assessment of Positive and Negative Predictive Values.

The predictive receiver operating characteristic (PROC) curve is a diagrammatic format with application in the statistical evaluation of probabilistic disease forecasts. The PROC curve differs from the more well-known receiver operating characteristic (ROC) curve in that it provides a basis for evaluation using metrics defined conditionally on the outcome of the forecast rather than metrics defined conditionally on the actual disease status. Starting from the binormal ROC curve formulation, an overview of some previously published binormal PROC curves is presented in order to place the PROC curve in the context of other methods used in statistical evaluation of probabilistic disease forecasts based on the analysis of predictive values; in particular, the index of separation (PSEP) and the leaf plot. An information theoretic perspective on evaluation is also outlined. Five straightforward recommendations are made with a view to aiding understanding and interpretation of the sometimes-complex patterns generated by PROC curve analysis. The PROC curve and related analyses augment the perspective provided by traditional ROC curve analysis. Here, the binormal ROC model provides the exemplar for investigation of the PROC curve, but potential application extends to analysis based on other distributional models as well as to empirical analysis.

Minimum distance estimation of the binormal ROC curve

The receiver operating characteristic (ROC) curve describes the performance of a diagnostic test, which classifies individuals into one of two categories. Many parametric, semiparametric and nonparametric estimation methods have been proposed for estimating the ROC curve and its functionals. In this paper the minimum distance estimation of the binormal ROC curve is considered. A modification of the estimator considered in the paper of Davidov and Nov (J Stat Plan Inference 142(4):872–877, 2012) and some new estimators are proposed. We compare the accuracy of the new estimators with known minimum distance estimators of the binormal ROC curve and we conclude that our estimators generally perform better than their competitors.

Estimating the Area Under ROC Curve When the Fitted Binormal Curves Demonstrate Improper Shape

The "binormal" model is the most frequently used tool for parametric receiver operating characteristic (ROC) analysis. The binormal ROC curves can have "improper" (non-concave) shapes that are unrealistic in many practical applications, and several tools (eg, PROPROC) have been developed to address this problem. However, due to the general robustness of binormal ROCs, the improperness of the fitted curves might carry little consequence for inferences about global summary indices, such as the area under the ROC curve (AUC). In this work, we investigate the effect of severe improperness of fitted binormal ROC curves on the reliability of AUC estimates when the data arise from an actually proper curve. We designed theoretically proper ROC scenarios that induce severely improper shape of fitted binormal curves in the presence of well-distributed empirical ROC points. The binormal curves were fitted using maximum likelihood approach. Using simulations, we estimated the frequency of severely improper fitted curves, bias of the estimated AUC, and coverage of 95% confidence intervals (CIs). In Appendix S1, we provide additional information on percentiles of the distribution of AUC estimates and bias when estimating partial AUCs. We also compared the results to a reference standard provided by empirical estimates obtained from continuous data. We observed up to 96% of severely improper curves depending on the scenario in question. The bias in the binormal AUC estimates was very small and the coverage of the CIs was close to nominal, whereas the estimates of partial AUC were biased upward in the high specificity range and downward in the low specificity range. Compared to a non-parametric approach, the binormal model led to slightly more variable AUC estimates, but at the same time to CIs with more appropriate coverage. The improper shape of the fitted binormal curve, by itself, ie, in the presence of a sufficient number of well-distributed points, does not imply unreliable AUC-based inferences.

ROC Curve for Extremely Subtle Lung Nodules on Chest Radiographs Confirmed by CT Scan

The aim of the present study was to test the hypothesis that when a radiologist does not perceive an abnormality in images that contain either extremely subtle abnormalities or no abnormalities, the radiologist cannot distinguish these two types of images and the receiver operating characteristic (ROC) curve reflects that performance. This retrospective study was conducted with approval of our institutional review board. Four general radiologists participated in an observer performance study of 100 chest images, each of which had a 5 × 5 cm region of interest (ROI) drawn (50 containing a lung nodule, and 50 did not, based on computed tomography [CT] confirmation). About half of the lung nodules were extremely subtle. The readers reported their confidence that a nodule was present within the ROI, from which empirical and maximum-likelihood "proper" binormal and conventional binormal ROC curves were estimated. The readers also reported whether they saw an abnormality that could be a nodule within the ROI. Empirical ROC curves deviated from typical ROC-curve shapes, and a portion of the curve leading to the northeast corner of the ROC space had relatively steep and constant slopes. The readers reported not seeing anything suggestive of a lung nodule in this portion of the ROC curve, which also corresponded to cases that either contained extremely subtle nodules or normal cases. The average area under the ROC curves (mean ± standard deviation) was 0.66 ± 0.02 for proper binormal, 0.62 ± 0.02 for conventional binormal, and 0.60 ± 0.03 for trapezoidal ROC curves. When a radiologist does not perceive an abnormality in images that contain either extremely subtle abnormalities or no abnormalities, the ROC curve (or a portion thereof) is characterized by a straight line, which is not consistent with conventional ROC theories.

Diagnostic accuracy of 3D color volume-rendered CT images for peroneal tendon dislocation in patients with acute calcaneal fractures

Use of three-dimensional (3D) color volume-rendered (VR) images has been reported to be more time-efficient compared to that of cross-sectional computed tomography (CT) images for the diagnosis of peroneal tendon dislocation. However, the diagnostic performance of this technique has not been studied. To test diagnostic accuracy of 3D color VR CT images of ankle for peroneal tendon dislocation in patients with acute calcaneal fractures. The study consisted of 121 ankle CT studies from 105 consecutive patients (85 men, 20 women; mean age, 42 years; age range, 16-75 years) with acute calcaneal fractures. Peroneal tendon dislocation was diagnosed on multiplanar CT images by consensus of two experienced musculoskeletal radiologists, which served as the reference standard. Three other musculoskeletal radiologists independently reviewed 3D images alone on a workstation. The readers determined whether or not there was peroneal tendon dislocation using three degrees of certainty (definite, probable, and possible). Diagnostic performance of 3D images for peroneal tendon dislocation was evaluated by calculating the sensitivities, specificities, and area under the receiver-operating characteristic (ROC) curves. Forty-eight (40%) out of 121 studies showed peroneal tendon dislocation based on the expert readings using multiplanar reformatted images. Sensitivities/specificities of 3D images measured 0.92/0.81, 0.88/0.90, and 0.81/0.92 for three readers, respectively. The area under the proper binormal ROC curve based on all three readers (0.93, 0.94, and 0.92) measured 0.93 with a 95% confidence interval of 0.89-0.98. Diagnostic accuracy of 3D images is comparable to, but not as good as that of MPR images for the diagnosis of peroneal tendon dislocation in patients with acute calcaneal fractures.

On the use of partial area under the ROC curve for comparison of two diagnostic tests.

Evaluation of diagnostic performance is typically based on the receiver operating characteristic (ROC) curve and the area under the curve (AUC) as its summary index. The partial area under the curve (pAUC) is an alternative index focusing on the range of practical/clinical relevance. One of the problems preventing more frequent use of the pAUC is the perceived loss of efficiency in cases of noncrossing ROC curves. In this paper, we investigated statistical properties of comparisons of two correlated pAUCs. We demonstrated that outside of the classic model there are practically reasonable ROC types for which comparisons of noncrossing concave curves would be more powerful when based on a part of the curve rather than the entire curve. We argue that this phenomenon stems in part from the exclusion of noninformative parts of the ROC curves that resemble straight-lines. We conducted extensive simulation studies in families of binormal, straight-line, and bigamma ROC curves. We demonstrated that comparison of pAUCs is statistically more powerful than comparison of full AUCs when ROC curves are close to a "straight line". For less flat binormal ROC curves an increase in the integration range often leads to a disproportional increase in pAUCs' difference, thereby contributing to an increase in statistical power. Thus, efficiency of differences in pAUCs of noncrossing ROC curves depends on the shape of the curves, and for families of ROC curves that are nearly straight-line shaped, such as bigamma ROC curves, there are multiple practical scenarios in which comparisons of pAUCs are preferable.

Comparison of Paired ROC Curves through a Two-Stage Test

The area under the receiver operating characteristic (ROC) curve (AUC) is a popularly used index when comparing two ROC curves. Statistical tests based on it for analyzing the difference have been well developed. However, this index is less informative when two ROC curves cross and have similar AUCs. In order to detect differences between ROC curves in such situations, a two-stage nonparametric test that uses a shifted area under the ROC curve (sAUC), along with AUCs, is proposed for paired designs. The new procedure is shown, numerically, to be effective in terms of power under a wide range of scenarios; additionally, it outperforms two conventional ROC-type tests, especially when two ROC curves cross each other and have similar AUCs. Larger sAUC implies larger partial AUC at the range of low false-positive rates in this case. Because high specificity is important in many classification tasks, such as medical diagnosis, this is an appealing characteristic. The test also implicitly analyzes the equality of two commonly used binormal ROC curves at every operating point. We also apply the proposed method to synthesized data and two real examples to illustrate its usefulness in practice.

Serum microRNA expression levels can predict lymph node metastasis in patients with early-stage cervical squamous cell carcinoma

Circulating microRNA expression levels can serve as diagnostic/prognostic biomarkers in several types of malignant tumors; however, to our knowledge, there have been reports describing their value in cervical squamous cell carcinoma (SCC). In this study, we used hybridization arrays to compare the microRNA expression profiles in cervical squamous cell carcinomas (SCC) samples among patients with lymph node metastasis (LNM) or without LNM; 89 microRNAs were found to fit our inclusion criteria. Using quantitative PCR (qPCR), we examined the expression levels of these microRNAs in cervical cancer tissue, as well as in serum from patients and healthy women. We compared the expression levels between patients with LNM (n=40) and those without LNM (n=40) and healthy controls (n=20). Using regression analysis, we generated a comprehensive set of marker microRNAs and drew the fitted binormal receiver operating characteristic (ROC) curves to access the predictive value. We identified 6 serum microRNAs that can predict LNM in cervical SCC patients; these microRNAs were miR-1246, miR-20a, miR-2392, miR-3147, miR-3162-5p and miR-4484. The area under the curve (AUC) of the comprehensive set of serum microRNAs predicting LNM was 0.932 (sensitivity, 0.856; specificity, 0.850). The predictive value of the serum microRNAs was inferior to that in tissue (AUC 0.992; sensitivity, 0.967; specificity, 0.950; P=0.018). We compared the LNM predictive value of serum microRNAs and SCC antigen (SCC-Ag) by drawing fitted binormal ROC curves However, serum microRNA analysis is by far superior to serum SCC-Ag analysis (AUC 0.713; sensitivity, 0.612; specificity, 0.700; P<0.0001). Serum microRNAs are a good predictor of LNM with clinical value in early-stage cervical SCC.

Symmetry Properties of Bi-Normal and Bi-Gamma Receiver Operating Characteristic Curves are Described by Kullback-Leibler Divergences

Receiver operating characteristic (ROC) curves have application in analysis of the performance of diagnostic indicators used in the assessment of disease risk in clinical and veterinary medicine and in crop protection. For a binary indicator, an ROC curve summarizes the two distributions of risk scores obtained by retrospectively categorizing subjects as cases or controls using a gold standard. An ROC curve may be symmetric about the negative diagonal of the graphical plot, or skewed towards the left-hand axis or the upper axis of the plot. ROC curves with different symmetry properties may have the same area under the curve. Here, we characterize the symmetry properties of bi-Normal and bi-gamma ROC curves in terms of the Kullback-Leibler divergences (KLDs) between the case and control distributions of risk scores. The KLDs describe the known symmetry properties of bi-Normal ROC curves, and newly characterize the symmetry properties of constant-shape and constant-scale bi-gamma ROC curves. It is also of interest to note an application of KLDs where their asymmetry—often an inconvenience—has a useful interpretation.

An Analytic Expression for the Binormal Partial Area under the ROC Curve

The partial area under the receiver operating characteristic (ROC) curve (pAUC) is a useful summary measure for diagnostic studies. Unlike most summary measures that are functions of the ROC curve, researchers have not been aware of an analytic expression available for computing the pAUC for an ROC curve based on a latent binormal model. Instead, the pAUC has been computed using numerical integration or a rational polynomial approximation. Our purpose is to provide analytic expressions for the two forms of pAUC. We discuss the two fundamentally different types of pAUC. We present analytic expressions for both types, provide corresponding proofs, and illustrate their application with an example comparing the performances of spin echo and cine magnetic resonance imaging for detecting thoracic aortic dissection. We provide an example of using the pAUC as the outcome in a multireader multicase analysis. We find that using the pAUC results in a more significant finding. We have provided analytic expressions for both types of pAUC, making it easier to compute the pAUCs corresponding to binormal ROC curves.

Teaching Receiver Operating Characteristic Analysis: An Interactive Laboratory Exercise

Despite its fundamental importance in the evaluation of diagnostic tests, receiver operating characteristic (ROC) analysis is not easily understood. The purpose of this project was to create a learning experience that resulted in an intuitive understanding of the basic principles of ROC analysis. An interactive laboratory exercise was developed for a class about radiology testing taught within a clinical epidemiology course between 2000 and 2009. The physician students in the course were clinical fellows from various medical specialties who were enrolled in a graduate degree program in clinical investigation. For the exercise, the class was divided into six groups. Each group interpreted radiographs from a set of 50 exams of the peripheral skeleton to determine the presence or absence of an acute fracture. Data from the class were pooled and given to each student. Students calculated the area under the ROC curve (AUC) corresponding to overall class performance. A binormal ROC curve was also fitted to the data from each class year. The laboratory exercise was conducted for 8 years with approximately 20-30 students per year. The mean AUC over the eight laboratory classes was 0.72 with a standard deviation of 0.08 (range, 0.60-0.85). With some simplifications in design, an observer study can be conducted in a laboratory classroom setting. Participatory data collection promotes the intuitive understanding of ROC analysis principles.

A Study on the Bi-Rayleigh ROC Curve Model

Receiver Operating Characteristic (ROC) curves are used to describe and compare the accuracy of diagnostic test or the ability of a continuous biomarker in discriminating between the subjects into healthy or diseased cases in medical field. The most familiar form of ROC curve is Bi-normal (Gaussian) ROC curve model, which assumes that the test scores or a monotone transformation of the test scores are from two normal populations (i.e. healthy and diseased). It may not be true all the time, it may violate the assumptions of normal distribution in some situations and also we cannot adopt the model as it is when the sample size is small. In this paper, we have proposed ROC curve model for Rayleigh distribution which can be used even when sample size is small. The properties of Bi-Rayleigh ROC model are studied and Area Under the ROC Curve (AUC) are derived. The proposed model is supported by real life example as well as simulation studies. The confidence interval for the population parameter is studied with simulation studies of varying sample sizes. It is found that Bi-Rayleigh ROC model provides better accuracy of classification than the conventional bi-normal ROC model. In the ROC model section, Bi-Rayleigh ROC model and its AUC are derived and its properties are discussed. In application section, two real life example viz. Head trauma data and Multiple Sclerosis data were used for the proposed model and then these are compared with the conventional bi- normal model. In the simulation section, simulation studies have been carried out for different values of parameters to support the Bi-Rayleigh ROC model. In the last section, the confidence intervals for the parameter σi, i=0,1 are discussed for real life example and simulated data sets. II. ROC MODEL AND AREA UNDER THE CURVE Let the scores (S) be a random variable in healthy (H) and diseased (D) populations respectively. The ROC curve is plotted over values of threshold value (t) and is given by y(t) = h (x(t)) where x(t) is the False positive rate which is given by x(t) = P(S>t|H) and y(t) is True positive rate which is given by y(t) = P (S>t|D) The general functional form of ROC can be represented as

The Bi-Gamma ROC Curve in a Straightforward Manner

In biomedical research, biomarkers (diagnostic tests) are used in distinguishing healthy and diseased populations. The effectiveness and accuracy of a biomarker generally assessed through the use of a Receiver Operating Characteristic (ROC) curve model, and its functional such as area under the curve (AUC). The parametric (smooth) ROC curves are obtained under the specific distributions assumptions. A resulting ROC curve model is the plot of sensitivity versus 1-specificity for all possible threshold values. Most popular and widely used ROC curve model is bi-normal ROC curve model under the assumptions of normality. When the biomarker results are continuous and positively skewed (non-normal). The gamma distribution is supposed to a flexible model for positively skewed measurements. In practice use of bi-gamma ROC curve model is hindered by the fact that ROC function cannot be written in closed-form. The solution of the problem is to use transformed invariance property of ROC curve model. Which assumes that the test results of both diseased and healthy are normally distributed after some monotone transformation [1]. In this paper we propose a simple approximation solution for the problem mentioned in above lines using a normal approximation due to Wilson and Hilfertys [2]. Which is useful to approximate gamma distribution results with classical normal distribution based results.

Confidence intervals and bands for the binormal ROC curve revisited

Two types of confidence intervals (CIs) and confidence bands (CBs) for the receiver operating characteristic (ROC) curve are studied: pointwise CIs and simultaneous CBs. An optimized version of the pointwise CI with the shortest width is developed. A new ellipse-envelope simultaneous CB for the ROC curve is suggested as an adaptation of the Working–Hotelling-type CB implemented in a paper by Ma and Hall (1993). Statistical simulations show that our ellipse-envelope CB covers the true ROC curve with a probability close to nominal while the coverage probability of the Ma and Hall CB is significantly smaller. Simulations also show that our CI for the area under the ROC curve is close to nominal while the coverage probability of the CI suggested by Hanley and McNail (1982) uniformly overestimates the nominal value. Two examples illustrate our simultaneous ROC bands: radiation dose estimation from time to vomiting and discrimination of breast cancer from benign abnormalities using electrical impedance measurements.

Improving an estimator of Hsieh and Turnbull for the binormal ROC curve

The estimator of Hsieh and Turnbull (1996) for the binormal receiver operating characteristic (ROC) curve is extended from grouped to ungrouped data. The new estimator is shown to be consistent and asymptotically normally distributed, and simulation results show that it outperforms Hsieh and Turnbull's original estimator.

Using the Mean-to-Sigma Ratio as a Measure of the Improperness of Binormal ROC Curves

A basic assumption for a meaningful diagnostic decision variable is that there is a monotone relationship between the decision variable and the likelihood of disease. This relationship, however, generally does not hold for the binormal model. As a result, receiver operating characteristic (ROC)-curve estimation based on the binormal model produces improper ROC curves that are not concave over the entire domain and cross the chance line. Although in practice the "improperness" is typically not noticeable, there are situations where the improperness is evident. Presently, standard statistical software does not provide diagnostics for assessing the magnitude of the improperness. We show how the mean-to-sigma ratio can be a useful, easy-to-understand and easy-to-use measure for assessing the magnitude of the improperness of a binormal ROC curve by showing how it is related to the chance-line crossing. We suggest an improperness criterion based on the mean-to-sigma ratio. Using a real-data example, we illustrate how the mean-to-sigma ratio can be used to assess the improperness of binormal ROC curves, compare the binormal method with an alternative proper method, and describe uncertainty in a fitted ROC curve with respect to improperness. By providing a quantitative and easily computable improperness measure, the mean-to-sigma ratio provides an easy way to identify improper binormal ROC curves and facilitates comparison of analysis strategies according to improperness categories in simulation and real-data studies.