Receiver Operating Characteristic Space Research Articles

High-performance imbalanced learning ensembles of decision trees for detecting mineralization anomalies from geochemical exploration data

How to effectively detect geochemical anomalies associated with mineralization is a challenging task due to the extreme class-imbalance of geochemical exploration data. To address this challenge, various machine learning techniques have been employed to detect geochemical anomalies associated with mineralization. However, almost all of these machine learning techniques have their limitations when it comes to modeling class-imbalanced geochemical exploration data. To establish efficient robust high-performance models for detecting geochemical anomalies associated mineralization, a case study was carried out in the Helong area, Jilin Province, China. Decision tree (C4.5) classifiers were used as the base or weak classifiers, four imbalanced learning ensemble models, including self-paced ensemble model, under-bagging ensemble model, synthetic minority oversampling technique (SMOTE)-boost ensemble model and random under sampling (RUS)-boost ensemble model were established and compared in the detection of polymetallic mineralization anomalies from the 1:50,000 stream sediment survey data. The output of the base classifier is a vector consisting of the probability that the sample belongs to a mineralization anomaly and the probability that the sample belongs to the background. The performance of each ensemble model was evaluated using the receiver operating characteristic (ROC) curve and the area under the ROC curve (AUC). The ROC curves of the four models are close to the upper left corner of the ROC space. The AUC values of the four models are not lower than 0.9589. The polymetallic mineralization anomalies detected by the four models account for less than 7 % of the entire study area, but contain not less than 93 % of the polymetallic deposits found in the study area. In addition, the polymetallic mineralization anomalies detected by the four models spatially coincide with the regional controlling factors of polymetallic mineralization in the study area. To summarize, the four models have very high performance in detecting polymetallic mineralization anomalies, and the polymetallic mineralization anomaly detection results are consistent with the regional geological and metallogenic characteristics in the study area. Therefore, the imbalanced learning ensemble techniques are powerful tools for the establishment of high-performance ensemble classification models for detecting mineralization anomalies from geochemical exploration data.

Brain natriuretic peptide and N-terminal brain natriuretic peptide for the diagnosis of haemodynamically significant patent ductus arteriosus in preterm neonates.

Brain natriuretic peptide and N-terminal brain natriuretic peptide for the diagnosis of haemodynamically significant patent ductus arteriosus in preterm neonates.

VISROC 2.0: Updated Software for the Visualization of the significance of Receiver Operating Characteristics based on confidence ellipses

The Receiver Operating Characteristics (ROC) method is used to evaluate the diagnostic accuracy of binary quantitative tests in a broad spectrum of disciplines, including medicine, physics of complex systems, geophysics, meteorology, etc. The estimation of the significance of the examined prediction method is of high importance and it's usually approximated by Monte Carlo calculations. To simplify this problem, a FORTRAN code called VISROC was submitted to the CPC Program Library in 2014. VISROC evaluates the significance of binary diagnostic and prognostic tools for a family of k-ellipses which are based on confidence ellipses and cover the whole ROC space. Since that time, the code has been significantly improved and several new capabilities have been added. Most importantly, a Graphical User Interface (GUI) has been implemented, which can be invoked using either the R shiny web application or the Python application available for Windows, Mac, and Linux operating systems, both of which are described here. New version program summaryProgram Title: VISROC 2.0CPC Library link to program files:https://doi.org/10.17632/dktybrydvf.1Licensing provisions: AGPLv3Programming language: R and PythonJournal reference of previous version: N.V. Sarlis, S.-R. G. Christopoulos, Comput. Phys. Commun. 185 (2014) 1172.Does the new version supersede the previous version?: YesReasons for the new version: Overcome previous version's limitations and implement GUI.Summary of revisions: R and Python applications, GUI, ability to input files containing ROC data.Nature of problem: The Receiver Operating Characteristics (ROC) [1] is a method used to evaluate the diagnostic ability of binary tests and prediction methods in a broad spectrum of disciplines. Apart from the sensitivity (or True Positive rate, TPr) and specificity (which is complementary to False Positive rate, FPr, i.e., specificity=1-FPr) measures, the estimation of the statistical significance of the examined method is of high importance. VISROC evaluates the significance of binary diagnostic and prognostic tools for a family of k-ellipses which are based on confidence ellipses and cover the whole ROC space.Solution method: Using the statistics of random binary predictions for a given value of the predictor threshold ϵt, one can construct the corresponding confidence ellipses. The envelope of these confidence ellipses is estimated by varying ϵt in the interval [0,1] and one obtains a new family of ellipses, called k-ellipses [2]. They cover the whole ROC space and lead to a well defined Area Under the Curve (AUC). Mason and Graham [3] have shown that AUC follows the Mann-Whitney U-statistics [4] which can be used [5] to estimate the statistical significance of each k-ellipse. As the transformation is invertible, any point on the ROC plane corresponds to a unique value of k, hence it belongs to a unique k-ellipse that allows the estimation of the probability (p-value) to obtain this point by chance. The present GUI applications provide the p-value on the ROC plane as well as the k-ellipses corresponding to the (p=)10%, 5% and 1% significance levels using as input the number of the positive (P) and negative (Q) cases to be predicted.

Multiclass Segmentation of Suspicious Findings in Simulated Breast Tomosynthesis Images Using a U-Net.

Our lab has built a next-generation tomosynthesis (NGT) system utilizing scanning motions with more degrees of freedom than clinical digital breast tomosynthesis systems. We are working toward designing scanning motions that are customized around the locations of suspicious findings. The first step in this direction is to demonstrate that these findings can be detected with a single projection image, which can guide the remainder of the scan. This paper develops an automated method to identify findings that are prone to be masked. Perlin-noise phantoms and synthetic lesions were used to simulate masked cancers. NGT projections of phantoms were simulated using ray-tracing software. The risk of masking cancers was mapped using the ground-truth labels of phantoms. The phantom labels were used to denote regions of low and high risk of masking suspicious findings. A U-Net model was trained for multiclass segmentation of phantom images. Model performance was quantified with a receiver operating characteristic (ROC) curve using area under the curve (AUC). The ROC operating point was defined to be the point closest to the upper left corner of ROC space. The output predictions showed an accurate segmentation of tissue predominantly adipose (mean AUC of 0.93). The predictions also indicate regions of suspicious findings; for the highest risk class, mean AUC was 0.89, with a true positive rate of 0.80 and a true negative rate of 0.83 at the operating point. In summary, this paper demonstrates with virtual phantoms that a single projection can indeed be used to identify suspicious findings.

Considerations on the region of interest in the ROC space.

Receiver Operating Characteristic curves have been widely used to represent the performance of diagnostic tests. The corresponding area under the curve, widely used to evaluate their performance quantitatively, has been criticized in several respects. Several proposals have been introduced to improve area under the curve by taking into account only specific regions of the Receiver Operating Characteristic space, that is, the plane to which Receiver Operating Characteristic curves belong. For instance, a region of interest can be delimited by setting specific thresholds for the true positive rate or the false positive rate. Different ways of setting the borders of the region of interest may result in completely different, even opposing, evaluations. In this paper, we present a method to define a region of interest in a rigorous and objective way, and compute a partial area under the curve that can be used to evaluate the performance of diagnostic tests. The method was originally conceived in the Software Engineering domain to evaluate the performance of methods that estimate the defectiveness of software modules. We compare this method with previous proposals. Our method allows the definition of regions of interest by setting acceptability thresholds on any kind of performance metric, and not just false positive rate and true positive rate: for instance, the region of interest can be determined by imposing that (also known as the Matthews Correlation Coefficient) is above a given threshold. We also show how to delimit the region of interest corresponding to acceptable costs, whenever the individual cost of false positives and false negatives is known. Finally, we demonstrate the effectiveness of the method by applying it to the Wisconsin Breast Cancer Data. We provide Python and R packages supporting the presented method.

A 3D deep learning model to predict the diagnosis of dementia with Lewy bodies, Alzheimer\u2019s disease, and mild cognitive impairment using brain 18F-FDG PET

PurposeThe purpose of this study is to develop and validate a 3D deep learning model that predicts the final clinical diagnosis of Alzheimer’s disease (AD), dementia with Lewy bodies (DLB), mild cognitive impairment due to Alzheimer’s disease (MCI-AD), and cognitively normal (CN) using fluorine 18 fluorodeoxyglucose PET (18F-FDG PET) and compare model’s performance to that of multiple expert nuclear medicine physicians’ readers.Materials and methodsRetrospective 18F-FDG PET scans for AD, MCI-AD, and CN were collected from Alzheimer’s disease neuroimaging initiative (556 patients from 2005 to 2020), and CN and DLB cases were from European DLB Consortium (201 patients from 2005 to 2018). The introduced 3D convolutional neural network was trained using 90% of the data and externally tested using 10% as well as comparison to human readers on the same independent test set. The model’s performance was analyzed with sensitivity, specificity, precision, F1 score, receiver operating characteristic (ROC). The regional metabolic changes driving classification were visualized using uniform manifold approximation and projection (UMAP) and network attention.ResultsThe proposed model achieved area under the ROC curve of 96.2% (95% confidence interval: 90.6–100) on predicting the final diagnosis of DLB in the independent test set, 96.4% (92.7–100) in AD, 71.4% (51.6–91.2) in MCI-AD, and 94.7% (90–99.5) in CN, which in ROC space outperformed human readers performance. The network attention depicted the posterior cingulate cortex is important for each neurodegenerative disease, and the UMAP visualization of the extracted features by the proposed model demonstrates the reality of development of the given disorders.ConclusionUsing only 18F-FDG PET of the brain, a 3D deep learning model could predict the final diagnosis of the most common neurodegenerative disorders which achieved a competitive performance compared to the human readers as well as their consensus.

Signs and symptoms to determine if a patient presenting in primary care or hospital outpatient settings has COVID-19 disease.

Signs and symptoms to determine if a patient presenting in primary care or hospital outpatient settings has COVID-19 disease.

Loop-mediated isothermal amplification (LAMP) test for diagnosis of uncomplicated malaria in endemic areas: a meta-analysis of diagnostic test accuracy

BackgroundThe global malaria decline has stalled and only a few countries are pushing towards pre-elimination. The aim of the malaria elimination phase is interruption of local transmission of a specified malaria parasite in a defined geographical area. New and improved screening tools and strategies are required for detection and management of very low-density parasitaemia in the field. The objective of this study was to synthesize evidence on the diagnostic accuracy of loop-mediated isothermal amplification (LAMP) test for the detection of malaria parasites among people living in endemic areas.MethodsThis study adhered to the Preferred Reporting Items for Systematic Reviews and Meta-Analysis for Diagnostic Test Accuracy (PRISMA-DTA) guideline. Relevant studies in the health-related electronic databases were searched. According to the criteria set for this study, eligible studies were identified. The quality of included studies was evaluated with the use of a quality assessment checklist. A summary performance estimates such as pooled sensitivity and specificity were stratified by type of LAMP. Bivariate model for data analyses was applied. Summary receiver operating characteristics plots were created to display the results of individual studies in a receiver operating characteristics space. Meta-regression analysis was performed to investigate the sources of heterogeneity among individual studies.ResultsTwenty-seven studies across 17 endemic countries were identified. The vast majority of studies were with unclear risk of bias in the selection of index test. Overall, the pooled test performances were high for Pan LAMP (sensitivity: 0.95, 95% CI 0.91 to 0.97; specificity: 0.98, 95% CI 0.95 to 0.99), Plasmodium falciparum (Pf) LAMP (sensitivity: 0.96, 95% CI 0.94 to 0.98; specificity: 0.99, 95% CI 0.96 to 1.00) or for Plasmodium vivax (Pv) LAMP from 6 studies (sensitivity: 0.98, 95% CI 0.92 to 0.99; specificity: 0.99, 95% CI 0.72 to 1.00). The area under the curve for Pan LAMP (0.99, 95% CI 0.98–1.00), Pf LAMP (0.99, 95% CI 0.97–0.99) and Pv LAMP was (1.00, 95% CI 0.98–1.00) indicated that the diagnostic performance of these tests were within the excellent accuracy range. Meta-regression analysis showed that sample size had the greatest impact on test performance, among other factors.ConclusionsThe current findings suggest that LAMP-based assays are appropriate for detecting low-level malaria parasite infections in the field and would become valuable tools for malaria control and elimination programmes. Future well-designed larger sample studies on LAMP assessment in passive and active malaria surveillances that use PCR as the reference standard and provide sufficient data to construct 2 × 2 diagnostic table are needed.

A Bat Algorithm-Based Data-Driven Model for Mineral Prospectivity Mapping

In data-driven mineral prospectivity mapping, a statistical model is established to represent the spatial relationship between layers of metallogenic evidence and locations of known mineral deposits, and then, the former are integrated into a mineral prospectivity model using the established model. Establishment of a data-driven mineral prospectivity model can be regarded as a process of searching for the optimal integration of layers of metallogenic evidence in order to maximize the spatial relationship between mineral prospectivity and the locations of known mineral deposits. Mineral prospectivity can be simply defined as the weighted sum of layers of metallogenic evidence. Then, the optimal integration of the layers of evidence can be determined by optimizing weight coefficients of the layers of evidence to maximize the area under the curve (AUC) of the defined model. To this end, a bat algorithm-based model is proposed for data-driven mineral prospectivity mapping. In this model, the AUC of the model is used as the objective function of the bat algorithm, and the ranges of the weight coefficients of layers of evidence are used to define the search space of the bat population, and the optimal weight coefficients are then automatically determined through the iterative search process of the bat algorithm. The bat algorithm-based model was used to map mineral prospectivity in the Helong district, Jilin Province, China. Because of the high performance of the traditional logistic regression model for data-driven mineral prospectivity mapping, it was used as a benchmark model for comparison with the bat algorithm-based model. The result shows that the receiver operating characteristic (ROC) curve of the bat algorithm-based model is coincident with that of the logistic regression model in the ROC space. The AUC of the bat algorithm-based model (0.88) is slightly larger than that of the logistic regression model (0.87). The optimal threshold for extracting mineral targets was determined by using the Youden index. The mineral targets optimally delineated by using the bat algorithm-based model and logistic regression model account for 8.10% and 11.24% of the study area, respectively, both of which contain 79% of the known mineral deposits. These results indicate that the performance of the bat algorithm-based model is comparable with that of the logistic regression model in data-driven mineral prospectivity mapping. Therefore, the bat algorithm-based model is a potentially useful high-performance data-driven mineral prospectivity mapping model.

Evaluating autostereoscopic 3D baseball games using signal detection theory and receiver operating characteristic space

This study investigates the effects of display technique (2D, autostereoscopic 3D), ball speed (138, 140 km/h), and operation time (5, 10, 15 min) on the four outcomes of signal detection theory (SDT) (i.e., hit, miss, false alarm, correct rejection), β, d’, receiver operating characteristic (ROC) space, Simulator Sickness Questionnaire (SSQ), and the iGroup Presence Questionnaire (IPQ). The results indicated that the display technique was significant on the hit rate, SSQ, and IPQ, where a higher hit rate, visual fatigue, and IPQ were found in the 3D technique. The results also showed that the ball speed was significant on the miss rate and d’, where a low miss rate and high d’ were found in the higher speed of 138 km/h. The results further demonstrated that the operation time was significant on the false alarm rate, correct rejection rate, d’, SSQ, and IPQ, with a long operation time being associated with a better performance for every variable, except the SSQ. Relevance to industryFrom the results obtained regarding the ROC space, this study found that the possibility of participants’ misjudgements was low, and the accuracy of this research is considered reliable. The results of this study could serve as a reference for N3DS and game manufacturers in designing future products.

A Bat-Optimized One-Class Support Vector Machine for Mineral Prospectivity Mapping

One-class support vector machine (OCSVM) is an efficient data-driven mineral prospectivity mapping model. Since the parameters of OCSVM directly affect the performance of the model, it is necessary to optimize the parameters of OCSVM in mineral prospectivity mapping. Trial and error method is usually used to determine the “optimal” parameters of OCSVM. However, it is difficult to find the globally optimal parameters by the trial and error method. By combining OCSVM with the bat algorithm, the intialization parameters of the OCSVM can be automatically optimized. The combined model is called bat-optimized OCSVM. In this model, the area under the curve (AUC) of OCSVM is taken as the fitness value of the objective function optimized by the bat algorithm, the value ranges of the initialization parameters of OCSVM are used to specify the search space of bat population, and the optimal parameters of OCSVM are automatically determined through the iterative search process of the bat algorithm. The bat-optimized OCSVMs were used to map mineral prospectivity of the Helong district, Jilin Province, China, and compared with the OCSVM initialized by the default parameters (i.e., common OCSVM) and the OCSVM optimized by trial and error. The results show that (a) the receiver operating characteristic (ROC) curve of the trial and error-optimized OCSVM is intersected with those of the bat-optimized OCSVMs and (b) the ROC curves of the optimized OCSVMs slightly dominate that of the common OCSVM in the ROC space. The area under the curves (AUCs) of the common and trial and error-optimized OCSVMs (0.8268 and 0.8566) are smaller than those of the bat-optimized ones (0.8649 and 0.8644). The optimal threshold for extracting mineral targets was determined by using the Youden index. The mineral targets predicted by the common and trial and error-optimized OCSVMs account for 29.61% and 18.66% of the study area respectively, and contain 93% and 86% of the known mineral deposits. The mineral targets predicted by the bat-optimized OCSVMs account for 19.84% and 14.22% of the study area respectively, and also contain 93% and 86% of the known mineral deposits. Therefore, we have 0.93/0.2961 = 3.1408 < 0.86/0.1866 = 4.6088 < 0.93/0.1984 = 4.6875 < 0.86/0.1422 = 6.0478, indicating that the bat-optimized OCSVMs perform slightly better than the common and trial and error-optimized OCSVMs in mineral prospectivity mapping.

Prehospital stroke scales as screening tools for early identification of stroke and transient ischemic attack.

This is the protocol for a review and there is no abstract. The objectives are as follows: Among the currently validated prehospital stroke scales, what is the diagnostic accuracy of the index tests for the diagnosis of stroke in prehospital screening of patients suspected of having a stroke? To assess the influence of the following potential sources of heterogeneity. Patient demographics (e.g. age, gender). Type of event (transient ischemic attack (TIA), ischemic or hemorrhagic). The definition of TIA used by the study. Level of training of paramedic staff. Items from the methodological quality checklist.

Automatic fault detection in seismic data using Gaussian process regression

Compartmentalization of hydrocarbon reservoirs, change in fluid contacts, effects of high permeable fractures around faults, and hydrocarbon traps created by seal faults make fault detection and extraction as a necessity in the analysis of seismic data. Faulting disrupts the smoothness trend of geological layers (reflections in seismic sections) and displaces layers along its plane. Thus, a fault could be considered as an abnormal phenomenon that globally deviates normal behavior of layers around its plane. In the present study, faults are considered as sparse global anomalies in a seismic section that can be extracted using Gaussian process regression. The Gaussian process regression is a nonparametric probabilistic model based on Bayesian statistics that can be used to model spatial properties as a regression problem. The Gaussian process usually is used to extract and describe normal interactions from the data set using smooth functions.The main idea of this study is to detect the global anomaly using Gaussian process regression. For this purpose, we considered geological layers as smooth normal events in seismic sections. Therefore, the location of the fault plane is where the Gaussian process gets an error during describing the layers. Abnormalities such as faults cause the Gaussian process to suffer an error near the anomaly. We used these errors and analyzed them to detect probable locations of fault edge. Finally, we used a consistent connection algorithm to separate most probable fault points and to connect them to an edge using morphological reconstruction algorithm. The proposed algorithm was evaluated based on the receiver operating characteristics analysis. Several synthetic seismic sections with different levels of signal to noise ratios were used to evaluate the algorithm in the presence of random noise. The results showed that all points predicted by a diagnostic test fell into the area above the diagonal of the receiver operating characteristics space, which represents a good diagnostic classification.

Diagnostic accuracy of contemporary and high-sensitivity cardiac troponin assays used in serial testing, versus single-sample testing as a comparator, to triage patients suspected of acute non-ST-segment elevation myocardial infarction: a systematic review protocol

IntroductionAlthough the new generation of cardiac troponin assays have revolutionised the diagnosis of myocardial infarction (MI), their application in triaging patients with suspected acute coronary syndrome requires further investigation. The...

3D ROC Histogram: A New ROC Analysis Tool Incorporating Information on Instances

While Receiver Operator Characteristic (ROC) curves have been a standard tool in the design and evaluation of binary classification problems, they have sometimes been blamed for ignoring some vital information in the evaluation process, such as predicted scores and the amount of information about the target that each instance carries. In this paper, a new classification performance method denoted as 3D ROC histogram is proposed for extending ROC curves into 3D space. In this histogram, the x-axis and the y-axis are respectively labeled as false positive rate, and true positive rate which are the same with traditional ROC space. The z-axis serves as a quantitative index that represents vital information, and the volume of the 3D ROC histogram (V3RH) acts as a summary index. The proposed method preserves merits such as robustness with respect to class imbalance and threshold independence, and also, it provides an easy way for incorporating additional information in the evaluation process. Experiments on real-world datasets were conducted, with results that confirmed it to be a reliable measure.

A Deep Learning Model to Predict a Diagnosis of Alzheimer Disease by Using 18F-FDG PET of the Brain.

Purpose To develop and validate a deep learning algorithm that predicts the final diagnosis of Alzheimer disease (AD), mild cognitive impairment, or neither at fluorine 18 (18F) fluorodeoxyglucose (FDG) PET of the brain and compare its performance to that of radiologic readers. Materials and Methods Prospective 18F-FDG PET brain images from the Alzheimer's Disease Neuroimaging Initiative (ADNI) (2109 imaging studies from 2005 to 2017, 1002 patients) and retrospective independent test set (40 imaging studies from 2006 to 2016, 40 patients) were collected. Final clinical diagnosis at follow-up was recorded. Convolutional neural network of InceptionV3 architecture was trained on 90% of ADNI data set and tested on the remaining 10%, as well as the independent test set, with performance compared to radiologic readers. Model was analyzed with sensitivity, specificity, receiver operating characteristic (ROC), saliency map, and t-distributed stochastic neighbor embedding. Results The algorithm achieved area under the ROC curve of 0.98 (95% confidence interval: 0.94, 1.00) when evaluated on predicting the final clinical diagnosis of AD in the independent test set (82% specificity at 100% sensitivity), an average of 75.8 months prior to the final diagnosis, which in ROC space outperformed reader performance (57% [four of seven] sensitivity, 91% [30 of 33] specificity; P < .05). Saliency map demonstrated attention to known areas of interest but with focus on the entire brain. Conclusion By using fluorine 18 fluorodeoxyglucose PET of the brain, a deep learning algorithm developed for early prediction of Alzheimer disease achieved 82% specificity at 100% sensitivity, an average of 75.8 months prior to the final diagnosis. © RSNA, 2018 Online supplemental material is available for this article. See also the editorial by Larvie in this issue.

Partial Youden index and its inferences

ABSTRACT In medical diagnostic research, medical tests with continuous values are widely employed to distinguish between diseased and non-diseased subjects. The diagnostic accuracy of a medical test can be assessed by using the receiver operating characteristic (ROC) curve of the test. To summarize the ROC curve and determine an optimal cut-off point for test results, the Youden index is commonly used. In particular, the Youden index is optimized over the entire range of values for sensitivity and specificity, which determine the ROC space. However in clinical practice, one may only be interested in the regions of the ROC curve that correspond to low false-positive rates or/and high sensitivities. In this paper, a new summary index for the ROC curve, called the “partial Youden index”, is defined on regions of the ROC space in which sensitivity/specificity values are of practical interest. The traditional Youden index is a special case of the partial Youden index. Various parametric and non-parametric interval estimation methods are proposed for the partial Youden index. Extensive simulation studies are conducted to evaluate the finite sample performances of the proposed methods. A real example is used to illustrate the application of the new methods.

3D fast convex-hull-based evolutionary multiobjective optimization algorithm

The receiver operating characteristic (ROC) and detection error tradeoff (DET) curves have been widely used in the machine learning community to analyze the performance of classifiers. The area (or volume) under the convex hull has been used as a scalar indicator for the performance of a set of classifiers in ROC and DET space. Recently, 3D convex-hull-based evolutionary multiobjective optimization algorithm (3DCH-EMOA) has been proposed to maximize the volume of convex hull for binary classification combined with parsimony and three-way classification problems. However, 3DCH-EMOA revealed high consumption of computational resources due to redundant convex hull calculations and a frequent execution of nondominated sorting. In this paper, we introduce incremental convex hull calculation and a fast replacement for non-dominated sorting. While achieving the same high quality results, the computational effort of 3DCH-EMOA can be reduced by orders of magnitude. The average time complexity of 3DCH-EMOA in each generation is reduced from O(n2logn) to O(nlogn) per iteration, where n is the population size. Six test function problems are used to test the performance of the newly proposed method, and the algorithms are compared to several state-of-the-art algorithms, including NSGA-III, RVEA, etc., which were not compared to 3DCH-EMOA before. Experimental results show that the new version of the algorithm (3DFCH-EMOA) can speed up 3DCH-EMOA for about 30 times for a typical population size of 300 without reducing the performance of the method. Besides, the proposed algorithm is applied for neural networks pruning, and several UCI datasets are used to test the performance.

The choice of methods in determining the optimal cut-off value for quantitative diagnostic test evaluation.

Introduction The choice of criteria in determining optimal cut-off value is a matter of concern in quantitative diagnostic tests. Several indexes such as Youden's index, Euclidean index, product of sensitivity and specificity in receiver operator characteristic space and diagnostic odds ratio have been used in clinical practices but their advantages and limitations are not well understood by clinicians. This study aimed to compare these methods in determining optimal cut-off values for quantitative diagnostic test. Methods The different configurations of binormal and bilogistic distributions with equal and unequal variances for nondiseased and diseased subjects were examined. The cut-off values with increment of 0.1 in Z-scale were varied. Then, the Youden's index, Euclidean index, product of sensitivity and specificity, and diagnostic odds ratio were calculated over various cut-off values under distributional assumptions with confirmed parameters. Results According to the obtained data from binormal model and equal variances, the optimal cut-off values derived from Youden's index, Euclidean index, and product method were similar but the diagnostic odds ratio yielded either extremely low or extremely high optimal cut-off value. For bilogistic pair distributions with equal variances, the Youden's, Euclidean indexes and product method resulted in an identical cut-off value but the diagnostic odds ratio was constant over various cut-points. By both binormal and bilogistic data with more variations in nondiseased population, the Youden's index produced a higher sensitive optimal cut-off value; but with more variation for diseased distribution, the Euclidean index showed a more sensitive optimal cut-off. For bilogistic data with unequal variance, the log(diagnostic odds ratio) had a straight line relationship over cut-off values with either positive or negative slope. Conclusion As a measure of association, diagnostic odds ratio cannot be informative in determining an optimal cut-off value. The advantage of receiver operator characteristic analysis to obtain the optimal cut-off value is to use Youden's index, Euclidean index, or product index which is recommended. The choice between them depends on variability of test results in diseased and nondiseased subjects and the desired sensitivity.

Optimal threshold estimation for binary classifiers using game theory.

Many bioinformatics algorithms can be understood as binary classifiers. They are usually compared using the area under the receiver operating characteristic ( ROC) curve. On the other hand, choosing the best threshold for practical use is a complex task, due to uncertain and context-dependent skews in the abundance of positives in nature and in the yields/costs for correct/incorrect classification. We argue that considering a classifier as a player in a zero-sum game allows us to use the minimax principle from game theory to determine the optimal operating point. The proposed classifier threshold corresponds to the intersection between the ROC curve and the descending diagonal in ROC space and yields a minimax accuracy of 1-FPR. Our proposal can be readily implemented in practice, and reveals that the empirical condition for threshold estimation of "specificity equals sensitivity" maximizes robustness against uncertainties in the abundance of positives in nature and classification costs.