Classification Performance Measures Research Articles

Fuzzified MCDM Consistent Ranking Feature Selection with Hybrid Algorithm for Credit Risk Assessment

Feature selection algorithms that are based on different single evaluation criterions for determining the subset of features shows varying result sets which lead to inconsistency in ranks. In contrary, Multiple Criteria Decision Making (MCDM) with Fuzzified Feature Selection methodology brings consistency in feature selection ranking with optimal features and improving the classification performance of credit risks. By adopting multiple evaluation criteria inconsistent ranks to Fuzzy Analytic Hierarchy Process (FAHP) for feature selection along with hybrid algorithm (K-Means clustering-Logistic Regression classification) results in enabling Consistent Ranking Feature Selection (CRFS) and significant improvement over classification performance measures. When the proposed methodology is used with two different credit risk data set from the UCI repository, the experimental results show that the optimal features with hybrid algorithm, indicating improvements in the performance of classification in credit risk prediction over the current existing techniques.

Visualization and Comparison of Single and Combined Parametric and Nonparametric Discriminant Methods for Leukemia Type Recognition Based on Gene Expression

Abstract A gene expression data set, containing 3051 genes and 38 tumor mRNA training samples, from a leukemia microarray study, was used for differentiation between ALL and AML groups of leukemia. In this paper, single and combined discriminant methods were applied on the basis of the selected few most discriminative variables according to Wilks’ lambda or the leave-one-out error of first nearest neighbor classifier. For the linear, quadratic, regularized, uncorrelated discrimination, kernel, nearest neighbor and naive Bayesian classifiers, two-dimensional graphs of the boundaries and discriminant functions for diagnostics are presented. Cross-validation and leave-one-out errors were used as measures of classifier performance to support diagnosis coming from this genomic data set. A small number of best discriminating genes, from two to ten, was sufficient to build discriminant methods of good performance. Especially useful were nearest neighbor methods. The results presented herein were comparable with outcomes obtained by other authors for larger numbers of applied genes. The linear, quadratic, uncorrelated Bayesian and regularized discrimination methods were subjected to bagging or boosting in order to assess the accuracy of the fusion. A conclusion drawn from the analysis was that resampling ensembles were not beneficial for two-dimensional discrimination.

Patient-Specific Variations in Biomarkers across Gingivitis and Periodontitis

This study investigates the use of saliva, as an emerging diagnostic fluid in conjunction with classification techniques to discern biological heterogeneity in clinically labelled gingivitis and periodontitis subjects (80 subjects; 40/group) A battery of classification techniques were investigated as traditional single classifier systems as well as within a novel selective voting ensemble classification approach (SVA) framework. Unlike traditional single classifiers, SVA is shown to reveal patient-specific variations within disease groups, which may be important for identifying proclivity to disease progression or disease stability. Salivary expression profiles of IL-1ß, IL-6, MMP-8, and MIP-1α from 80 patients were analyzed using four classification algorithms (LDA: Linear Discriminant Analysis [LDA], Quadratic Discriminant Analysis [QDA], Naïve Bayes Classifier [NBC] and Support Vector Machines [SVM]) as traditional single classifiers and within the SVA framework (SVA-LDA, SVA-QDA, SVA-NB and SVA-SVM). Our findings demonstrate that performance measures (sensitivity, specificity and accuracy) of traditional classification as single classifier were comparable to that of the SVA counterparts using clinical labels of the samples as ground truth. However, unlike traditional single classifier approaches, the normalized ensemble vote-counts from SVA revealed varying proclivity of the subjects for each of the disease groups. More importantly, the SVA identified a subset of gingivitis and periodontitis samples that demonstrated a biological proclivity commensurate with the other clinical group. This subset was confirmed across SVA-LDA, SVA-QDA, SVA-NB and SVA-SVM. Heatmap visualization of their ensemble sets revealed lack of consensus between these subsets and the rest of the samples within the respective disease groups indicating the unique nature of the patients in these subsets. While the source of variation is not known, the results presented clearly elucidate the need for novel approaches that accommodate inherent heterogeneity and personalized variations within disease groups in diagnostic characterization. The proposed approach falls within the scope of P4 medicine (predictive, preventive, personalized, and participatory) with the ability to identify unique patient profiles that may predict specific disease trajectories and targeted disease management.

Neural network based speech enhancement applied to cochlear implant coding strategies

Traditionally, algorithms that attempt to significantly improve speech intelligibility in noise for cochlear implant (CI) users have met with limited success, especially in the presence of a fluctuating masker. Motivated by previous intelligibility studies of speech synthesized using the ideal binary mask, we propose a framework that integrates a multi-layer feed-forward artificial neural network (ANN) into CI coding strategies. The algorithm decomposes the noisy input signal into time-frequency units, extracts a set of auditory-inspired features and feeds them to the ANN to produce an estimation of which frequency channels contain more perceptually important information (higher signal-to-noise ratio, (SNR)). This estimate is then used accordingly to suppress the noise and retain the appropriate subset of channels for electrical stimulation, as in traditional N-of-M coding strategies. Speech corrupted by various noise types at different SNRs is processed by the algorithm and re-synthesized with a vocoder. Evaluation has been performed in comparison with the Advanced Combination Encoder (ACE) in terms of classification performance and objective intelligibility measures. Results indicated significant improvement in Hit—False Alarm rates and intelligibility prediction scores, especially in negative SNR conditions. Findings suggested that the use of ANNs could potentially improve speech intelligibility in noise for CI users and motivated subjective listening experiments that will be presented together with the objective results.

Effects of imaging modalities, brain atlases and feature selection on prediction of Alzheimer's disease

BackgroundThe choice of biomarkers for early detection of Alzheimer's disease (AD) is important for improving the accuracy of imaging-based prediction of conversion from mild cognitive impairment (MCI) to AD. The primary goal of this study was to assess the effects of imaging modalities and brain atlases on prediction. We also investigated the influence of support vector machine recursive feature elimination (SVM-RFE) on predictive performance. MethodsEighty individuals with amnestic MCI [40 developed AD within 3 years] underwent structural magnetic resonance imaging (MRI) and 18F-fluorodeoxyglucose positron emission tomography (FDG-PET) scans at baseline. Using Automated Anatomical Labeling (AAL) and LONI Probabilistic Brain Atlas (LPBA40), we extracted features representing gray matter density and relative cerebral metabolic rate for glucose in each region of interest from the baseline MRI and FDG-PET data, respectively. We used linear SVM ensemble with bagging and computed the area under the receiver operating characteristic curve (AUC) as a measure of classification performance. We performed multiple SVM-RFE to compute feature ranking. We performed analysis of variance on the mean AUCs for eight feature sets. ResultsThe interactions between atlas and modality choices were significant. The main effect of SVM-RFE was significant, but the interactions with the other factors were not significant. Comparison with existing methodMultimodal features were found to be better than unimodal features to predict AD. FDG-PET was found to be better than MRI. ConclusionsImaging modalities and brain atlases interact with each other and affect prediction. SVM-RFE can improve the predictive accuracy when using atlas-based features.

Identifying (Quasi) Equally Informative Subsets in Feature Selection Problems for Classification: A Max-Relevance Min-Redundancy Approach.

An emerging trend in feature selection is the development of two-objective algorithms that analyze the tradeoff between the number of features and the classification performance of the model built with these features. Since these two objectives are conflicting, a typical result stands in a set of Pareto-efficient subsets, each having a different cardinality and a corresponding discriminating power. However, this approach overlooks the fact that, for a given cardinality, there can be several subsets with similar information content. The study reported here addresses this problem, and introduces a novel multiobjective feature selection approach conceived to identify: 1) a subset that maximizes the performance of a given classifier and 2) a set of subsets that are quasi equally informative, i.e., have almost same classification performance, to the performance maximizing subset. The approach consists of a wrapper [Wrapper for Quasi Equally Informative Subset Selection (W-QEISS)] built on the formulation of a four-objective optimization problem, which is aimed at maximizing the accuracy of a classifier, minimizing the number of features, and optimizing two entropy-based measures of relevance and redundancy. This allows conducting the search in a larger space, thus enabling the wrapper to generate a large number of Pareto-efficient solutions. The algorithm is compared against the mRMR algorithm, a two-objective wrapper and a computationally efficient filter [Filter for Quasi Equally Informative Subset Selection (F-QEISS)] on 24 University of California, Irvine, (UCI) datasets including both binary and multiclass classification. Experimental results show that W-QEISS has the capability of evolving a rich and diverse set of Pareto-efficient solutions, and that their availability helps in: 1) studying the tradeoff between multiple measures of classification performance and 2) understanding the relative importance of each feature. The quasi equally informative subsets are identified at the cost of a marginal increase in the computational time thanks to the adoption of Borg Multiobjective Evolutionary Algorithm and Extreme Learning Machine as global optimization and learning algorithms, respectively.

Assessing the predictive power of newly added biomarkers

As medical research and technology advance, there are always new biomarkers found and predictive models proposed for improving the diagnostic performance of diseases. Therefore, in addition to the existing biomarkers and predictive models, how to assess new biomarkers becomes an important research problem. Many classification performance measures, which are usually based on the performance on the whole cut-off values, were applied directly to this type of problems. However, in a medical diagnosis, some cut-off points are more important, such as those points within the range of high specificity. Thus, as the partial area under the ROC curve to the area under ROC curve, we study the partial integrated discriminant improvement (pIDI) for evaluating the predictive ability of a newly added marker at a prespecified range of cut-offs. Theoretical property of estimate of the proposed measure is reported. The performance of this new measure is then compared with that of the partial area under an ROC curve. The numerical results use synthesized are presented, and a liver cancer dataset is used for demonstration purposes.

Why Does Rebalancing Class-Unbalanced Data Improve AUC for Linear Discriminant Analysis?

Many established classifiers fail to identify the minority class when it is much smaller than the majority class. To tackle this problem, researchers often first rebalance the class sizes in the training dataset, through oversampling the minority class or undersampling the majority class, and then use the rebalanced data to train the classifiers. This leads to interesting empirical patterns. In particular, using the rebalanced training data can often improve the area under the receiver operating characteristic curve (AUC) for the original, unbalanced test data. The AUC is a widely-used quantitative measure of classification performance, but the property that it increases with rebalancing has, as yet, no theoretical explanation. In this note, using Gaussian-based linear discriminant analysis (LDA) as the classifier, we demonstrate that, at least for LDA, there is an intrinsic, positive relationship between the rebalancing of class sizes and the improvement of AUC. We show that the largest improvement of AUC is achieved, asymptotically, when the two classes are fully rebalanced to be of equal sizes.

PRROC: computing and visualizing precision-recall and receiver operating characteristic curves in R.

Summary: Precision-recall (PR) and receiver operating characteristic (ROC) curves are valuable measures of classifier performance. Here, we present the R-package PRROC, which allows for computing and visualizing both PR and ROC curves. In contrast to available R-packages, PRROC allows for computing PR and ROC curves and areas under these curves for soft-labeled data using a continuous interpolation between the points of PR curves. In addition, PRROC provides a generic plot function for generating publication-quality graphics of PR and ROC curves.Availability and implementation: PRROC is available from CRAN and is licensed under GPL 3.Contact: grau@informatik.uni-halle.de

SLGAS: Supervised Learning Using Gain Ratio as Attribute Selection Measure to Nowcast Snow/No-Snow

Considering the weather nowcasting, which has no prospect of intervention, they cause the vital results in human life and animal life, accurate analysis and estimation of these variables is very important and crucial. There is an increased interest in nowcasting the bad weather conditions, among which presence of snow/no-snow is crucial, in order to fully capture the global atmospheric water cycle. This paper introduces an efficient decision tree algorithm named Supervised Learning using Gain Ratio as Attribute Selection measure (SLGAS), expanded to our previous algorithm Supervised Learning Using Entropy as Attribute Selection Measure (SLEAS), for the prediction of snow/no-snow using 31 international locations historical datasets, collected from various meteorological departments. The algorithm has been validated extensively with five performance measures namely accuracy, specificity, precision, dice and error rate respectively. Further, we compared our proposed method with the SLIQ and SLEAS decision tree algorithms in terms of the overall classification performance measures and it is clearly showing that the SLGAS algorithm is outperforming with an average accuracy of 85.92%, error rate of 14.07%.

SLGAS: Supervised Learning Using Gain Ratio as Attribute Selection Measure to Nowcast Snow/No-Snow

Considering the weather nowcasting, which has no prospect of intervention, they cause the vital results in human life and animal life, accurate analysis and estimation of these variables is very important and crucial. There is an increased interest in nowcasting the bad weather conditions, among which presence of snow/no-snow is crucial, in order to fully capture the global atmospheric water cycle. This paper introduces an efficient decision tree algorithm named Supervised Learning using Gain Ratio as Attribute Selection measure (SLGAS), expanded to our previous algorithm Supervised Learning Using Entropy as Attribute Selection Measure (SLEAS), for the prediction of snow/no-snow using 31 international locations historical datasets, collected from various meteorological departments. The algorithm has been validated extensively with five performance measures namely accuracy, specificity, precision, dice and error rate respectively. Further, we compared our proposed method with the SLIQ and SLEAS decision tree algorithms in terms of the overall classification performance measures and it is clearly showing that the SLGAS algorithm is outperforming with an average accuracy of 85.92%, error rate of 14.07%.

ClassifyR: an R package for performance assessment of classification with applications to transcriptomics.

Although a large collection of classification software packages exist in R, a new generic framework for linking custom classification functions with classification performance measures is needed. A generic classification framework has been designed and implemented as an R package in an object oriented style. Its design places emphasis on parallel processing, reproducibility and extensibility. Finally, a comprehensive set of performance measures are available to ease post-processing. Taken together, these important characteristics enable rapid and reproducible benchmarking of alternative classifiers. ClassifyR is implemented in R and can be obtained from the Bioconductor project: http://bioconductor.org/packages/release/bioc/html/ClassifyR.html.

Analysis of Demographic Characteristics Creating Coronary Artery Disease Susceptibility Using Random Forests Classifier

Abstract Cardiovascular system diseases are an important health problem. These diseases are very common also responsible for many deaths. With this study, it is aimed to analyze factors that cause Coronary Artery Disease using Random Forests Classifier. According to the analysis, we observed correct classification ratio and performance measure that creates susceptibility to Coronary Artery Disease for each factor. The performance measure results clearly show the impact of demographic characteristics on CAD. Additionally, this study shows that random forests algorithm can be used to the processing and classification of medical data such as CAD.

Relative cost curves: An alternative to AUC and an extension to 3-class problems

to specify exactly the consequences or costs of misclassifications. This paper is devoted to Relative Cost Curves (RCC) ‐ a graphical technique for visualising the performance of binary classifiers over the full range of possible relative misclassification costs. This curve provides helpful information to choose the best set of classifiers or to estimate misclassification costs if those are not known precisely. In this paper, the concept of Area Above the RCC (AAC) is introduced, a scalar measure of classifier performance under unequal misclassification costs problem. We also extend RCC to multicategory problems when misclassification costs depend only on the true class.

Investigating fitness functions for a hyper-heuristic evolutionary algorithm in the context of balanced and imbalanced data classification

In this paper, we analyse in detail the impact of different strategies to be used as fitness function during the evolutionary cycle of a hyper-heuristic evolutionary algorithm that automatically designs decision-tree induction algorithms (HEAD-DT). We divide the experimental scheme into two distinct scenarios: (1) evolving a decision-tree induction algorithm from multiple balanced data sets; and (2) evolving a decision-tree induction algorithm from multiple imbalanced data sets. In each of these scenarios, we analyse the difference in performance of well-known classification performance measures such as accuracy, F-Measure, AUC, recall, and also a lesser-known criterion, namely the relative accuracy improvement. In addition, we analyse different schemes of aggregation, such as simple average, median, and harmonic mean. Finally, we verify whether the best-performing fitness functions are capable of providing HEAD-DT with algorithms more effective than traditional decision-tree induction algorithms like C4.5, CART, and REPTree. Experimental results indicate that HEAD-DT is a good option for generating algorithms tailored to (im)balanced data, since it outperforms state-of-the-art decision-tree induction algorithms with statistical significance.

Development and application of consumer credit scoring models using profit-based classification measures

This paper presents a new approach for consumer credit scoring, by tailoring a profit-based classification performance measure to credit risk modeling. This performance measure takes into account the expected profits and losses of credit granting and thereby better aligns the model developers’ objectives with those of the lending company. It is based on the Expected Maximum Profit (EMP) measure and is used to find a trade-off between the expected losses – driven by the exposure of the loan and the loss given default – and the operational income given by the loan. Additionally, one of the major advantages of using the proposed measure is that it permits to calculate the optimal cutoff value, which is necessary for model implementation. To test the proposed approach, we use a dataset of loans granted by a government institution, and benchmarked the accuracy and monetary gain of using EMP, accuracy, and the area under the ROC curve as measures for selecting model parameters, and for determining the respective cutoff values. The results show that our proposed profit-based classification measure outperforms the alternative approaches in terms of both accuracy and monetary value in the test set, and that it facilitates model deployment.

100% Classification Accuracy Considered Harmful: The Normalized Information Transfer Factor Explains the Accuracy Paradox

The most widely spread measure of performance, accuracy, suffers from a paradox: predictive models with a given level of accuracy may have greater predictive power than models with higher accuracy. Despite optimizing classification error rate, high accuracy models may fail to capture crucial information transfer in the classification task. We present evidence of this behavior by means of a combinatorial analysis where every possible contingency matrix of 2, 3 and 4 classes classifiers are depicted on the entropy triangle, a more reliable information-theoretic tool for classification assessment.Motivated by this, we develop from first principles a measure of classification performance that takes into consideration the information learned by classifiers. We are then able to obtain the entropy-modulated accuracy (EMA), a pessimistic estimate of the expected accuracy with the influence of the input distribution factored out, and the normalized information transfer factor (NIT), a measure of how efficient is the transmission of information from the input to the output set of classes.The EMA is a more natural measure of classification performance than accuracy when the heuristic to maximize is the transfer of information through the classifier instead of classification error count. The NIT factor measures the effectiveness of the learning process in classifiers and also makes it harder for them to “cheat” using techniques like specialization, while also promoting the interpretability of results. Their use is demonstrated in a mind reading task competition that aims at decoding the identity of a video stimulus based on magnetoencephalography recordings. We show how the EMA and the NIT factor reject rankings based in accuracy, choosing more meaningful and interpretable classifiers.

A better Beta for the H measure of classification performance

The area under the ROC curve is widely used as a measure of performance of classification rules. However, it has recently been shown that the measure is fundamentally incoherent, in the sense that it treats the relative severities of misclassifications differently when different classifiers are used. The H measure overcomes this by allowing a researcher to fix the distribution of relative severities to a classifier-independent setting on a given problem. This note extends the discussion, and proposes a modified standard distribution for the H measure, which better matches the requirements of researchers, the Beta(π1+1,π0+1) distribution.

An Extensive Evaluation of Decision Tree–Based Hierarchical Multilabel Classification Methods and Performance Measures

Hierarchical multilabel classification is a complex classification problem where an instance can be assigned to more than one class simultaneously, and these classes are hierarchically organized with superclasses and subclasses, that is, an instance can be classified as belonging to more than one path in the hierarchical structure. This article experimentally analyses the behavior of different decision tree–based hierarchical multilabel classification methods based on the local and global classification approaches. The approaches are compared using distinct hierarchy‐based and distance‐based evaluation measures, when they are applied to a variation of real multilabel and hierarchical datasets' characteristics. Also, the different evaluation measures investigated are compared according to their degrees of consistency, discriminancy, and indifferency. As a result of the experimental analysis, we recommend the use of the global classification approach and suggest the use of the Hierarchical Precision and Hierarchical Recall evaluation measures.

Automated Comparison of State-Based Software Models in Terms of Their Language and Structure

State machines capture the sequential behavior of software systems. Their intuitive visual notation, along with a range of powerful verification and testing techniques render them an important part of the model-driven software engineering process. There are several situations that require the ability to identify and quantify the differences between two state machines (e.g. to evaluate the accuracy of state machine inference techniques is measured by the similarity of a reverse-engineered model to its reference model). State machines can be compared from two complementary perspectives: (1) In terms of theirlanguage-- the externally observable sequences of events that are permitted or not, and (2) in terms of theirstructure-- the actual states and transitions that govern the behavior. This article describes two techniques to compare models in terms of these two perspectives. It shows how the difference can be quantified and measured by adapting existing binary classification performance measures for the purpose. The approaches have been implemented by the authors, and the implementation is openly available. Feasibility is demonstrated via a case study to compare two real state machine inference approaches. Scalability and accuracy are assessed experimentally with respect to a large collection of randomly synthesized models.