Evaluating Classification Accuracy Research Articles

Selecting and Interpreting Multiclass Loss and Accuracy Assessment Metrics for Classifications with Class Imbalance: Guidance and Best Practices

Evaluating classification accuracy is a key component of the training and validation stages of thematic map production, and the choice of metric has profound implications for both the success of the training process and the reliability of the final accuracy assessment. We explore key considerations in selecting and interpreting loss and assessment metrics in the context of data imbalance, which arises when the classes have unequal proportions within the dataset or landscape being mapped. The challenges involved in calculating single, integrated measures that summarize classification success, especially for datasets with considerable data imbalance, have led to much confusion in the literature. This confusion arises from a range of issues, including a lack of clarity over the redundancy of some accuracy measures, the importance of calculating final accuracy from population-based statistics, the effects of class imbalance on accuracy statistics, and the differing roles of accuracy measures when used for training and final evaluation. In order to characterize classification success at the class level, users typically generate averages from the class-based measures. These averages are sometimes generated at the macro-level, by taking averages of the individual-class statistics, or at the micro-level, by aggregating values within a confusion matrix, and then, calculating the statistic. We show that the micro-averaged producer’s accuracy (recall), user’s accuracy (precision), and F1-score, as well as weighted macro-averaged statistics where the class prevalences are used as weights, are all equivalent to each other and to the overall accuracy, and thus, are redundant and should be avoided. Our experiment, using a variety of loss metrics for training, suggests that the choice of loss metric is not as complex as it might appear to be, despite the range of choices available, which include cross-entropy (CE), weighted CE, and micro- and macro-Dice. The highest, or close to highest, accuracies in our experiments were obtained by using CE loss for models trained with balanced data, and for models trained with imbalanced data, the highest accuracies were obtained by using weighted CE loss. We recommend that, since weighted CE loss used with balanced training is equivalent to CE, weighted CE loss is a good all-round choice. Although Dice loss is commonly suggested as an alternative to CE loss when classes are imbalanced, micro-averaged Dice is similar to overall accuracy, and thus, is particularly poor for training with imbalanced data. Furthermore, although macro-Dice resulted in models with high accuracy when the training used balanced data, when the training used imbalanced data, the accuracies were lower than for weighted CE. In summary, the significance of this paper lies in its provision of readers with an overview of accuracy and loss metric terminology, insight regarding the redundancy of some measures, and guidance regarding best practices.

Manifold cluster-based evolutionary ensemble imbalance learning

For an imbalanced dataset, traditional machine learning methods usually misclassify minority samples due to the indicator evaluating classification accuracy biased toward majority class. To address the issue, manifold cluster-based evolutionary ensemble imbalance learning is proposed, with the purpose of providing a more effective framework for building an optimal imbalance classifier. After mapping the original data to manifold space, majority samples are removed from each sub-cluster in terms of their distribution characteristic. Following that, a new one is generated in each minority sub-cluster by over-sampling, with the purpose of avoiding a misclassified new minority sample that produced from small disjuncts. In above manifold clustering-based resampling techniques, optional operations and key parameters for normalization, manifold learning, clustering, under-sampling and over-sampling form various combination. Thus, evolutionary algorithm is introduced to seek the optimal structure for MECS-Ensemble. Each individual is encoded by five integer and six real number, and a fitness function is designed to evaluate its classification accuracy and the diversity of majority samples. The statistical experimental results for 39 imbalanced datasets show that MECS-Ensemble proposed in the paper is superior to the other imbalance learning methods, especially, manifold clustering-based resampling technique contributes to significant performance improvements.

Re] Resampling methods for evaluating classification accuracy of wildlife habitat models

Re] Resampling methods for evaluating classification accuracy of wildlife habitat models

Evaluating classification accuracy for modern learning approaches.

Deep learning neural network models such as multilayer perceptron (MLP) and convolutional neural network (CNN) are novel and attractive artificial intelligence computing tools. However, evaluation of the performance of these methods is not readily available for practitioners yet. We provide a tutorial for evaluating classification accuracy for various state-of-the-art learning approaches, including familiar shallow and deep learning methods. For qualitative response variables with more than two categories, many traditional accuracy measures such as sensitivity, specificity, and area under the receiver operating characteristic curve are not applicable and we have to consider their extensions properly. In this paper, a few important statistical concepts for multicategory classification accuracy are reviewed and their utilities for various learning algorithms are demonstrated with real medical examples. We offer problem-based R code to illustrate how to perform these statistical computations step by step. We expect that such analysis tools will become more familiar to practitioners and receive broader applications in biostatistics.

Universal screening with automated essay scoring: Evaluating classification accuracy in grades 3 and 4

The adoption of the Common Core State Standards and its associated assessments has placed increased focus on writing performance. Consequently, weak writers may be at risk of failing Common Core English language arts (ELA) assessments. Thus, the current study sampled a diverse group of third and fourth grade students (n=100 and 130, respectively) who were administered Fall and Spring writing screeners using the procedures of a Direct Assessment of Writing (DAW). Results were used to predict whether students did or did not attain grade-level standards as evaluated by the summative Smarter Balanced ELA assessment. Writing screeners were scored using the Project Essay Grade (PEG) automated essay scoring system. ROC curve analysis and logistic regression were used to evaluate screening models. Area under the ROC curve (AUC) values for grade 3 were in the acceptable range (Fall=0.74, Spring=0.75). AUCs approached or fell within the excellent range for grade 4 (Fall=0.79, Spring=0.83). Sensitivity-based and d-based cutpoints were selected and measures of diagnostic accuracy, including sensitivity and specificity, are reported. Results indicate that automatically-scored DAW has promise for universal screening for writing risk.

Evaluating classification accuracy: the impact of resampling and dataset size

Correct prediction is important criterion in evaluating classifiers in supervised learning context. The accuracy rate is a widely accepted indicator of the probability of misclassification of a classifier. Nevertheless, true accuracy remains unknown in most cases since it is not always possible to include the whole population in a study, and it is difficult to calculate the probability distribution of the data. Therefore, researchers often rely on computing estimation from the available data through sampling. When the available data is small or limited, it is common to rely on a resampling technique for accuracy estimation. In this paper, we study the impact of the resampling against non-resampling estimation method, with different dataset sizes on the sample distribution variance. Initial results indicate that there is a significant difference in the variance of the sample distribution between resampling and non-resampling. We also found that the larger the dataset size, the less significant the difference in variance.

Evaluating classification accuracy: the impact of resampling and dataset size

Evaluating classification accuracy: the impact of resampling and dataset size

MapReduce based parallel gene selection method

Microarray data analysis has been widely used for extracting relevant biological information from thousands of genes simultaneously expressed in a specific cell. Although many genes are expressed in a sample tissue, most of these are irrelevant or insignificant for clinical diagnosis or disease classification because of missing values and noises. Thus, finding a small, closely related gene set to accurately classify disease cells is an important research problem. At the same time, scalable gene selection methods are required for microarray data analysis due to rapidly increasing volume of microarray data. In this paper, we propose a scalable parallel gene selection method using the M a p R e u d c e programming model. The proposed method utilizes the kNN classifier algorithm for evaluating classification accuracy and uses four real and three synthetic datasets for experiments. Experimental results show that the proposed method can offer good scalability on large data with increasing number of nodes and it can also provide higher classification accuracy rather than using whole gene set for classification.

Automatic intelligibility classification of sentence-level pathological speech

Pathological speech usually refers to the condition of speech distortion resulting from atypicalities in voice and/or in the articulatory mechanisms owing to disease, illness or other physical or biological insult to the production system. Although automatic evaluation of speech intelligibility and quality could come in handy in these scenarios to assist experts in diagnosis and treatment design, the many sources and types of variability often make it a very challenging computational processing problem. In this work we propose novel sentence-level features to capture abnormal variation in the prosodic, voice quality and pronunciation aspects in pathological speech. In addition, we propose a post-classification posterior smoothing scheme which refines the posterior of a test sample based on the posteriors of other test samples. Finally, we perform feature-level fusions and subsystem decision fusion for arriving at a final intelligibility decision. The performances are tested on two pathological speech datasets, the NKI CCRT Speech Corpus (advanced head and neck cancer) and the TORGO database (cerebral palsy or amyotrophic lateral sclerosis), by evaluating classification accuracy without overlapping subjects’ data among training and test partitions. Results show that the feature sets of each of the voice quality subsystem, prosodic subsystem, and pronunciation subsystem, offer significant discriminating power for binary intelligibility classification. We observe that the proposed posterior smoothing in the acoustic space can further reduce classification errors. The smoothed posterior score fusion of subsystems shows the best classification performance (73.5% for unweighted, and 72.8% for weighted, average recalls of the binary classes).

Remote sensing of tea plantations using an SVM classifier and pattern-based accuracy assessment technique

Tea (Camellia sp.) and its plantation are very important on a worldwide scale as it is the second-most consumed beverage after water. Therefore, it becomes necessary to map the widely distributed tea plantations under various geographies and conditions. Remote-sensing techniques are effective tools to map and monitor the impact of tea plantation on land-use/land-cover (LULC). Remote sensing of tea plantations suffers from spectral mixing as these plantation areas are generally surrounded by similar types of green vegetation such as orchards and bushes. This problem is mainly tied to planting style, topography, and spectral characteristics of tea plantations, and the side effects are observed as low classification accuracies after the classification process. In this study, to overcome this problem, a three-step approach was proposed and implemented on a test area with high slope. As a first step, spectral and multi-scale textural features based on Gabor filters were extracted from high resolution multispectral digital aerial images. Similarly, based on the wavelength range of the sensor, a modified normalized difference vegetation index (MNDVI) was applied to distinguish the green vegetation cover from other LULCs. The second step involves the classification of multidimensional textural and spectral feature combinations using a support vector machine (SVM) algorithm. As a final step, two different techniques were applied for evaluating classification accuracy. The first one is a traditional site-specific accuracy assessment based on a confusion matrix calculating statistical metrics for different feature combinations. The overall accuracy and kappa values were calculated as 93.68% and 0.92, 93.82% and 0.92, and 97.40% and 0.97 for LULC maps produced by red, green, and blue (RGB), RGB + MNDVI, and RGB + MNDVI + Gabor features, respectively. The second accuracy assessment technique was the pattern-based accuracy assessment. The technique involves polygon-based fuzzy local matching. Three comparison maps showing local matching indices were obtained and used to compute the global matching index (g) for LULC maps of each feature set combination. The g values were g(RGB) (0.745), g(RGB+MNDVI) (0.745), and g(RGB+MNDVI+Gabor) (0.765) for comparison maps. Finally, based on accuracy assessment metrics, the study area was successfully classified and tea plantation features were extracted with high accuracy.

An Evaluation of Item Response Theory Classification Accuracy and Consistency Indices

This article introduces two new classification consistency indices that can be used when item response theory (IRT) models have been applied. The new indices are shown to be related to Rudner’s classification accuracy index and Guo’s classification accuracy index. The Rudner- and Guo-based classification accuracy and consistency indices are evaluated and compared with estimates from the more commonly applied IRT-recursive procedure using a simulation study and data from two large-scale assessments. Results from the simulation study and practical examples suggested that the Guo- and Rudner-based indices tended to produce estimates that were closer to the simulated values and exceeded those from the IRT-recursive-based procedure. However, results did suggest that the Rudner- and Guo-based indices can have some undesirable features that are important to keep in mind when applying them in practice. The values of the classification accuracy and consistency indices appeared to be affected by a number of factors including the distribution of examinees, test length, the placement of the cut-scores, and the proficiency estimators applied to estimate examinee ability. Suggestions are made that an important part of investigations evaluating classification accuracy and consistency indices should be the creation of figures that show the value of the classification accuracy and classification consistency for individual examinees across the range of possible scores as these figures can help provide indications into subtle and important differences between indices.

Evaluating technologies for classification and prediction in medicine

Modern technologies promise to provide new ways of diagnosing disease, detecting subclinical disease, predicting prognosis, selecting patient specific treatment, identifying subjects at risk for disease, and so forth. Advances in genomics, proteomics and imaging modalities in particular hold great potential for assisting with classification/prediction in medicine. Before a classifier can be adopted for routine use in health care, its classification accuracy must be determined. Standards for evaluating new clinical classifiers however, lag far behind the well established standards that exist for evaluating new clinical treatments. In this paper, we discuss a phased approach to developing a new classifier (or biomarker). It mirrors the internationally established phase 1-2-3 paradigm for therapeutic drugs. The defined phases lead to a logical sequence of studies for classifier development. We emphasize that evaluating classification accuracy is fundamentally different from simply establishing association with outcome. Therefore, study objectives and designs differ from the familiar methods of clinical trials. We discuss these briefly for each phase.Finally, we argue that classifier development requires some rethinking of traditional data analysis techniques. As an example we show that maximizing the likelihood function to fit a logistic regression model to multiple predictors, can yield a poor classifier. Instead we demonstrate that an approach that maximizes an alternative objective function characterizing classification accuracy performs better.

Resampling methods for evaluating classification accuracy of wildlife habitat models

Predictive models of wildlife-habitat relationships often have been developed without being tested The apparent classification accuracy of such models can be optimistically biased and misleading. Data resampling methods exist that yield a more realistic estimate of model classification accuracy These methods are simple and require no new sample data. We illustrate these methods (cross-validation, jackknife resampling, and bootstrap resampling) with computer simulation to demonstrate the increase in precision of the estimate. The bootstrap method is then applied to field data as a technique for model comparison We recommend that biologists use some resampling procedure to evaluate wildlife habitat models prior to field evaluation.