Classification Error Estimation Research Articles

Robust importance sampling for error estimation in the context of optimal Bayesian transfer learning

SummaryClassification has been a major task for building intelligent systems because it enables decision-making under uncertainty. Classifier design aims at building models from training data for representing feature-label distributions—either explicitly or implicitly. In many scientific or clinical settings, training data are typically limited, which impedes the design and evaluation of accurate classifiers. Atlhough transfer learning can improve the learning in target domains by incorporating data from relevant source domains, it has received little attention for performance assessment, notably in error estimation. Here, we investigate knowledge transferability in the context of classification error estimation within a Bayesian paradigm. We introduce a class of Bayesian minimum mean-square error estimators for optimal Bayesian transfer learning, which enables rigorous evaluation of classification error under uncertainty in small-sample settings. Using Monte Carlo importance sampling, we illustrate the outstanding performance of the proposed estimator for a broad family of classifiers that span diverse learning capabilities.

Robust Importance Sampling for Error Estimation in the Context of Optimal Bayesian Transfer Learning

Classification has been a major task for building intelligent systems as it enables decision-making under uncertainty. Classifier design aims at building models from training data for representing feature-label distributions--either explicitly or implicitly. In many scientific or clinical settings, training data are typically limited, which makes designing accurate classifiers and evaluating their classification error extremely challenging. While transfer learning (TL) can alleviate this issue by incorporating data from relevant source domains to improve learning in a different target domain, it has received little attention for performance assessment, notably in error estimation. In this paper, we fill this gap by investigating knowledge transferability in the context of classification error estimation within a Bayesian paradigm. We introduce a novel class of Bayesian minimum mean-square error (MMSE) estimators for optimal Bayesian transfer learning (OBTL), which enables rigorous evaluation of classification error under uncertainty in a small-sample setting. Using Monte Carlo importance sampling, we employ the proposed estimator to evaluate the classification accuracy of a broad family of classifiers that span diverse learning capabilities. Experimental results based on both synthetic data as well as real-world RNA sequencing (RNA-seq) data show that our proposed OBTL error estimation scheme clearly outperforms standard error estimators, especially in a small-sample setting, by tapping into the data from other relevant domains.

Categorizing renal oncocytic neoplasms on core needle biopsy: a morphologic and immunophenotypic study of 144 cases with clinical follow-up. Alderman MA, Daignault S, Wolf JS Jr, Palapattu GS, Weizer AZ, Hafez KS, Kunju LP, Wu AJ. Hum Pathol.September 2016;55:1–10.

There is limited literature on renal oncocytic neoplasms diagnosed on core biopsy. All renal oncocytic neoplasm core biopsies from 2006 to 2013 were, retrospectively, reviewed. Morphologic features and an immunohistochemical panel of CK7, c-KIT, and S100A1 were assessed. Concordance with resection diagnosis, statistical analysis including a random forest classification, and follow-up were recorded. The postimmunohistochemical diagnoses of 144 renal oncocytic core biopsies were favor oncocytoma (67%), favor renal cell carcinoma (RCC) (12%), and cannot exclude RCC (21%). Diagnosis was revised following immunohistochemistry in 7% of cases. The most common features for oncocytoma (excluding dense granular cytoplasm) were nested architecture, edematous stroma, binucleation and tubular architecture; the most common features for favor RCC were sheet-like architecture, nuclear pleomorphism, papillary architecture, and prominent cell borders. High nuclear grade, necrosis, extensive papillary architecture, raisinoid nuclei, and frequent mitoses were not seen in oncocytomas. Comparing the pathologist and random forest classification, the overall out-of-bag estimate of classification error dropped from 23% to 13% when favor RCC and cannot exclude RCC was combined into 1 category. Resection was performed in 19% (28 cases) with a 94% concordance (100% of favor RCC biopsies and 90% of cannot exclude RCC biopsies confirmed as RCC; 83% of favor oncocytomas confirmed); ablation in 23%; and surveillance in 46%. Follow-up was available in 92% (median follow-up, 33 months) with no adverse outcomes. Renal oncocytic neoplasms comprise a significant subset (16%) of all core biopsies, and the majority (78%) can be classified as favor oncocytoma or favor RCC.

Structural MRI-based detection of Alzheimer's disease using feature ranking and classification error

Background and objectiveThis paper presents an automatic computer-aided diagnosis (CAD) system based on feature ranking for detection of Alzheimer's disease (AD) using structural magnetic resonance imaging (sMRI) data. MethodsThe proposed CAD system is composed of four systematic stages. First, global and local differences in the gray matter (GM) of AD patients compared to the GM of healthy controls (HCs) are analyzed using a voxel-based morphometry technique. The aim is to identify significant local differences in the volume of GM as volumes of interests (VOIs). Second, the voxel intensity values of the VOIs are extracted as raw features. Third, the raw features are ranked using a seven-feature ranking method, namely, statistical dependency (SD), mutual information (MI), information gain (IG), Pearson's correlation coefficient (PCC), t-test score (TS), Fisher's criterion (FC), and the Gini index (GI). The features with higher scores are more discriminative. To determine the number of top features, the estimated classification error based on training set made up of the AD and HC groups is calculated, with the vector size that minimized this error selected as the top discriminative feature. Fourth, the classification is performed using a support vector machine (SVM). In addition, a data fusion approach among feature ranking methods is introduced to improve the classification performance. ResultsThe proposed method is evaluated using a data-set from ADNI (130 AD and 130 HC) with 10-fold cross-validation. The classification accuracy of the proposed automatic system for the diagnosis of AD is up to 92.48% using the sMRI data. ConclusionsAn automatic CAD system for the classification of AD based on feature-ranking method and classification errors is proposed. In this regard, seven-feature ranking methods (i.e., SD, MI, IG, PCC, TS, FC, and GI) are evaluated. The optimal size of top discriminative features is determined by the classification error estimation in the training phase. The experimental results indicate that the performance of the proposed system is comparative to that of state-of-the-art classification models.

Categorizing renal oncocytic neoplasms on core needle biopsy: a morphologic and immunophenotypic study of 144 cases with clinical follow-up

There is limited literature on renal oncocytic neoplasms diagnosed on core biopsy. All renal oncocytic neoplasm core biopsies from 2006 to 2013 were retrospectively reviewed. Morphologic features and an immunohistochemical panel of CK7, c-KIT, and S100A1 were assessed. Concordance with resection diagnosis, statistical analysis including a random forest classification, and follow-up were recorded. The post-immunohistochemical diagnoses of 144 renal oncocytic core biopsies were favor oncocytoma (67%), favor renal cell carcinoma (RCC) (12%), and cannot exclude RCC (21%). Diagnosis was revised following immunohistochemistry in 7% of cases. The most common features for oncocytoma (excluding dense granular cytoplasm) were nested architecture, edematous stroma, binucleation and tubular architecture; the most common features for favor RCC were sheet-like architecture, nuclear pleomorphism, papillary architecture, and prominent cell borders. High nuclear grade, necrosis, extensive papillary architecture, raisinoid nuclei, and frequent mitoses were not seen in oncocytomas. Comparing the pathologist and random forest classification, the overall out-of-bag estimate of classification error dropped from 23% to 13% when favor RCC and cannot exclude RCC were combined into 1 category. Resection was performed in 19% (28 cases) with a 94% concordance (100% of favor RCC biopsies and 90% of cannot exclude RCC biopsies confirmed as RCC; 83% of favor oncocytomas confirmed); ablation in 23%; and surveillance in 46%. Follow-up was available in 92% (median follow-up, 33months) with no adverse outcomes. Renal oncocytic neoplasms comprise a significant subset (16%) of all core biopsies, and the majority (78%) can be classified as favor oncocytoma or favor RCC.

A wrapper-based approach for feature selection and classification of major depressive disorder–bipolar disorders

Feature selection (FS) and classification are consecutive artificial intelligence (AI) methods used in data analysis, pattern classification, data mining and medical informatics. Beside promising studies in the application of AI methods to health informatics, working with more informative features is crucial in order to contribute to early diagnosis. Being one of the prevalent psychiatric disorders, depressive episodes of bipolar disorder (BD) is often misdiagnosed as major depressive disorder (MDD), leading to suboptimal therapy and poor outcomes. Therefore discriminating MDD and BD at earlier stages of illness could help to facilitate efficient and specific treatment. In this study, a nature inspired and novel FS algorithm based on standard Ant Colony Optimization (ACO), called improved ACO (IACO), was used to reduce the number of features by removing irrelevant and redundant data. The selected features were then fed into support vector machine (SVM), a powerful mathematical tool for data classification, regression, function estimation and modeling processes, in order to classify MDD and BD subjects. Proposed method used coherence, a promising quantitative electroencephalography (EEG) biomarker, values calculated from alpha, theta and delta frequency bands. The noteworthy performance of novel IACO–SVM approach stated that it is possible to discriminate 46 BD and 55 MDD subjects using 22 of 48 features with 80.19% overall classification accuracy. The performance of IACO algorithm was also compared to the performance of standard ACO, genetic algorithm (GA) and particle swarm optimization (PSO) algorithms in terms of their classification accuracy and number of selected features. In order to provide an almost unbiased estimate of classification error, the validation process was performed using nested cross-validation (CV) procedure.

A continental‐scale tool for acoustic identification of European bats

Summary Acoustic methods are used increasingly to survey and monitor bat populations. However, the use of acoustic methods at continental scales can be hampered by the lack of standardized and objective methods to identify all species recorded. This makes comparable continent‐wide monitoring difficult, impeding progress towards developing biodiversity indicators, trans‐boundary conservation programmes and monitoring species distribution changes. Here we developed a continental‐scale classifier for acoustic identification of bats, which can be used throughout Europe to ensure objective, consistent and comparable species identifications. We selected 1350 full‐spectrum reference calls from a set of 15 858 calls of 34 European species, from EchoBank, a global echolocation call library. We assessed 24 call parameters to evaluate how well they distinguish between species and used the 12 most useful to train a hierarchy of ensembles of artificial neural networks to distinguish the echolocation calls of these bat species. Calls are first classified to one of five call‐type groups, with a median accuracy of 97·6%. The median species‐level classification accuracy is 83·7%, providing robust classification for most European species, and an estimate of classification error for each species. These classifiers were packaged into an online tool, iBatsID, which is freely available, enabling anyone to classify European calls in an objective and consistent way, allowing standardized acoustic identification across the continent. Synthesis and applications. iBatsID is the first freely available and easily accessible continental‐scale bat call classifier, providing the basis for standardized, continental acoustic bat monitoring in Europe. This method can provide key information to managers and conservation planners on distribution changes and changes in bat species activity through time.

On the Comparison of Classifiers for Microarray Data

The aim of many microarray experiments is to build discriminatory diagnosis and prognosis models. A large number of supervised methods have been proposed in literature for microarray-based classification. Model comparison, which is based on the classification error estimation, is a critical issue. Previous studies have shown that error estimation is unreliable in high-dimensional small-sample settings. This leads naturally to questioning the validity of classificationrule comparison approaches being used in the literature. In this paper we present a brief review of the different comparison methods used in bioinformatics. Then, we test these methods on a set of simulations based on both synthetic and real data. These simulations include different feature-label distributions, classification rules, error estimators and variance estimators. The results show that none of these methods can provide reliable comparison across a wide spectrum of feature-label distributions and classification rules. Keywords: Microarray classification, error estimation, classifier comparison, variance study

A Novel Feature Selection Methodology for Automated Inspection Systems

This paper proposes a new feature selection methodology. The methodology is based on the stepwise variable selection procedure, but, instead of using the traditional discriminant metrics such as Wilks' Lambda, it uses an estimation of the misclassification error as the figure of merit to evaluate the introduction of new features. The expected misclassification error rate (MER) is obtained by using the densities of a constructed function of random variables, which is the stochastic representation of the conditional distribution of the quadratic discriminant function estimate. The application of the proposed methodology results in significant savings of computational time in the estimation of classification error over the traditional simulation and cross-validation methods. One of the main advantages of the proposed method is that it provides a direct estimation of the expected misclassification error at the time of feature selection, which provides an immediate assessment of the benefits of introducing an additional feature into an inspection/classification algorithm.

Fast and robust bootstrap

In this paper we review recent developments on a bootstrap method for robust estimators which is computationally faster and more resistant to outliers than the classical bootstrap. This fast and robust bootstrap method is, under reasonable regularity conditions, asymptotically consistent. We describe the method in general and then consider its application to perform inference based on robust estimators for the linear regression and multivariate location-scatter models. In particular, we study confidence and prediction intervals and tests of hypotheses for linear regression models, inference for location-scatter parameters and principal components, and classification error estimation for discriminant analysis.

Prioritizing Sanitary Sewers for Rehabilitation Using Least-Cost Classifiers

Sewer rehabilitation is one control option for relieving wet-weather sanitary sewer overflows by reducing rainfall dependent inflow and infiltration. The prioritization of rehabilitation activities is based on the physical condition of a collection system, which is usually only partially known. Linear regression methods have been used to estimate the condition of the unobserved set of pipes based on relationships derived from the observed set of pipes. This method can provide unsatisfactory relationships between the sewer condition and measured independent variables, resulting in highly uncertain rehabilitation cost estimates. Discriminant analysis has been effective at weighing the costs of misclassifying pipes and deriving least-cost classification rules. Classification rules may be derived within a data-mining framework using evolutionary algorithms or with logistic regression methods. The estimated spatial distribution of deficient pipes may then be refined on an aggregated scale with a bootstrap estimate of classification error. The result is a screening and prioritization tool for providing cost estimates for rehabilitation and replacement activities. This method is demonstrated with an 8,919 pipe sanitary sewer system in Vallejo, Calif.

A linear feature extraction for multiclass classification problems based on class mean and covariance discriminant information

A parametric linear feature extraction method is proposed for multiclass classification. The skeleton of the proposed method consists of two types of schemes that are complementary to each other with regard to the discriminant information used. The approximate pairwise accuracy criterion (aPAC) and the common-mean feature extraction (CMFE) are chosen to exploit the discriminant information about class mean and about class covariance, respectively. Choosing aPAC rather than the linear discriminant analysis (LDA) can also resolve the problem of overemphasized large distances introduced by LDA, while maintaining other decent properties of LDA. To alleviate the suboptimum problem caused by a direct cascading of the two different types of schemes, there should be a mechanism for sorting and merging features based on their effectiveness. Usage of a sample-based classification error estimation for evaluation of effectiveness of features usually costs a lot of computational time. Therefore, we develop a fast spanning-tree-based parametric classification accuracy estimator as an intermediary for the aPAC and CMFE combination. The entire framework is parametric-based. This avoids paying a costly price in computation, which normally happens to the sample-based approach. Our experiments have shown that the proposed method can achieve a satisfactory performance on real data as well as simulated data.

AN INTEGRATED APPROACH TO AIR MASS CLASSIFICATION IN THE NORTH CENTRAL UNITED STATES

Both subjective and objective schemes for air mass classification offer advantages to climatological studies. An integrated approach is developed and applied to 850 mb level data in the north central United States. Each air mass type was initially delimited by a subjective analysis. Final limits were then determined with an objective technique that separates temperature curves into normal components based on interpretation from the subjective method. Continental air was most frequent in all seasons except summer. The methodology also allows estimation of classification error.

On the estimation of probability of error

This paper considers the problem of estimation of classification error in pattern recognition. A theorem is presented to obtain the changes in the eigenvalues and eigenvectors of matrices of the form S 2 −1 S 1, when there are changes of first order of smallness in the real symmetric matrices S i , i = 1, 2. Based on this theory a computational algorithm is developed for the estimation of classification error of Fisher classifier, using leaving groups out method.