Bayes Error Rate Research Articles

Application of Bayes Discriminant Functions to Classification of the Spatial Multivariate Gaussian Data

For given training sample, the problem of supervised classifying the multivariate stationary Gaussian random field (GRF) observations into one of two populations is considered. The populations are specified by different regression mean models and by common factorized covariance function. For completely specified populations the formula of Bayes error rate is derived. In the case of unknown regression parameters and covariance function parameters, the plug-in Bayes discriminant function based on ML estimators of parameters is used for spatial classification. The actual error rate and the asymptotic approximation of the expected error rate (AER) associated with plug-in Bayes discriminant function are derived. These results are multivariate generalization of previous one [1] for complete parametric uncertainty case. Numerical analysis of the derived formulas is implemented for the bivariate stationary GRF observations at locations belonging to the 2-dimensional lattice with unit spacing and exponential spatial correlation. Numerical comparison of two training label configurations (TLC) by the values of AER as optimality criterion is implemented.

Structure and Majority Classes in Decision Tree Learning

To provide good classification accuracy on unseen examples, a decision tree, learned by an algorithm such as ID3, must have sufficient structure and also identify the correct majority class in each of its leaves. If there are inadequacies in respect of either of these, the tree will have a percentage classification rate below that of the maximum possible for the domain, namely (100 - Bayes error rate). An error decomposition is introduced which enables the relative contributions of deficiencies in structure and in incorrect determination of majority class to be isolated and quantified. A sub-decomposition of majority class error permits separation of the sampling error at the leaves from the possible bias introduced by the attribute selection method of the induction algorithm. It is shown that sampling error can extend to 25% when there are more than two classes. Decompositions are obtained from experiments on several data sets. For ID3, the effect of selection bias is shown to vary from being statistically non-significant to being quite substantial, with the latter appearing to be associated with a simple underlying model.

On the Relationship Between Dependence Tree Classification Error and Bayes Error Rate

Wong and Poon [1] showed that Chow and Liu's tree dependence approximation can be derived by minimizing an upper bound of the Bayes error rate. Wong and Poon's result was obtained by expanding the conditional entropy H(w|X). We derive the correct expansion of H(w|X) and present its implication.

A Bayesian determination of threshold for identifying differentially expressed genes in microarray experiments

The original definitions of false discovery rate (FDR) and false non-discovery rate (FNR) can be understood as the frequentist risks of false rejections and false non-rejections, respectively, conditional on the unknown parameter, while the Bayesian posterior FDR and posterior FNR are conditioned on the data. From a Bayesian point of view, it seems natural to take into account the uncertainties in both the parameter and the data. In this spirit, we propose averaging out the frequentist risks of false rejections and false non-rejections with respect to some prior distribution of the parameters to obtain the average FDR (AFDR) and average FNR (AFNR), respectively. A linear combination of the AFDR and AFNR, called the average Bayes error rate (ABER), is considered as an overall risk. Some useful formulas for the AFDR, AFNR and ABER are developed for normal samples with hierarchical mixture priors. The idea of finding threshold values by minimizing the ABER or controlling the AFDR is illustrated using a gene expression data set. Simulation studies show that the proposed approaches are more powerful and robust than the widely used FDR method.

COMBINATION OF MULTIPLE CLASSIFIERS BY MINIMIZING THE UPPER BOUND OF BAYES ERROR RATE FOR UNCONSTRAINED HANDWRITTEN NUMERAL RECOGNITION

In order to raise a class discrimination power by the combination of multiple classifiers, the upper bound of Bayes error rate which is bounded by the conditional entropy of a class and decisions should be minimized. Based on the minimization of the upper bound of the Bayes error rate, Wang and Wong proposed only a tree dependence approximation scheme of a high-dimensional probability distribution composed of a class and patterns. This paper extends such a tree dependence approximation scheme to higher order dependency for improving the classification performance and thus optimally approximates the high-dimensional probability distribution with a product of low-dimensional distributions. And then, a new combination method by the proposed approximation scheme is presented and evaluated with classifiers recognizing unconstrained handwritten numerals.

Statistical behavior and consistency of classification methods based on convex risk minimization

We study how closely the optimal Bayes error rate can be approximately reached using a classification algorithm that computes a classifier by minimizing a convex upper bound of the classification error function. The measurement of closeness is characterized by the loss function used in the estimation. We show that such a classification scheme can be generally regarded as a (nonmaximum-likelihood) conditional in-class probability estimate, and we use this analysis to compare various convex loss functions that have appeared in the literature. Furthermore, the theoretical insight allows us to design good loss functions with desirable properties. Another aspect of our analysis is to demonstrate the consistency of certain classification methods using convex risk minimization. This study sheds light on the good performance of some recently proposed linear classification methods including boosting and support vector machines. It also shows their limitations and suggests possible improvements.

AXS and SOM: A new statistical approach for treating within-subject, time-varying, multivariate data collected using the AXS Test Battery

The Auditory Cross-Section (AXS) Test Battery [J. L. Lauter, Behav. Res. Methods Instrum. Comput. 32, 180–190 (2000)], described in presentations to ASA in 2002 and 2003, is designed to document dynamic relations linking the cortex, brainstem, and body periphery (whether physics, physiology, or behavior) on an individually-specific basis. Data collections using the battery typically employ a within-subject, time-varying, multivariate design, yet conventional group statistics do not provide satisfactory means of treating such data. We have recently developed an approach based on Kohonens (2001) Self-Organizing Maps (SOM) algorithm, which categorizes time-varying profiles across variables, either within- or between-subjects. The treatment entails three steps: (1) z-score transformation of all raw data; (2) employing the SOM to sort the time-varying profiles into groups; and (3) deriving an estimate of the bounds for the Bayes error rate. Our three-step procedure will be briefly described and illustrated with data from a recent study combining otoacoustic emissions, auditory brainstem responses, and cortical qEEG.

Bayes Error Rate Estimation Using Classifier Ensembles

The Bayes error rate gives a statistical lower bound on the error achievable for a given classification problem and the associated choice of features. By reliably estimating th is rate, one can assess the usefulness of the feature set that is being used for classification. Moreover, by comparing the accuracy achieved by a given classifier with the Bayes rate, one can quantify how effective that classifier is. Classical approaches for estimating or finding bounds for the Bayes error, in general, yield rather weak results for small sample sizes; unless the problem has some simple characteristics, such as Gaussian class-conditional likelihoods. This article shows how the outputs of a classifier ensemble can be used to provide reliable and easily obtainable estimates of the Bayes error with negligible extra computation. Three methods of varying sophistication are described. First, we present a framework that estimates the Bayes error when multiple classifiers, each providing an estimate of the a posteriori class probabilities, a recombined through averaging. Second, we bolster this approach by adding an information theoretic measure of output correlation to the estimate. Finally, we discuss a more general method that just looks at the class labels indicated by ensem ble members and provides error estimates based on the disagreements among classifiers. The methods are illustrated for artificial data, a difficult four-class problem involving underwater acoustic data, and two problems from the Problem benchmarks. For data sets with known Bayes error, the combiner-based methods introduced in this article outperform existing methods. The estimates obtained by the proposed methods also seem quite reliable for the real-life data sets for which the true Bayes rates are unknown.

Analysis of Pattern Discovery in Sequences Using a Bayes Error Framework

In this paper we investigate the general problem of discovering recurrent patterns that are embedded in categorical sequences. An important real-world problem of this nature is motif discovery in DNA sequences. There are a number of fundamental aspects of this data mining problem that can make discovery “easy” or “hard”—we characterize the difficulty of this problem using an analysis based on the Bayes error rate under a Markov assumption. The Bayes error framework demonstrates why certain patterns are much harder to discover than others. It also explains the role of different parameters such as pattern length and pattern frequency in sequential discovery. We demonstrate how the Bayes error can be used to calibrate existing discovery algorithms, providing a lower bound on achievable performance. We discuss a number of fundamental issues that characterize sequential pattern discovery in this context, present a variety of empirical results to complement and verify the theoretical analysis, and apply our methodology to real-world motif-discovery problems in computational biology.

A Bayesian framework for the combination of classifier outputs

We explore a Bayesian framework for constructing combinations of classifier outputs, as a means to improving overall classification results. We propose a sequential Bayesian framework to estimate the posterior probability of being in a certain class given multiple classifiers. This framework, which employs meta-Gaussian modelling but makes no assumptions about the distribution of classifier outputs, allows us to capture nonlinear dependencies between the combined classifiers and individuals. An important property of our method is that it produces a combined classifier that dominates the individuals upon which it is based in terms of Bayes risk, error rate, and receiver operating characteristic (ROC) curve. To illustrate the method, we show empirical results from the combination of credit scores generated from four different scoring models.

Quadratic Discriminant Analysis of Spatially Correlated Data

The problem of classification of the realisation of the stationary univariate Gaussian random field into one of two populations with different means and different factorised covariance matrices is considered. In such a case optimal classification rule in the sense of minimum probability of misclassification is associated with non-linear (quadratic) discriminant function. Unknown means and the covariance matrices of the feature vector components are estimated from spatially correlated training samples using the maximum likelihood approach and assuming spatial correlations to be known. Explicit formula of Bayes error rate and the first-order asymptotic expansion of the expected error rate associated with quadratic plug-in discriminant function are presented. A set of numerical calculations for the spherical spatial correlation function is performed and two different spatial sampling designs are compared.

Presupervised and post-supervised prototype classifier design

We extend the nearest prototype classifier to a generalized nearest prototype classifier (GNPC). The GNPC uses "soft" labeling of the prototypes in the classes, thereby encompassing a variety of classifiers. Based on how the prototypes are found we distinguish between presupervised and postsupervised GNPC designs. We derive the conditions for optimality (relative to the standard Bayes error rate) of two designs where prototypes represent: 1) the components of class-conditional mixture densities (presupervised design) or 2) the components of the unconditional mixture density (postsupervised design). An artificial data set and the "satimage" data set from the database ELENA are used to experimentally study the two approaches. A radial basis function (RBF) network is used as a representative of each GNPC type. Neither the theoretical nor the experimental results indicate clear reasons to prefer one of the approaches. The postsupervised GNPC design tends to be more robust and less accurate than the presupervised one.

ASSESSING ERROR RATE ESTIMATORS: THE LEAVE‐ONE‐OUT METHOD RECONSIDERED

summaryMany comparative studies of the estimators of error rates of supervised classification rules are based on inappropriate criteria. In particular, although they fix the Bayes error rate, their summary statistics aggregate a range of true error rates. This means that their conclusions about the performance of classification rules cannot be trusted. This paper discusses the general issues involved, and then focuses attention specifically on the leave‐one‐out estimator. The estimator is investigated in a simulation study, both in absolute terms and in comparison with a popular bootstrap estimator. An improvement to the leave‐one‐out estimator is suggested, but the bootstrap estimator appears to maintain superiority even when the criteria are adjusted.

Admissible stochastic complexity models for classification problems

In this paper we investigate the application of stochastic complexity theory to classification problems. In particular, we define the notion of admissible models as a function of problem complexity, the number of data pointsN, and prior belief. This allows us to derive general bounds relating classifier complexity with data-dependent parameters such as sample size, class entropy and the optimal Bayes error rate. We discuss the application of these results to a variety of problems, including decision tree classifiers, Markov models for image segmentation, and feedforward multilayer neural network classifiers.

Comments on approximating discrete probability distributions with dependence trees

C.K. Chow and C.N. Liu (1968) introduced the notion of three dependence to approximate a kth-order probability distribution. More recently, A.K.C. Wong and C.C. Wang (1977) proposed a different product approximation. The present authors show that the tree dependence approximation suggested by Chow and Liu can be derived by minimizing an upper bound of the Bayes error rate under certain assumptions. They also show that the method proposed by Wong and Wang does not necessarily lead to fewer misclassifications, because it is a special case of such a minimization procedure.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

The asymptotic distribution of an estimator of the Bayes error rate

The asymptotic distribution of a class of estimators of the Bayes error rate in pattern recognition problems is derived. The class consists of estimators of the risk-averaging type, applied to parametric models such as the logistic function.

New error bounds with the nearest neighbor rule (Corresp.)

A distribution-free lower bound on the Bayes error rate is formulated in terms of the asymptotic error rate of the nearest neighbor rule with a reject option. Next, a closed form expression for an upper bound of the <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">k</tex> th nearest neighbor error rate in terms of the Bayes rate is established. These results are discussed in the framework of recent works on nonparametric estimation of the Bayes error rate.

Considerations of sample and feature size

In many practical pattern-classification problems the underlying probability distributions are not completely known. Consequently, the classification logic must be determined on the basis of vector samples gathered for each class. Although it is common knowledge that the error rate on the design set is a biased estimate of the true error rate of the classifier, the amount of bias as a function of sample size per class and feature size has been an open question. In this paper, the design-set error rate for a two-class problem with multivariate normal distributions is derived as a function of the sample size per class <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">(N)</tex> and dimensionality <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">(L)</tex> . The design-set error rate is compared to both the corresponding Bayes error rate and the test-set error rate. It is demonstrated that the design-set error rate is an extremely biased estimate of either the Bayes or test-set error rate if the ratio of samples per class to dimensions <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">(N/L)</tex> is less than three. Also the variance of the design-set error rate is approximated by a function that is bounded by <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1/8N</tex> .