Linear Classification Rule Research Articles

SURE estimates for high dimensional classification

AbstractIn the present paper, we consider the high dimensional classification problem, which has become much important in many modern statistical studies and applications. We develop new classifiers based on Fisher's linear classification rule and empirical Bayes. In particular, we propose to employ the Stein's unbiased risk estimate (SURE) to estimate the sparse or non‐sparse mean difference, which could be plugged into the linear classification rules. Using simulation studies under a variety of settings, we demonstrate that our classifiers perform well especially when the features are non‐sparse. We also illustrate the use of the new proposal to classification problems in some real data examples.

Linear components of quadratic classifiers

We obtain a decomposition of any quadratic classifier in terms of products of hyperplanes. These hyperplanes can be viewed as relevant linear components of the quadratic rule (with respect to the underlying classification problem). As an application, we introduce the associated multidirectional classifier; a piecewise linear classification rule induced by the approximating products. Such a classifier is useful to determine linear combinations of the predictor variables with ability to discriminate. We also show that this classifier can be used as a tool to reduce the dimension of the data and helps identify the most important variables to classify new elements. Finally, we illustrate with a real data set the use of these linear components to construct oblique classification trees.

Asymptotic Optimality of Sparse Linear Discriminant Analysis with Arbitrary Number of Classes

AbstractMany sparse linear discriminant analysis (LDA) methods have been proposed to overcome the major problems of the classic LDA in high‐dimensional settings. However, the asymptotic optimality results are limited to the case with only two classes. When there are more than two classes, the classification boundary is complicated and no explicit formulas for the classification errors exist. We consider the asymptotic optimality in the high‐dimensional settings for a large family of linear classification rules with arbitrary number of classes. Our main theorem provides easy‐to‐check criteria for the asymptotic optimality of a general classification rule in this family as dimensionality and sample size both go to infinity and the number of classes is arbitrary. We establish the corresponding convergence rates. The general theory is applied to the classic LDA and the extensions of two recently proposed sparse LDA methods to obtain the asymptotic optimality.

Block models and personalized PageRank

Methods for ranking the importance of nodes in a network have a rich history in machine learning and across domains that analyze structured data. Recent work has evaluated these methods through the "seed set expansion problem": given a subset [Formula: see text] of nodes from a community of interest in an underlying graph, can we reliably identify the rest of the community? We start from the observation that the most widely used techniques for this problem, personalized PageRank and heat kernel methods, operate in the space of "landing probabilities" of a random walk rooted at the seed set, ranking nodes according to weighted sums of landing probabilities of different length walks. Both schemes, however, lack an a priori relationship to the seed set objective. In this work, we develop a principled framework for evaluating ranking methods by studying seed set expansion applied to the stochastic block model. We derive the optimal gradient for separating the landing probabilities of two classes in a stochastic block model and find, surprisingly, that under reasonable assumptions the gradient is asymptotically equivalent to personalized PageRank for a specific choice of the PageRank parameter [Formula: see text] that depends on the block model parameters. This connection provides a formal motivation for the success of personalized PageRank in seed set expansion and node ranking generally. We use this connection to propose more advanced techniques incorporating higher moments of landing probabilities; our advanced methods exhibit greatly improved performance, despite being simple linear classification rules, and are even competitive with belief propagation.

SUPERVISE DIFFERENCE LINEAR CLASSIFICATION TECHNIQUE FOR TWO GROUPS PROBLEM

Classification techniques can be categorized as parametric and nonparametric. The former strictly depends on assumptions, and is a supervised linear classification procedure whereas the later does not rely on any assumptions and is an unsupervised linear classification technique. This paper considers supervised linear classification techniques in which the course of dimensionality is negated. In this regard, the sample size is greater than the sample dimension. We investigated the performance of the Fisher, and the difference linear classification techniques based on the classification performance and the acceptance or rejection of the null or alternative hypothesis in which the mean of the optimal probability of correct classification is used as the hypothesized mean, and the computed mean probability is derived from the mean probability of correct classification based on 1000 replications. The robustness of these techniques is determined by the sensitivity of these methods to contaminated data set. Relying on the performance analysis, we further investigated the acceptance or rejection of the null or alternate hypothesis based on the proportion of contamination using the Hoteling test statistics. The comparative analysis indicates that the difference linear classification rule outperformed the conventional Fisher’s technique. The analysis revealed that using 95% level of significant, the null hypothesis is rejected.

Best Linear Classification Rule for Multivariate Interval Screened Data

In classification analysis, the target variable is often in practice defined by an underlying multivariate interval screening scheme. This engenders the problem of properly characterizing the screened populations as well as that of obtaining a classification procedure. Such problems paved the way for the development of yet another linear classification procedure and the incorporation of a class of skew-elliptical distributions for describing evolutions in the populations. To render the linear procedure effective, this article considers derivation and properties of the classification procedure as well as efficient estimation. The procedure is illustrated in applications to real and simulation data.

Asymptotic expansion and estimation of EPMC for linear classification rules in high dimension

The problem of classifying a new observation vector into one of the two known groups distributed as multivariate normal with common covariance matrix is considered. In this paper, we handle the situation that the dimension, p, of the observation vectors is less than the total number, N, of observation vectors from the two groups, but both p and N tend to infinity with the same order. Since the inverse of the sample covariance matrix is close to an ill condition in this situation, it may be better to replace it with the inverse of the ridge-type estimator of the covariance matrix in the linear discriminant analysis (LDA). The resulting rule is called the ridge-type linear discriminant analysis (RLDA). The second-order expansion of the expected probability of misclassification (EPMC) for RLDA is derived, and the second-order unbiased estimator of EMPC is given. These results not only provide the corresponding conclusions for LDA, but also clarify the condition that RLDA improves on LDA in terms of EPMC. Finally, the performances of the second-order approximation and the unbiased estimator are investigated by simulation.

The Tangent Classifier

Given a classifier, we describe a general method to construct a simple linear classification rule. This rule, called the tangent classifier, is obtained by computing the tangent hyperplane to the separation boundary of the groups (generated by the initial classifier) at a certain point. When applied to a quadratic region, the tangent classifier has a neat closed-form expression. We discuss various examples and the application of this new linear classifier in two situations under which standard rules may fail: when there is a fraction of outliers in the training sample and when the dimension of the data is large in comparison with the sample size.

Two-group classification with high-dimensional correlated data: A factor model approach

A class of linear classification rules, specifically designed for high-dimensional problems, is proposed. The new rules are based on Gaussian factor models and are able to incorporate successfully the information contained in the sample correlations. Asymptotic results, that allow the number of variables to grow faster than the number of observations, demonstrate that the worst possible expected error rate of the proposed rules converges to the error of the optimal Bayes rule when the postulated model is true, and to a slightly larger constant when this model is a reasonable approximation to the data generating process. Numerical comparisons suggest that, when combined with appropriate variable selection strategies, rules derived from one-factor models perform comparably, or better, than the most successful extant alternatives under the conditions they were designed for. The proposed methods are implemented as an R package named HiDimDA, available from the CRAN repository.

RISK MINIMIZATION FOR TIME SERIES BINARY CHOICE WITH VARIABLE SELECTION

This paper considers the problem of predicting binary choices by selecting from a possibly large set of candidate explanatory variables, which can include both exogenous variables and lagged dependent variables. We consider risk minimization with the risk function being the predictive classification error. We study the convergence rates of empirical risk minimization in both the frequentist and Bayesian approaches. The Bayesian treatment uses a Gibbs posterior constructed directly from the empirical risk instead of using the usual likelihood-based posterior. Therefore these approaches do not require a correctly specified probability model. We show that the proposed methods have near optimal performance relative to a class of linear classification rules with selected variables. Such results in classification are obtained in a framework of dependent data with strong mixing.

35 years of studies on business failure: an overview of the classic statistical methodologies and their related problems

Over the last 35 years, business failure prediction has become a major research domain within corporate finance. Numerous corporate failure prediction models have been developed, based on various modelling techniques. The most popular are the classic cross-sectional statistical methods, which have resulted in various ‘single-period’ or static models, especially multivariate discriminant models and logit models. To date, there has been no clear overview and discussion of the application of classic statistical methods to business failure prediction. Therefore, this paper extensively elaborates on the application of (1) univariate analysis, (2) risk index models, (3) multivariate discriminant analysis, and (4) conditional probability models in corporate failure prediction. In addition, because there is no clear and comprehensive analysis in the existing literature of the diverse problems related to the application of these methods to the topic of corporate failure prediction, this paper brings together all problem issues and enlarges upon each of them. It discusses all problems related to: (1) the classical paradigm (i.e. the arbitrary definition of failure, non-stationarity and data instability, sampling selectivity, and the choice of the optimisation criteria); (2) the neglect of the time dimension of failure; and (3) the application focus in failure prediction modelling. Further, the paper elaborates on a number of other problems related to the use of a linear classification rule, the use of annual account information, and neglect of the multidimensional nature of failure. This paper contributes towards a thorough understanding of the features of the classic statistical business failure prediction models and their related problems.

Bolstered error estimation

We propose a general method for error estimation that displays low variance and generally low bias as well. This method is based on “bolstering” the original empirical distribution of the data. It has a direct geometric interpretation and can be easily applied to any classification rule and any number of classes. This method can be used to improve the performance of any error-counting estimation method, such as resubstitution and all cross-validation estimators, particularly in small-sample settings. We point out some similarities shared by our method with a previously proposed technique, known as smoothed error estimation. In some important cases, such as a linear classification rule with a Gaussian bolstering kernel, the integrals in the bolstered error estimate can be computed exactly. In the general case, the bolstered error estimate may be computed by Monte-Carlo sampling; however, our experiments show that a very small number of Monte-Carlo samples is needed. This results in a fast error estimator, which is in contrast to other resampling techniques, such as the bootstrap. We provide an extensive simulation study comparing the proposed method with resubstitution, cross-validation, and bootstrap error estimation, for three popular classification rules (linear discriminant analysis, k-nearest-neighbor, and decision trees), using several sample sizes, from small to moderate. The results indicate the proposed method vastly improves on resubstitution and cross-validation, especially for small samples, in terms of bias and variance. In that respect, it is competitive with, and in many occasions superior to, bootstrap error estimation, while being tens to hundreds of times faster. We provide a companion web site, which contains: (1) the complete set of tables and plots regarding the simulation study, and (2) C source code used to implement the bolstered error estimators proposed in this paper, as part of a larger library for classification and error estimation, with full documentation and examples. The companion web site can be accessed at the URL http://ee.tamu.edu/~edward/bolster.

Automated location of dysplastic fields in colorectal histology using image texture analysis.

Automation in histopathology is an attractive concept and recent advances in the application of computerized expert systems and machine vision have made automated image analysis of histological images possible. Systems capable of complete automation not only require the ability to segment tissue features and grade histological abnormalities, but, must also be capable of locating diagnostically useful areas from within complex histological scenes. This is the first stage of the diagnostic process. The object of this study was to develop criteria for the automatic identification of focal areas of colorectal dysplasia from a background of histologically normal tissue. Fields of view representing normal colorectal mucosa (n = 120) and dysplastic mucosa (n = 120) were digitally captured and subjected to image texture analysis. Two features were selected as being the most important in the discrimination of normal and adenomatous colorectal mucosa. The first was a feature of the co-occurrence matrix and the second was the number of low optical density pixels in the image. A linear classification rule defined using these two features was capable of correctly classifying 86 per cent of a series of training images into their correct groups. In addition, large histological scenes were digitally captured, split into their component images, analysed according to texture, and classified as normal or abnormal using the previously defined classification rule. Maps of the histological scenes were constructed and in most cases, dysplastic colorectal mucosa was correctly identified on the basis of image texture: 83 per cent of test images were correctly classified. This study demonstrates that abnormalities in low-power tissue morphology can be identified using quantitative image analysis. The identification of diagnostically useful fields advances the potential of automated systems in histopathology: these regions could than be scrutinized at high power using knowledge-guided image segmentation for disease grading. Systems of this kind have the potential to provide objectivity, unbiased sampling, and valuable diagnostic decision support.

AUTOMATED LOCATION OF DYSPLASTIC FIELDS IN COLORECTAL HISTOLOGY USING IMAGE TEXTURE ANALYSIS

Automation in histopathology is an attractive concept and recent advances in the application of computerized expert systems and machine vision have made automated image analysis of histological images possible. Systems capable of complete automation not only require the ability to segment tissue features and grade histological abnormalities, but must also be capable of locating diagnostically useful areas from within complex histological scenes. This is the first stage of the diagnostic process. The object of this study was to develop criteria for the automatic identification of focal areas of colorectal dysplasia from a background of histologically normal tissue. Fields of view representing normal colorectal mucosa (n=120) and dysplastic mucosa (n=120) were digitally captured and subjected to image texture analysis. Two features were selected as being the most important in the discrimination of normal and adenomatous colorectal mucosa. The first was a feature of the co-occurrence matrix and the second was the number of low optical density pixels in the image. A linear classification rule defined using these two features was capable of correctly classifying 86 per cent of a series of training images into their correct groups. In addition, large histological scenes were digitally captured, split into their component images, analysed according to texture, and classified as normal or abnormal using the previously defined classification rule. Maps of the histological scenes were constructed and in most cases, dysplastic colorectal mucosa was correctly identified on the basis of image texture: 83 per cent of test images were correctly classified. This study demonstrates that abnormalities in low-power tissue morphology can be identified using quantitative image analysis. The identification of diagnostically useful fields advances the potential of automated systems in histopathology: these regions could than be scrutinized at high power using knowledge-guided image segmentation for disease grading. Systems of this kind have the potential to provide objectivity, unbiased sampling, and valuable diagnostic decision support. © 1997 John Wiley & Sons, Ltd.

Linear Versus Quadratic Multivariate Classification.

A linear classification rule (used with equal covariance matrices) was contrasted with a quadratic rule (used with unequal covariance matrices) for accuracy of internal and external classification. The comparisons were made for seven situations which resulted from combining conditions (equal and unequal covariance matrices, and two and three criterion groups) for different sets of real data. For the internal analysis the quadratic rule was superior to the linear rule in all seven situations. For the external analysis the linear rule was nearly as good as or superior to the quadratic rule in all seven situations.

Bound on the Classification Error for Discriminating Between Multivariate Populations with Specified Means and Covariance Matrices

Let $\mathscr{F}_1, \mathscr{F}_2$ be two families of $p$-variate distribution functions with specified means $\mathbf{\mu}_i (i = 1,2)$ and nonsingular covariance matrices $\Sigma_i$, and let $\pi_i$ be the prior probability assigned to $\mathscr{F}_i$ for $i = 1, 2$. The objective is to discriminate whether an observation $\mathbf{x}$ is from a distribution $F_1 \in \mathscr{F}_1$ or $F_2 \in \mathscr{F}_2$. Given a pair $F = (F_1, F_2)$ the error probability for classification rule $\phi$ is denoted by $e(\phi, F)$. In this paper the values of $\sup_F \inf_\phi e(\phi, F)$ and $\inf_\phi \sup_F e(\phi, F)$ are found and conditions for the existence of a saddle point of $e(\phi, F)$ are given. Also a saddle point is found when it exists. When $\phi$ is restricted to linear classification rules the same problems are considered. The mathematical programming method for finding a saddle point is also outlined.

Optimum Classification Rules for Classification into Two Multivariate Normal Populations

The problem of classifying an observation into one of two multivariate normal populations with the same covariance matrix has been thoroughly discussed by Anderson [1] when the populations are completely known. Anderson and Bahadur [2] treated the case of different and known covariance matrices and obtained the minimal complete class restricting to linear classification rules. Wald [11], Anderson [1], and Rao [7] suggested some heuristic classification rules based on the sample estimates of the unknown parameters of the two normal distributions having the same covariance matrix. One of these heuristic rules is the maximum likelihood rule (ML rule) which classifies the observation into the population $\Pi_i$ if the maximum likelihood (likelihood maximized under the variation of the unknown parameters) obtained under the assumption that the observation to be classified comes from $\Pi_i$ is greater than the corresponding maximum likelihood assuming that the observation comes from $\Pi_j(i \neq j; i, j = 1, 2)$. Sitgreaves [8], [9], and John [5] obtained the explicit forms of the distributions of the classification statistics proposed by Anderson and Wald. Many other papers in this line are included in the book cited in the reference [9]. Ellison [4] derived a class of admissible rules which includes the ML rule for the problem of classification into more than two normal populations with different and known covariance matrices. Cacoullos [3] obtained an invariant Bayes rule and an admissible minimax rule for the problem of selecting one out of a finite number of completely specified normal populations which is closest to a given normal population whose covariance matrix is unknown. It will be shown in this paper that the ML rule is an unbiased admissible minimax rule when the common covariance matrix of the two normal populations is known; and, when the common covariance matrix is unknown, that the corresponding ML rule is unbiased and is an admissible minimax rule in an invariant class. The loss function in each problem is assumed to be a function (satisfying some mild restrictions) of the Mahalanobis distance between the two populations.