Discriminant Analysis For High-dimensional Data Research Articles

A modified linear discriminant analysis for high-dimensional data

We deal with the problem of classifying a new observation vector into one of two known multivariate normal populations. Linear discriminant analysis (LDA) is now widely available. However, for high-dimensional data classification problem, due to the small number of samples and the large number of variables, classical LDA has poor performance corresponding to the singularity and instability of the sample covariance matrix. Recently, Xu et al. suggested modified linear discriminant analysis (MLDA). This method is based on the shrink type estimator of the covariance matrix derived by Ledoit and Wolf. This estimator was proposed under the asymptotic framework ${\rm A}_0:n=O(p)$ and $p=O(n)$ when $p\to\infty$. In this paper, we propose a shrink type estimator under more flexible high-dimensional framework. Using this estimator, we define the new MLDA. Through the numerical simulation, the expected correct classification rate of our MLDA is larger than the ones of other discrimination methods when $p>n$. In addition, we consider the limiting value of the expected probability of misclassification (EPMC) under some assumptions.

A model selection criterion for discriminant analysis of high-dimensional data with fewer observations

This paper is concerned with the problem of selecting variables in two-group discriminant analysis for high-dimensional data with fewer observations than the dimension. We consider a selection criterion based on approximately unbiased for AIC type of risk. When the dimension is large compared to the sample size, AIC type of risk cannot be defined. We propose AIC by replacing maximum likelihood estimator with ridge-type estimator. This idea follows Srivastava and Kubokawa (2008). It has been further extended by Yamamura et al. (2010). Simulation revealed that the proposed AIC performs well.

HDclassif: AnRPackage for Model-Based Clustering and Discriminant Analysis of High-Dimensional Data

This paper presents the R package HDclassif which is devoted to the clustering and the discriminant analysis of high-dimensional data. The classification methods proposed in the package result from a new parametrization of the Gaussian mixture model which combines the idea of dimension reduction and model constraints on the covariance matrices. The supervised classification method using this parametrization has been called High Dimensional Discriminant Analysis (HDDA). In a similar manner, the associated clustering method has been called High Dimensional Data Clustering (HDDC) and uses the Expectation-Maximization (EM) algorithm for inference. In order to correctly fit the data, both methods estimate the specific subspace and the intrinsic dimension of the groups. Due to the constraints on the covariance matrices, the number of parameters to estimate is significantly lower than other model-based methods and this allows the methods to be stable and efficient in high-dimensional spaces. Experiments on artificial and real datasets show that HDDC and HDDA perform better than existing classical methods on high-dimensional datasets, even with small datasets. HDclassif is a free software and distributed under the GNU General Public License, as part of the R software project.

A Direct Estimation Approach to Sparse Linear Discriminant Analysis

This article considers sparse linear discriminant analysis of high-dimensional data. In contrast to the existing methods which are based on separate estimation of the precision matrix Ω and the difference δ of the mean vectors, we introduce a simple and effective classifier by estimating the product Ωδ directly through constrained ℓ1 minimization. The estimator can be implemented efficiently using linear programming and the resulting classifier is called the linear programming discriminant (LPD) rule.The LPD rule is shown to have desirable theoretical and numerical properties. It exploits the approximate sparsity of Ωδ and as a consequence allows cases where it can still perform well even when Ω and/or δ cannot be estimated consistently. Asymptotic properties of the LPD rule are investigated and consistency and rate of convergence results are given. The LPD classifier has superior finite sample performance and significant computational advantages over the existing methods that require separate estimation of Ω and δ. The LPD rule is also applied to analyze real datasets from lung cancer and leukemia studies. The classifier performs favorably in comparison to existing methods.

Multicategory vertex discriminant analysis for high-dimensional data

In response to the challenges of data mining, discriminant analysis continues to evolve as a vital branch of statistics. Our recently introduced method of vertex discriminant analysis (VDA) is ideally suited to handle multiple categories and an excess of predictors over training cases. The current paper explores an elaboration of VDA that conducts classification and variable selection simultaneously. Adding lasso ($\ell_1$-norm) and Euclidean penalties to the VDA loss function eliminates unnecessary predictors. Lasso penalties apply to each predictor coefficient separately; Euclidean penalties group the collective coefficients of a single predictor. With these penalties in place, cyclic coordinate descent accelerates estimation of all coefficients. Our tests on simulated and benchmark real data demonstrate the virtues of penalized VDA in model building and prediction in high-dimensional settings.

Discriminant analysis of high-dimensional data over limited samples

Discriminant analysis of high-dimensional data over limited samples

A modification of kernel discriminant analysis for high-dimensional data—with application to face recognition

Kernel discriminant analysis (KDA) is an effective statistical method for dimensionality reduction and feature extraction. However, traditional KDA methods suffer from the small sample size problem. Moreover, they endure the Fisher criterion that is nonoptimal with respect to classification rate. This paper presents a variant of KDA that deals with both of the shortcomings in an efficient and cost effective manner. The key to the approach is to use simultaneous diagonalization technique for optimization and meanwhile utilize a modified Fisher criterion that it is more closely related to classification error. Extensive experiments on face recognition task show that the proposed method is an effective nonlinear feature extractor.

Feature Extraction and Uncorrelated Discriminant Analysis for High-Dimensional Data

High-dimensional data and the small sample size problem occur in many modern pattern classification applications such as face recognition and gene expression data analysis. To deal with such data, one important step is dimensionality reduction. Principal component analysis (PCA) and between-group analysis (BGA) are two commonly used methods, and various extensions of these two methods exist. The principle of these two approaches comes from their best approximation property. From a pattern recognition perspective, we show that PCA, which is based on the total scatter matrix, preserves linear separability, and BGA, which is based on between-class scatter matrix, retains the distance between class centroids. Moreover, we propose an automatic nonparameter uncorrelated discriminant analysis (UDA) algorithm based on the maximum margin criterion (MMC). The extracted features via UDA are statistically uncorrelated. UDA combines rank-preserving dimensionality reduction and constraint discriminant analysis and also serves as an effective solution for the small-sample-size problem. Experiments with face images and gene expression data sets are conducted to evaluate UDA in terms of classification accuracy and robustness.

Antedependence models in the analysis of multi-group high-dimensional data

Antedependence modelling has previously been shown to be useful for twogroup discriminant analysis of high-dimensional data. In this paper, the theory of such models is extended to multi-group discriminant analysis and to canonical variate analysis for data display. The application of antedependence models of orders 1, 2 and 3 to spectroscopic analyses of rice samples is described, and the results are compared with those from standard methods based on principal component scores calculated from the data.

Discriminant analysis of high-dimensional data: a comparison of principal components analysis and partial least squares data reduction methods

Partial least squares (PLS) methods are presented as valuable alternatives to principal components analysis (PCA) for compressing high-dimensional data before performing linear discriminant analysis (LDA). It is shown that using PLS, considerable improvement in class separation and thus discriminant ability can be obtained. In general, fewer of the compressed dimensions are required to give the same level of prediction successes, and for some data sets, PLS methods yield higher prediction success rates than those obtainable using PCA scores. Results are presented for two experimental data sets, comprising mid-infrared spectra of edible oils and plant seeds. The potential dangers of PLS methods are also demonstrated, in particular its ability to introduce apparent groupings into data where there is no inherent class structure.