Null Space Linear Discriminant Analysis Research Articles

On the Equivalence of Linear Discriminant Analysis and Least Squares Regression.

Studying the relationship between linear discriminant analysis (LDA) and least squares regression (LSR) is of great theoretical and practical significance. It is well-known that the two-class LDA is equivalent to an LSR problem, and directly casting multiclass LDA as an LSR problem, however, becomes more challenging. Recent study reveals that the equivalence between multiclass LDA and LSR can be established based on a special class indicator matrix, but under a mild condition which may not hold under the scenarios with low-dimensional or oversampled data. In this article, we show that the equivalence between multiclass LDA and LSR can be established based on arbitrary linearly independent class indicator vectors and without any condition. In addition, we show that LDA is also equivalent to a constrained LSR based on the data-dependent indicator vectors. It can be concluded that under exactly the same mild condition, such two regressions are both equivalent to the null space LDA method. Illuminated by the equivalence of LDA and LSR, we propose a direct LDA classifier to replace the conventional framework of LDA plus extra classifier. Extensive experiments well validate the above theoretic analysis.

Representative null space LDA for discriminative dimensionality reduction

Null space Linear Discriminant Analysis (NLDA) was proposed twenty years ago to overcome the singularity problem of LDA in practical applications. With two decades of technique development, many Discriminative Dimensionality Reduction (DDR) methods that outperform NLDA have been proposed. This paper provides new insight into NLDA and illustrates that NLDA is much more powerful after solving its inherent problem. The main problem of NLDA is the intrinsic overfitting problem. An ideal NLDA model is proposed to analyze its overfitting problem. Based on the ideal NLDA model, a more reasonable Representative NLDA (RNLDA) method is proposed to prevent overfitting. Two simple but efficient RNLDA algorithms are proposed to implement the RNLDA method with a theoretical proof. This study theoretically analyzed and indicated that applying the classical but simple hold-out pretraining method can automatically set the only parameter to achieve high performance. Extensive experiments with eight databases demonstrate the superior performance of the RNLDA method over state-of-the-art DDR methods.

User Identification from Gait Analysis Using Multi-Modal Sensors in Smart Insole.

Recent studies indicate that individuals can be identified by their gait pattern. A number of sensors including vision, acceleration, and pressure have been used to capture humans’ gait patterns, and a number of methods have been developed to recognize individuals from their gait pattern data. This study proposes a novel method of identifying individuals using null-space linear discriminant analysis on humans’ gait pattern data. The gait pattern data consists of time series pressure and acceleration data measured from multi-modal sensors in a smart insole used while walking. We compare the identification accuracies from three sensing modalities, which are acceleration, pressure, and both in combination. Experimental results show that the proposed multi-modal features identify 14 participants with high accuracy over 95% from their gait pattern data of walking.

LDA Extension via Oblique Projection

The classical linear discriminant analysis (LDA) was previously modified by orthogonal projection into null space LDA (N_LDA) and direct LDA (D_LDA) for solving small sample size (SSS) problem. In this paper, the author proposes an extension of LDA by oblique projection, wherein N_LDA and D_LDA are included as special cases, to reduce discriminative information loss resulted from single N_LDA or D_LDA. The effectiveness of the proposed algorithm is tested by image forensics and face recognition.

Face recognition using discriminative locality preserving vectors

We proposed an effective face recognition method based on the discriminative locality preserving vectors method (DLPV). Using the analysis of eigenspectrum modeling of locality preserving projections, we selected the reliable face variation subspace of LPP to construct the locality preserving vectors to characterize the data set. The discriminative locality preserving vectors (DLPV) method is based on the discriminant analysis on the locality preserving vectors. Furthermore, the theoretical analysis showed that the DLPV is viewed as a generalized discriminative common vector, null space linear discriminant analysis and null space discriminant locality preserving projections, which gave the intuitive motivation of our method. Extensive experimental results obtained on four well-known face databases (ORL, Yale, Extended Yale B and CMU PIE) demonstrated the effectiveness of the proposed DLPV method.

Face recognition using nonparametric-weighted Fisherfaces

Abstract This study presents an appearance-based face recognition scheme called the nonparametric-weighted Fisherfaces (NW-Fisherfaces). Pixels in a facial image are considered as coordinates in a high-dimensional space and are transformed into a face subspace for analysis by using nonparametric-weighted feature extraction (NWFE). According to previous studies of hyperspectral image classification, NWFE is a powerful tool for extracting hyperspectral image features. The Fisherfaces method maximizes the ratio of between-class scatter to that of within-class scatter. In this study, the proposed NW-Fisherfaces weighted the between-class scatter to emphasize the boundary structure of the transformed face subspace and, therefore, enhances the separability for different persons' face. The proposed NW-Fisherfaces was compared with Orthogonal Laplacianfaces, Eigenfaces, Fisherfaces, direct linear discriminant analysis, and null space linear discriminant analysis methods for tests on five facial databases. Experimental results showed that the proposed approach outperforms other feature extraction methods for most databases.

Orthogonal vs. uncorrelated least squares discriminant analysis for feature extraction

In this paper, a new discriminant analysis for feature extraction is derived from the perspective of least squares regression. To obtain great discriminative power between classes, all the data points in each class are expected to be regressed to a single vector, and the basic task is to find a transformation matrix such that the squared regression error is minimized. To this end, two least squares discriminant analysis methods are developed under the orthogonal or the uncorrelated constraint. We show that the orthogonal least squares discriminant analysis is an extension to the null space linear discriminant analysis, and the uncorrelated least squares discriminant analysis is exactly equivalent to the traditional linear discriminant analysis. Comparative experiments show that the orthogonal one is more preferable for real world applications.

From NLDA to LDA/GSVD: a modified NLDA algorithm

Linear discriminant analysis (LDA) often encounters small sample size (SSS) problem for high-dimensional data. Null space linear discriminant analysis (NLDA) and linear discriminant analysis based on generalized singular value decomposition (LDA/GSVD) are two popular methods that can solve SSS problem of LDA. In this paper, we present the relation between NLDA and LDA/GSVD under a condition and at the same time propose a modified NLDA (MNLDA) algorithm which has the same discriminating power as LDA/GSVD and is more efficient. In addition, we compare the discriminating capability of NLDA and MNLDA and present our interpretation about this. Experimental results on ORL, FERET, Yale face databases, and the PolyU FKP database support our viewpoints.

Generalized Discriminant Analysis: A Matrix Exponential Approach

Linear discriminant analysis (LDA) is well known as a powerful tool for discriminant analysis. In the case of a small training data set, however, it cannot directly be applied to high-dimensional data. This case is the so-called small-sample-size or undersampled problem. In this paper, we propose an exponential discriminant analysis (EDA) technique to overcome the undersampled problem. The advantages of EDA are that, compared with principal component analysis (PCA) + LDA, the EDA method can extract the most discriminant information that was contained in the null space of a within-class scatter matrix, and compared with another LDA extension, i.e., null-space LDA (NLDA), the discriminant information that was contained in the non-null space of the within-class scatter matrix is not discarded. Furthermore, EDA is equivalent to transforming original data into a new space by distance diffusion mapping, and then, LDA is applied in such a new space. As a result of diffusion mapping, the margin between different classes is enlarged, which is helpful in improving classification accuracy. Comparisons of experimental results on different data sets are given with respect to existing LDA extensions, including PCA + LDA, LDA via generalized singular value decomposition, regularized LDA, NLDA, and LDA via QR decomposition, which demonstrate the effectiveness of the proposed EDA method.