Null Space-based Linear Discriminant Analysis Research Articles

Comparison of Matrix Decomposition in Null Space-Based LDA Method

Problems with small sample sizes and high dimensionality are common in pattern recognition. Almost all machine learning algorithms degrade in high-dimensional data, so that singularities in the scatter matrices, the main problem of the Linear Discriminant Analysis (LDA) technique, might result. A null space-based LDA (NLDA) has been conceived to address the singularity issue. NLDA aims to maximize the distance between classes in the null space of the within-class scatter matrix. In the first research, the NLDA method was performed by computing the eigenvalue decomposition and singular value decomposition (SVD). This research led to several new implementations of the NLDA method that use other matrix decompositions. The new implementations include NLDA using Cholesky decomposition and NLDA using QR decomposition. This paper compares the performance of three NLDA methods that use different matrix decompositions, namely, SVD, Cholesky decomposition, and QR decomposition. Two sets of data were used in the experiments that used three different NLDA algorithms. To determine the performance of the NLDA methods, the classification accuracy of the three methods was measured using the confusion matrix. The results show that the NLDA method using SVD has the best performance when compared to the other two methods, achieving a precision of 77.8% accuracy for the Colon dataset and a precision of 98.8% accuracy for the TKI-resistance dataset.

L1-norm based null space discriminant analysis

Null space based linear discriminant analysis (NSLDA) is a well-known feature extraction method, which can make use of the most discriminant information in the null space of within-class scatter matrix. However, the conventional formulation of NSLDA is based on L2-norm which makes NSLDA be sensitive to outlier. To address the problem of NSLDA, in this paper, we propose a simple and robust NSLDA based on L1-norm (L1-NSLDA). An iterative algorithm for solving L1-NSLDA is also proposed. Compared to NSLDA, L1-NSLDA is more robust than NSLDA since it is more robust to outliers and noise. Experiment results on some image databases confirm the effectiveness of the proposed L1-NSLDA.

A computationally efficient scheme for feature extraction with kernel discriminant analysis

The kernel discriminant analysis (KDA), an extension of the linear discriminant analysis (LDA) and null space-based LDA into the kernel space, generally provides good pattern recognition (PR) performance for both small sample size (SSS) and non-SSS PR problems. Due to the eigen-decomposition technique adopted, however, the original scheme for the feature extraction with the KDA suffers from a high complexity burden. In this paper, we derive a transformation of the KDA into a linear equation problem, and propose a novel scheme for the feature extraction with the KDA. The proposed scheme is shown to provide us with a reduction of complexity without degradation of PR performance. In addition, to enhance the PR performance further, we address the incorporation of regularization into the proposed scheme.

Complexity-reduced implementations of complete and null-space-based linear discriminant analysis

Dimensionality reduction has become an important data preprocessing step in a lot of applications. Linear discriminant analysis (LDA) is one of the most well-known dimensionality reduction methods. However, the classical LDA cannot be used directly in the small sample size (SSS) problem where the within-class scatter matrix is singular. In the past, many generalized LDA methods has been reported to address the SSS problem. Among these methods, complete linear discriminant analysis (CLDA) and null-space-based LDA (NLDA) provide good performances. The existing implementations of CLDA are computationally expensive. In this paper, we propose a new and fast implementation of CLDA. Our proposed implementation of CLDA, which is the most efficient one, is equivalent to the existing implementations of CLDA in theory. Since CLDA is an extension of null-space-based LDA (NLDA), our implementation of CLDA also provides a fast implementation of NLDA. Experiments on some real-world data sets demonstrate the effectiveness of our proposed new CLDA and NLDA algorithms.

라그랑지 기법을 쓴 영 공간 기반 선형 판별 분석법의 변형 기법

Due to the singularity of the within-class scatter, linear discriminant analysis (LDA) becomes ill-posed for small sample size (SSS) problems. An extension of LDA, the null space-based LDA (NLDA) provides good discriminant performances for SSS problems. In this paper, by applying the Lagrange technique, the procedure of transforming the problem of finding the feature extractor of NLDA into a linear equation problem is derived.

Complexity-reduced scheme for feature extraction with linear discriminant analysis.

Owing to the singularity of the within-class scatter, linear discriminant analysis (LDA) becomes ill-posed for small sample size (SSS) problems. Null-space-based LDA (NLDA), which is an extension of LDA, provides good discriminant performances for SSS problems. Yet, as the original scheme for the feature extractor (FE) of NLDA suffers from a complexity burden, a few modified schemes have since been proposed for complexity reduction. In this brief, by transforming the problem of finding the FE of NLDA into a linear equation problem, a novel scheme is derived, offering a further reduction of the complexity.

Feature extraction using a fast null space based linear discriminant analysis algorithm

The small sample size problem is often encountered in pattern recognition. Several algorithms for null space based linear discriminant analysis (NLDA) have been developed to solve the problem. However, these algorithms for NLDA have high computational cost. In this paper, we simplify the recently proposed algorithm for NLDA in Chu and Thye (2010) [5] with the assumption that all the training data vectors are linearly independent and propose a new and fast algorithm for NLDA. Our main observation is that two steps of economic QR decomposition with column pivoting can be replaced by one step of economic QR decomposition without column pivoting if the related matrix is of full column rank. The main features of our algorithm for NLDA include: (i) our NLDA algorithm is carried out by only one step of economic QR decomposition and does not compute any singular value decomposition (SVD) when all the training data vectors are linearly independent; (ii) the main cost of our method is the cost of an economic QR decomposition of an m×(n−1) matrix, here m is the dimension of the training data and n is the number of samples. Our method is a fast one. Experimental studies on ORL, FERET and PIE face databases demonstrate the effectiveness of our new algorithm for NLDA.

FUSING GLOBAL AND LOCAL COMPLETE LINEAR DISCRIMINANT FEATURES BY FUZZY INTEGRAL FOR FACE RECOGNITION

Face recognition becomes very difficult in a complex environment, and the combination of multiple classifiers is a good solution to this problem. A novel face recognition algorithm GLCFDA-FI is proposed in this paper, which fuses the complementary information extracted by complete linear discriminant analysis from the global and local features of a face to improve the performance. The Choquet fuzzy integral is used as the fusing tool due to its suitable properties for information aggregation. Experiments are carried out on the CAS-PEAL-R1 database, the Harvard database and the FERET database to demonstrate the effectiveness of the proposed method. Results also indicate that the proposed method GLCFDA-FI outperforms five other commonly used algorithms — namely, Fisherfaces, null space-based linear discriminant analysis (NLDA), cascaded-LDA, kernel-Fisher discriminant analysis (KFDA), and null-space based KFDA (NKFDA).