Problem Of Linear Discriminant Analysis Research Articles

An Efficient Greedy Search Algorithm for High-dimensional Linear Discriminant Analysis.

High-dimensional classification is an important statistical problem that has applications in many areas. One widely used classifier is the Linear Discriminant Analysis (LDA). In recent years, many regularized LDA classifiers have been proposed to solve the problem of high-dimensional classification. However, these methods rely on inverting a large matrix or solving large-scale optimization problems to render classification rules-methods that are computationally prohibitive when the dimension is ultra-high. With the emergence of big data, it is increasingly important to develop more efficient algorithms to solve the high-dimensional LDA problem. In this paper, we propose an efficient greedy search algorithm that depends solely on closed-form formulae to learn a high-dimensional LDA rule. We establish theoretical guarantee of its statistical properties in terms of variable selection and error rate consistency; in addition, we provide an explicit interpretation of the extra information brought by an additional feature in a LDA problem under some mild distributional assumptions. We demonstrate that this new algorithm drastically improves computational speed compared with other high-dimensional LDA methods, while maintaining comparable or even better classification performance.

Sparse linear discriminant analysis in structured covariates space

Classification with high‐dimensional variables is a popular goal in many modern statistical studies. Fisher's linear discriminant analysis (LDA) is a common and effective tool for classifying entities into existing groups. It is well known that classification using Fisher's discriminant for high‐dimensional data is as bad as random guessing because of the use of many noise features, which increases the misclassification rate. Recently, it is being acknowledged that complex biological mechanisms occur through multiple features working together, though individually these features may contribute to noise accumulation in the data. In view of these, it is important to perform classification with discriminant vectors that use a subset of important variables, while also utilizing prior biological relationships among features. We tackle this problem in this paper and propose methods that incorporate variable selection into the classification problem for the identification of important biomarkers. Furthermore, we incorporate into the LDA problem prior information on the relationships among variables using undirected graphs in order to identify functionally meaningful biomarkers. We compare our methods with existing sparse LDA approaches via simulation studies and real data analysis.

Multispectral palmprint recognition using Pascal coefficients-based LBP and PHOG descriptors with random sampling

Local binary pattern (LBP) algorithm and its variants have been used extensively to analyse the local textural features of digital images with great success. Numerous extensions of LBP descriptors have been suggested, focusing on improving their robustness to noise and changes in image conditions. In our research, inspired by the concepts of LBP feature descriptors and a random sampling subspace, we propose an ensemble learning framework, using a variant of LBP constructed from Pascal’s coefficients of n-order and referred to as a multiscale local binary pattern. To address the inherent overfitting problem of linear discriminant analysis, PCA was applied to the training samples. Random sampling was used to generate multiple feature subsets. In addition, in this work, we propose a new feature extraction technique that combines the pyramid histogram of oriented gradients and LBP, where the features are concatenated for use in the classification. Its performance in recognition was evaluated using the Hong Kong Polytechnic University database. Extensive experiments unmistakably show the superiority of the proposed approach compared to state-of-the-art techniques.

Big Feature Data Analytics: Split and Combine Linear Discriminant Analysis (SC-LDA) for Integration Towards Decision Making Analytics

This paper introduces a novel big feature data analytics scheme for integration toward data analytics with decision making. In this scheme, a split and combine approach for a linear discriminant analysis (LDA) algorithm termed SC-LDA is proposed. The SC-LDA replaces the full eigenvector decomposition of LDA with much cheaper eigenvector decompositions on smaller sub-matrices, and then recombines the intermediate results to obtain the exact reconstruction as for the original algorithm. The splitting or decomposition can be further applied recursively to obtain a multi-stage SC-LDA algorithm. The smaller sub-matrices can then be computed in parallel to reduce the time complexity for big data applications. The approach is discussed for an LDA algorithm variation (LDA/QR), which is suitable for the analytics of Big Feature data sets. The projected data vectors into the LDA subspace can then be integrated toward the decision-making process involving classification. Experiments are conducted on real-world data sets to confirm that our approach allows the LDA problem to be divided into the size-reduced sub-problems and can be solved in parallel while giving an exact reconstruction as for the original LDA/QR.

Kernel Locality Preserving Symmetrical Weighted Fisher Discriminant Analysis based subspace approach for expression recognition

This paper mainly focuses on dimensional reduction of fused dataset of holistic and geometrical face features vectors by solving singularity problem of linear discriminant analysis and maximizing the Fisher ratio in nonlinear subspace region with the preservation of local discriminative features. The combinational feature vector space is projected into low dimensional subspace using proposed Kernel Locality Preserving Symmetrical Weighted Fisher Discriminant Analysis (KLSWFDA) method. Matching score level fusion technique has been applied on projected subspace and combinational entire Gabor subspace is framed. Euclidean distance metric (L2) and support vector machine (SVM) classifier has been implemented to recognize and classify the expressions. Performance of proposed approach is evaluated and compared with state of art approaches. Experimental results on JAFFE, YALE and FD expression database demonstrate the effectiveness of the proposed approach.

Image clustering using exponential discriminant analysis

Local learning based image clustering models are usually employed to deal with images sampled from the non‐linear manifold. Recently, linear discriminant analysis (LDA) based various clustering models were proposed. However, in these clustering models, regularisation parameter was added to handle the small‐sample‐size (SSS) problem of LDA for high‐dimensional image data. Owing to this, the authors had to tune a number of clustering parameters for optimal clustering performance in existing local learning based clustering approaches. In this study, the less‐parameterised local learning based image clustering model is proposed. The proposed local exponential discriminant clustering (LEDC) model is based on exponential discriminant analysis (EDA). In the LEDC model, local scatter matrices are projected in the exponential domain in order to handle the SSS problem of LDA without adding regularisation parameter. In the proposed LEDC model, k‐nearest neighbours are the only clustering parameter as compared with existing local learning based clustering approaches such as normalised cut, spectral embedded clustering and local discriminant model and global integration (LDMGI). Experimental results on twelve benchmark image datasets show that the LEDC model achieved a comparable clustering performance as that of the near competitor LDMGI model. Clustering performance is comparable because no discriminant information of LDA is lost in EDA. The authors concluded that the proposed LEDC clustering model is less‐parameterised with a comparable clustering performance as that of existing state‐of‐the‐art local learning based clustering approaches.

A Rayleigh–Ritz style method for large-scale discriminant analysis

Linear Discriminant Analysis (LDA) is one of the most popular approaches for supervised feature extraction and dimension reduction. However, the computation of LDA involves dense matrices eigendecomposition, which is time-consuming for large-scale problems. In this paper, we present a novel algorithm called Rayleigh–Ritz Discriminant Analysis (RRDA) for efficiently solving LDA. While much of the prior research focus on transforming the generalized eigenvalue problem into a least squares formulation, our method is instead based on the well-established Rayleigh–Ritz framework for general eigenvalue problems and seeks to directly solve the generalized eigenvalue problem of LDA. By exploiting the structures in LDA problems, we are able to design customized and highly efficient subspace expansion and extraction strategy for the Rayleigh–Ritz procedure. To reduce the storage requirement and computational complexity of RRDA for high dimensional, low sample size data, we also establish an equivalent reduced model of RRDA. Practical implementations and the convergence result of our method are also discussed. Our experimental results on several real world data sets indicate the performance of the proposed algorithm.

Efficient optimally regularized discriminant analysis

Regularized discriminant analysis (RDA) and its special case uncorrelated linear discriminant analysis (ULDA) are important subspace learning methods proposed recently to handle the small sample size (SSS) problem of linear discriminant analysis (LDA). One important unsolved issue of RDA is how to automatically determine an appropriate regularization parameter without resorting to unscalable procedures like cross-validation (CV). In this paper, we develop a novel efficient algorithm to automatically estimate the regularization parameter based on a geometric interpretation of RDA. We further provide a formal analysis of the proposed method, and show that it is robust to the perturbation in the feature space of the training data. The extensive experiments on various benchmark datasets verify the scalability and effectiveness of our approach, compared with the state-of-the-art algorithms.

Maxi-Min discriminant analysis via online learning

Linear Discriminant Analysis (LDA) is an important dimensionality reduction algorithm, but its performance is usually limited on multi-class data. Such limitation is incurred by the fact that LDA actually maximizes the average divergence among classes, whereby similar classes with smaller divergence tend to be merged in the subspace. To address this problem, we propose a novel dimensionality reduction method called Maxi-Min Discriminant Analysis (MMDA). In contrast to the traditional LDA, MMDA attempts to find a low-dimensional subspace by maximizing the minimal (worst-case) divergence among classes. This "minimal" setting overcomes the problem of LDA that tends to merge similar classes with smaller divergence when used for multi-class data. We formulate MMDA as a convex problem and further as a large-margin learning problem. One key contribution is that we design an efficient online learning algorithm to solve the involved problem, making the proposed method applicable to large scale data. Experimental results on various datasets demonstrate the efficiency and the efficacy of our proposed method against five other competitive approaches, and the scalability to the data with thousands of classes.

From NLDA to LDA/GSVD: a modified NLDA algorithm

Linear discriminant analysis (LDA) often encounters small sample size (SSS) problem for high-dimensional data. Null space linear discriminant analysis (NLDA) and linear discriminant analysis based on generalized singular value decomposition (LDA/GSVD) are two popular methods that can solve SSS problem of LDA. In this paper, we present the relation between NLDA and LDA/GSVD under a condition and at the same time propose a modified NLDA (MNLDA) algorithm which has the same discriminating power as LDA/GSVD and is more efficient. In addition, we compare the discriminating capability of NLDA and MNLDA and present our interpretation about this. Experimental results on ORL, FERET, Yale face databases, and the PolyU FKP database support our viewpoints.

Multi-Task Learning in Heterogeneous Feature Spaces

Multi-task learning aims at improving the generalization performance of a learning task with the help of some other related tasks. Although many multi-task learning methods have been proposed, they are all based on the assumption that all tasks share the same data representation. This assumption is too restrictive for general applications. In this paper, we propose a multi-task extension of linear discriminant analysis (LDA), called multi-task discriminant analysis (MTDA), which can deal with learning tasks with different data representations. For each task, MTDA learns a separate transformation which consists of two parts, one specific to the task and one common to all tasks. A by-product of MTDA is that it can alleviate the labeled data deficiency problem of LDA. Moreover, unlike many existing multi-task learning methods, MTDA can handle binary and multi-class problems for each task in a generic way. Experimental results on face recognition show that MTDA consistently outperforms related methods.

Learning Semi-Riemannian Metrics for Semisupervised Feature Extraction

Discriminant feature extraction plays a central role in pattern recognition and classification. Linear Discriminant Analysis (LDA) is a traditional algorithm for supervised feature extraction. Recently, unlabeled data have been utilized to improve LDA. However, the intrinsic problems of LDA still exist and only the similarity among the unlabeled data is utilized. In this paper, we propose a novel algorithm, called Semisupervised Semi-Riemannian Metric Map (S3RMM), following the geometric framework of semi Riemannian manifolds. S3RMM maximizes the discrepancy of the separability and similarity measures of scatters formulated by using semi-Riemannian metric tensors. The metric tensor of each sample is learned via semisupervised regression. Our method can also be a general framework for proposing new semisupervised algorithms, utilizing the existing discrepancy-criterion-based algorithms. The experiments demonstrated on faces and handwritten digits show that S3RMM is promising for semisupervised feature extraction.

Normalized Direct Linear Discriminant Analysis with its Application to Face Recognition

The dimensionality of sample is often larger than the number of training samples for highdimensional pattern recognition such as face recognition. Here linear discriminant analysis (LDA) cannot be performed directly because of the singularity of the within-class scatter matrix. This is socalled “small sample size” (SSS) problem. PCA plus LDA (FDA) and Direct LDA (DLDA) are two popular methods to solve the SSS problem of LDA. In this paper, we point out the relationship of these two methods and discuss the deficiency of DLDA. Then a normalized direct linear discriminant analysis (NDLDA) method which overcomes DLDA’s deficiency is proposed. Experiments on ORL, YALE and AR face databases show NDLDA’s superiority over DLDA and FDA.

Robust kernel discriminant analysis using fuzzy memberships

Linear discriminant analysis (LDA) is a simple but widely used algorithm in the area of pattern recognition. However, it has some shortcomings in that it is sensitive to outliers and limited to linearly separable cases. To solve these problems, in this paper, a non-linear robust variant of LDA, called robust kernel fuzzy discriminant analysis (RKFDA) is proposed. RKFDA uses fuzzy memberships to reduce the effect of outliers and adopts kernel methods to accommodate non-linearly separable cases. There have been other attempts to solve the problems of LDA, including attempts using kernels. However, RKFDA, encompassing previous methods, is the most general one. Furthermore, theoretical analysis and experimental results show that RKFDA is superior to other existing methods in solving the problems.

자료별 분류분석(DDA)에 의한 특징추출

본 논문은 선형차원감소(Linear Dimensionality Reduction)을 위해 널리 이용되고 있는 특징추출 알고리듬인 선형판별분석(Linear Discriminant Analysis)의 문제점을 해결할 수 있는 새로운 특징추출 알고리듬을 제안한다. 선형판별분석에 포함되는 평균-자료 간 거리 및 평균-평균 간의 거리에 기반한 분산행렬은 역행렬 연산, 계수의 제한 등으로 인하여 계산상의 문제와 추출되는 특징의 수가 제한되는 한계를 가지고 있다. 또한 자료의 집단이 단일 모드의 정규 분포로부터 얻어진 것으로 가정되며 그렇지 않은 경우에 대해서는 적절한 결과를 얻을 수 없다. 본 논문에서는 자료-자료 간의 거리에 기반하고 적절하게 가중치가 추가된 새로운 행렬을 정의하였으며. 이에 기반하여 특징을 추출하는 방법을 제안하였다. 그럼으로써 앞서 선형판별분석의 여러 문제를 해결하고자 시도하였다. 제안된 방법의 성능을 실험을 통해 확인하였다. This paper presents a new feature extraction algorithm which can deal with the problems of linear discriminant analysis, widely used for linear dimensionality reduction. The scatter matrices included in linear discriminant analysis are defined by the distances between each datum and its class mean, and those between class means and mean of whole data. Use of these scatter matrices can cause computational problems and the limitation on the number of features. In addition, these definition assumes that the data distribution is unimodal and normal, for the cases not satisfying this assumption the appropriate features are not achieved. In this paper we define a new scatter matrix which is based on the differently weighted distances between individual data, and presents a feature extraction algorithm using this scatter matrix. With this new method. the mentioned problems of linear discriminant analysis can be avoided, and the features appropriate for discriminating data can be achieved. The performance of this new method is shown by experiments.

Optimization of Polarimetric Contrast Enhancement Based on Fisher Criterion

The optimization of polarimetric contrast enhancement (OPCE) is a widely used method for maximizing the received power ratio of a desired target versus an undesired target (clutter). In this letter, a new model of the OPCE is proposed based on the Fisher criterion. By introducing the well known two-class problem of linear discriminant analysis (LDA), the proposed model is to enlarge the normalized distance of mean value between the target and the clutter. In addition, a cross-iterative numerical method is proposed for solving the optimization with a quadratic constraint. Experimental results with the polarimetric SAR (POLSAR) data demonstrate the effectiveness of the proposed method.

Three-dimensional distortion-tolerant object recognition using photon-counting integral imaging

This paper addresses three-dimensional distortion-tolerant object recognition using photon-counting integral imaging (II). A photon-counting linear discriminant analysis (LDA) is proposed for classification photonlimited images. In the photon-counting LDA, classical irradiance images are used to train the classifier. The unknown objects used to test the classifier are labeled by the number of photons detected. The optimal solution of the Fisher's LDA for photon-limited images is found to be different from the case when irradiance values are used. This difference results in one of the merits of a photon-counting LDA, namely that the high dimensionality of the image can be handled without preprocessing. Thus, the singularity problem of the Fisher's LDA encountered in the use of irradiance images can be avoided. By using photon-counting II, we build a compact distortiontolerant recognition system that makes use of the multiple-perspective imaging of II to enhance the recognition performance. Experimental and simulation results are presented to classify out-of-plane rotated objects. The performance is analyzed in terms of mean-squared distance (MSD) between the irradiance images. It is shown that a low level of photons is sufficient in the proposed technique.

Monitoring of an industrial liquid–liquid extraction system with kernel-based methods

The behaviour of liquid–liquid extraction systems can be complex and as a result linear methods of process condition monitoring such as principal component analysis or partial least squares may not be able to detect and identify process faults when they occur. In contrast, kernel-based methods represent a general framework that can be used where linear methods do not perform satisfactorily. This is demonstrated in a case study on an industrial liquid–liquid extraction system, where a linear discriminant analysis problem is recast as a support vector machine learning problem. The support vector machine is subsequently used to extract features from the plant data that can be used for considerably more accurate fault detection than is possible with its linear equivalent or with other nonlinear methods. Fault identification can be accomplished from an analysis of the residuals of models using the features to reconstruct the original plant data.

A two-stage linear discriminant analysis via QR-decomposition

Linear Discriminant Analysis (LDA) is a well-known method for feature extraction and dimension reduction. It has been used widely in many applications involving high-dimensional data, such as image and text classification. An intrinsic limitation of classical LDA is the so-called singularity problems; that is, it fails when all scatter matrices are singular. Many LDA extensions were proposed in the past to overcome the singularity problems. Among these extensions, PCA+LDA, a two-stage method, received relatively more attention. In PCA+LDA, the LDA stage is preceded by an intermediate dimension reduction stage using Principal Component Analysis (PCA). Most previous LDA extensions are computationally expensive, and not scalable, due to the use of Singular Value Decomposition or Generalized Singular Value Decomposition. In this paper, we propose a two-stage LDA method, namely LDA/QR, which aims to overcome the singularity problems of classical LDA, while achieving efficiency and scalability simultaneously. The key difference between LDA/QR and PCA+LDA lies in the first stage, where LDA/QR applies QR decomposition to a small matrix involving the class centroids, while PCA+LDA applies PCA to the total scatter matrix involving all training data points. We further justify the proposed algorithm by showing the relationship among LDA/QR and previous LDA methods. Extensive experiments on face images and text documents are presented to show the effectiveness of the proposed algorithm.

Comparison of two estimators of mean function in LDA of spatially correlated Gaussian data

Abstract The problem of linear discriminant analysis of an observation of Gaussian random field into one of two populations is considered. In this paper we analyze the performance of the plug‐in linear discriminant function, when unknown means are estimated from the training samples. The generalized least squares and the ordinary least squares estimators are used. Obtained asymptotic expansions for the expected error rate are compared numerically in the case of spherical models for population covariances.