LDA-based Methods Research Articles

Segmentation-based linear discriminant analysis with information theoretic feature selection for hyperspectral image classification

ABSTRACT The use of hyperspectral imaging sensors has greatly improved the classification of remotely sensed data because of the abundant spectral information they offer. However, the numerous contiguous, tiny wavelength bands captured in hyperspectral images (HSIs) often hinder the classification process. To overcome the aforementioned problem, various feature reduction techniques, including feature extraction (FE) and feature selection (FS), are commonly employed to improve classification performance. Linear discriminant analysis (LDA) is a well-established approach that has been utilized for the FE of HSI. LDA’s consideration of global characteristics and the variance accumulator for FS can lead to the poor reduction of HIS’s intrinsic characteristics. Furthermore, LDA’s limited ability to select a very low number of features, i.e. the number of classes minus one, restricts its effectiveness in HSI classification. Therefore, we introduce an FS method based on a non-linear information-theoretic measure, normalized mutual information (nMI) combined with minimum redundancy maximum relevance (mRMR). This approach is used to identify inherent features from the transformed space of our proposed correlation-based segmented-LDA (SLDA) and spectral region-based segmented-LDA (SSLDA) FE methods. We thoroughly compare the performance of the SLDA-mRMR and SSLDA-mRMR methods with the existing linear and non-linear state-of-the-art techniques, including unsupervised principal component analysis (PCA), PCA-based methods, supervised LDA, and LDA-based methods. Additionally, we explore the performance of nMI-based mRMR selection with all FE methods by incorporating a cumulative variance-based top features pick-up strategy. Based on the experimental results, we observe that SSLDA-mRMR and SLDA-mRMR achieve the highest classification accuracy, such as 91.91%, and 91.57% for agricultural Indian Pines, respectively, 97.66%, and 97.54%, respectively, for Kennedy Space Center, and 96.69%, and 96.57%, respectively, for Pavia University. In contrast, the classification accuracies using all original features of the HSIs are 71.39%, 70.01%, and 83.52%, respectively. Moreover, our proposed SSLDA-mRMR and SLDA-mRMR combinations outperformed all other FE and FS combinations that were examined using actual HSI datasets.

Linear discriminant analysis with generalized kernel constraint for robust image classification

Linear discriminant analysis (LDA) as a classical supervised dimensionality reduction method has shown powerful capability in various image classification tasks. The purpose of LDA seeks an optimal linear transformation that maps the original data to a low-dimensional space. Inspired by the fact that the kernel trick can capture the nonlinear similarity of features, we propose a novel generalized distance constraint dubbed intra-class and inter-class kernel constraint (IIKC). The proposed IIKC explicitly models the category kernel distance and focuses on helping the original LDA capture more discriminant features in order to further improve the separability and magnitude difference between nearby data points. Our proposed method with IIKC aims to achieve maximum category separability by minimizing the intra-class kernel distances as well as maximizing the inter-class kernel distance, simultaneously. Extensive experimental results on six publicly available benchmark databases illustrate that the LDA-based methods embedded with the proposed IIKC significantly improve the discrimination ability and achieve a better classification performance than the original and state-of-the-art LDA algorithms.

Don’t Be Spoiled by Your Friends: Spoiler Detection in TV Program Tweets

Providing a convenient mechanism for accessing the Internet, smartphones have led to the rapid growth of Social Networking Services (SNSs) such as Twitter and have served as a major platform for SNSs. Nowadays, people are able to check conveniently the SNS messages posted by their friends and followers via their smartphones. As a consequence, people are exposed to spoilers of TV programs that they follow. So far, there are two previous works that explored the detection of spoilers in texts, not SNS: (1) keyword matching method and (2) machine-learning method based on Latent Dirichlet Allocation (LDA). The keyword matching method evaluates most tweets as spoilers; hence its poor recall performance. The other method based on LDA, although successful on large text, works poorly on short segments of text such as those found on Twitter and evaluates most tweets as non-spoilers. This paper presents four features that are significant in the classification of spoiler tweets. Using those features, we classified spoiler tweets pertaining to a reality TV show (“Dancing with the Stars”). We experimentally compared our method with previous methods, with our method achieving substantially higher precision compared to the keyword matching and LDA-based methods while maintaining comparable recalls.

Fast local representation learning via adaptive anchor graph for image retrieval

Linear Discriminant Analysis (LDA) is one of most important methods in dimensionality reduction domain, which is limited with Gaussian assumption. Because of the complexity of real data, data often presents non-Gaussian distribution that points in same class can be divided into several sub-clusters and center point is not enough to describe the distribution of data. In order to solve non-Gaussian data, LDA-based methods consider local structure information through measuring each pairwise distance of full connection graph. However, the strategy of establishing fully-connected graph is at expense of high computational complexity and limits it practical and industrial applications. We propose a Fast Local Representation Learning (FLRL) method which leverages anchor points to establish anchor-based graph and uses similarity matrix to depict the relationships of each pairwise connections. Notably, to avoid the affect of noises and redundant features in original space, anchor points and similarity matrix are updated alternately in subspace that local structure of data will be more precise to learn. Extensive pattern classification and image retrieval experiments on several synthetic datasets, well-known datasets and deep features datasets demonstrate the advantages of our method over the state-of-the-art methods. Our source code available on:https://github.com/superzcy/FLRL.

Examining LDA2Vec and Tweet Pooling for Topic Modeling on Twitter Data

The short lengths of tweets present a challenge for topic modeling to extend beyond what is provided explicitly from hashtag information. This is particularly true for LDAbased methods because the amount of information available from pertweet statistical analysis is severely limited. In this paper we present LDA2Vec paired with temporal tweet pooling (LDA2VecTTP) and assess its performance on this problem relative to traditional LDA and to Biterm Topic Model (Biterm), which was developed specifically for topic modeling on short text documents. We paired each of the three topic modeling algorithms with three tweet pooling schemes: no pooling, authorbased pooling, and temporal pooling. We then conducted topic modeling on two Twitter datasets using each of the algorithms and the tweet pooling schemes. Our results on the largest dataset suggest that LDA2VecTTP can produce higher coherence scores and more logically coherent and interpretable topics.

TULIP: A Toolbox for Linear Discriminant Analysis with Penalties

Linear discriminant analysis (LDA) is a powerful tool in building classifiers with easy computation and interpretation. Recent advancements in science technology have led to the popularity of datasets with high dimensions, high orders and complicated structure. Such datasetes motivate the generalization of LDA in various research directions. The R package TULIP integrates several popular high-dimensional LDA-based methods and provides a comprehensive and user-friendly toolbox for linear, semi-parametric and tensor-variate classification. Functions are included for model fitting, cross validation and prediction. In addition, motivated by datasets with diverse sources of predictors, we further include functions for covariate adjustment. Our package is carefully tailored for low storage and high computation efficiency. Moreover, our package is the first R package for many of these methods, providing great convenience to researchers in this area.

Towards a Framework for Noctilucent Cloud Analysis

In this paper, we present a framework to study the spatial structure of noctilucent clouds formed by ice particles in the upper atmosphere at mid and high latitudes during summer. We studied noctilucent cloud activity in optical images taken from three different locations and under different atmospheric conditions. In order to identify and distinguish noctilucent cloud activity from other objects in the scene, we employed linear discriminant analysis (LDA) with feature vectors ranging from simple metrics to higher-order local autocorrelation (HLAC), and histogram of oriented gradients (HOG). Finally, we propose a convolutional neural networks (CNN)-based method for the detection of noctilucent clouds. The results clearly indicate that the CNN-based approach outperforms the LDA-based methods used in this article. Furthermore, we outline suggestions for future research directions to establish a framework that can be used for synchronizing the optical observations from ground-based camera systems with echoes measured with radar systems like EISCAT in order to obtain independent additional information on the ice clouds.

Sparse approximation to discriminant projection learning and application to image classification

Subspace learning for dimensionality reduction is an important topic in pattern analysis and machine learning, and it has extensive applications in feature representation and image classification. Linear discriminant analysis (LDA) is a well-known subspace learning approach for supervised dimensionality reduction due to its effectiveness and efficacy in discriminant analysis. However, LDA is not stable and suffers from the singularity problem when addressing small sample size and high-dimensional data. In this paper, we develop a novel subspace learning model, named sparse approximation to discriminant projection learning (SADPL), to learn the sparse projection matrix. Different from the traditional LDA-based methods, we learn the projection matrix based on a new objective function rather than the Fisher criterion, which avoids the matrix singularity problem. In order to distinguish which features play an important role in discriminant analysis, we embed a feature selection framework to the subspace learning model to select the informative features. Finally, we can attain a convex objective function which can be solved by an effective optimization algorithm, and theoretically prove the convergence of the proposed optimization algorithm. Extensive experiments on all sorts of image classification tasks, such as face recognition, palmprint recognition, object categorization and texture classification show that our SADPL achieves competitive performance compared to the state-of-the-art methods.

ITWF: A framework to apply term weighting schemes in topic model

Topic models like Latent Dirichlet Allocation (LDA) and its variants is a type of statistical model for discovering latent topics. However, as revealed by the previous research, some topics generated by LDA may be uninterpretable and semantically incoherent due to the occurrence of irrelevant words in these topics. To improve the semantic qualities of automatically discovered topics, we explore the distributional characteristics of words across topics to identify topic-indiscriminate words which are blamed for the low-quality topics. The main contribution of our research reported in this paper is that we develop a novel framework named Iterative Term Weighting Framework (ITWF) which can effectively identify and filter out topic-indiscriminate words from uncovered topics. In particular, the proposed framework first applies an entropy-based term weighting schemes and adopts a novel iterative method to identify topic-indiscriminate words. To the best of our knowledge, our research is among the very few successful work that aims to enhance both the semantic coherence and the interpretability of LDA-based topic modeling methods. The experimental results show that the proposed framework improves the effectiveness of LDA as well as its variants.

Probabilistic Linear Discriminant Analysis With Vectorial Representation for Tensor Data.

Linear discriminant analysis (LDA) has been a widely used supervised feature extraction and dimension reduction method in pattern recognition and data analysis. However, facing high-order tensor data, the traditional LDA-based methods take two strategies. One is vectorizing original data as the first step. The process of vectorization will destroy the structure of high-order data and result in high dimensionality issue. Another is tensor LDA-based algorithms that extract features from each mode of high-order data and the obtained representations are also high-order tensor. This paper proposes a new probabilistic LDA (PLDA) model for tensorial data, namely, tensor PLDA. In this model, each tensorial data are decomposed into three parts: the shared subspace component, the individual subspace component, and the noise part. Furthermore, the first two parts are modeled by a linear combination of latent tensor bases, and the noise component is assumed to follow a multivariate Gaussian distribution. Model learning is conducted through a Bayesian inference process. To further reduce the total number of model parameters, the tensor bases are assumed to have tensor CandeComp/PARAFAC (CP) decomposition. Two types of experiments, data reconstruction and classification, are conducted to evaluate the performance of the proposed model with the convincing result, which is superior or comparable against the existing LDA-based methods.

Multi-View Classification via Adaptive Discriminant Analysis

In many real applications, an object is usually represented with multiple views, providing compatible and complementary information to each other. Therefore, it is highly desirable to recognize the object from distinct and even heterogeneous views. In this paper, we propose a novel method, the named multi-view locality adaptively discriminant analysis (MvLADA), for multi-view classification. The MvLADA integrates subspace learning and weighted matrix learning into a uniform framework, where the weighted matrix is adaptively attained and shared by all views. Compared with the most existing LDA-based multi-view methods, the MvLADA adaptively assigns different weights to each sample, which enhances MvLADA's flexibility in practical applications. Moreover, the learned weighted matrix shared by all views exploits the point's neighbor relationship automatically without requiring a k NN procedure. Besides, the MvLADA is a parameter-free method without imposing any additional parameters. We validate the proposed MvLADA on three real-world datasets, indicating a better performance than the state-of-the-art multi-view algorithms.

Latent tree models for hierarchical topic detection

We present a novel method for hierarchical topic detection where topics are obtained by clustering documents in multiple ways. Specifically, we model document collections using a class of graphical models called hierarchical latent tree models (HLTMs). The variables at the bottom level of an HLTM are observed binary variables that represent the presence/absence of words in a document. The variables at other levels are binary latent variables that represent word co-occurrence patterns or co-occurrences of such patterns. Each latent variable gives a soft partition of the documents, and document clusters in the partitions are interpreted as topics. Latent variables at high levels of the hierarchy capture long-range word co-occurrence patterns and hence give thematically more general topics, while those at low levels of the hierarchy capture short-range word co-occurrence patterns and give thematically more specific topics. In comparison with LDA-based methods, a key advantage of the new method is that it represents co-occurrence patterns explicitly using model structures. Extensive empirical results show that the new method significantly outperforms the LDA-based methods in term of model quality and meaningfulness of topics and topic hierarchies.

Robust Matrix Discriminative Analysis for Feature Extraction From Hyperspectral Images

Linear discriminative analysis (LDA) is an effective feature extraction method for hyperspectral image (HSI) classification. Most of the existing LDA-related methods are based on spectral features, ignoring spatial information. Recently, a matrix discriminative analysis (MDA) model has been proposed to incorporate the spatial information into the LDA. However, due to sensor interferers, calibration errors, and other issues, HSIs can be noisy. These corrupted data easily degrade the performance of the MDA. In this paper, a robust MDA (RMDA) model is proposed to address this important issue. Specifically, based on the prior knowledge that the pixels in a small spatial neighborhood of the HSI lie in a low-rank subspace, a denoising model is first employed to recover the intrinsic components from the noisy HSI. Then, the MDA model is used to extract discriminative spatial–spectral features from the recovered components. Besides, different HSIs exhibit different spatial contextual structures, and even a single HSI may contain both large and small homogeneous regions simultaneously. To sufficiently describe these multiscale spatial structures, a multiscale RMDA model is further proposed. Experiments have been conducted using three widely used HSIs, and the obtained results show that the proposed method allows for a significant improvement in the classification performance when compared to other LDA-based methods.

Progressive EM for Latent Tree Models and Hierarchical Topic Detection

Hierarchical latent tree analysis (HLTA) is recently proposed as a new method for topic detection. It differs fundamentally from the LDA-based methods in terms of topic definition, topic-document relationship, and learning method. It has been shown to discover significantly more coherent topics and better topic hierarchies. However, HLTA relies on the Expectation-Maximization (EM) algorithm for parameter estimation and hence is not efficient enough to deal with large datasets. In this paper, we propose a method to drastically speed up HLTA using a technique inspired by the advances in the method of moments. Empirical experiments show that our method greatly improves the efficiency of HLTA. It is as efficient as the state-of-the-art LDA-based method for hierarchical topic detection and finds substantially better topics and topic hierarchies.

The application of sparse estimation of covariance matrix to quadratic discriminant analysis.

BackgroundAlthough Linear Discriminant Analysis (LDA) is commonly used for classification, it may not be directly applied in genomics studies due to the large p, small n problem in these studies. Different versions of sparse LDA have been proposed to address this significant challenge. One implicit assumption of various LDA-based methods is that the covariance matrices are the same across different classes. However, rewiring of genetic networks (therefore different covariance matrices) across different diseases has been observed in many genomics studies, which suggests that LDA and its variations may be suboptimal for disease classifications. However, it is not clear whether considering differing genetic networks across diseases can improve classification in genomics studies.ResultsWe propose a sparse version of Quadratic Discriminant Analysis (SQDA) to explicitly consider the differences of the genetic networks across diseases. Both simulation and real data analysis are performed to compare the performance of SQDA with six commonly used classification methods.ConclusionsSQDA provides more accurate classification results than other methods for both simulated and real data. Our method should prove useful for classification in genomics studies and other research settings, where covariances differ among classes.

KERNEL-BASED 2D FISHER DISCRIMINANT ANALYSIS WITH PARAMETER OPTIMIZATION FOR FACE RECOGNITION

As is known, kernel methods are developed to handle nonlinear classification. However, we have found that based on vector representation of face images, it is not easy to improve the performance of LDA-based methods by incorporating the kernel trick. Actually, in the case that the dimensionality of face images in vector form is far greater than the number of samples, linear classifiers are adequate and it is illogical to increase dimensions of vectors using kernel methods. In order to give full play to the strong point of the kernel technique for manipulating the nonlinearity of pattern distribution, we propose a new image feature extraction method for face recognition, called kernel-based two-dimensional Fisher discriminant analysis (K2DFDA), which deals with a face image directly as a matrix, instead of a stacked vector from rows or columns of the image. Moreover, we present a kernel parameter optimization scheme for K2DFDA, based on the maximum margin criterion and the damped Newton's method. Experimental results show the effectiveness of K2DFDA and its parameter optimization scheme.

Face Recognition Systems Using Relevance Weighted Two Dimensional Linear Discriminant Analysis Algorithm

Low-dimensional feature representation with enhanced discriminatory power of paramount importance to face recognition systems. Most of traditional linear discriminant analysis (LDA)-based methods suffer from the disadvantage that their optimality criteria are not directly related to the classification ability of the obtained feature representation. Moreover, their classification accuracy is affected by the “small sample size” (SSS) problem which is often encountered in face recognition tasks. In this paper, we propose a new technique coined Relevance-Weighted Two Dimensional Linear Discriminant Analysis (RW2DLDA). Its over comes the singularity problem implicitly, while achieving efficiency. Moreover, a weight discriminant hyper plane is used in the between class scatter matrix, and RW method is used in the within class scatter matrix to weigh the information to resolve confusable data in these classes. Experiments on two well known facial databases show the effectiveness of the proposed method. Comparisons with other LDA-based methods show that our method improves the LDA classification performance.

Face Biometrics Based on Principal Component Analysis and Linear Discriminant Analysis

Problem statement: In facial biometrics, face features are used as the required human traits for automatic recognition. Feature extracted from face images are significant for face biometrics system performance. Approach: In this thesis, a framework of facial biometric was designed based on two subspace methods i.e., Principal Component Analysis (PCA) and Linear Discriminant Analysis (LDA). First, PCA is used for dimension reduction, where original face images are projected into lower-dimensional face representations. Second, LDA was proposed to provide a solution of better discriminant. Both PCA and LDA features were presented to Euclidean distance measurement which is conveniently used as a benchmark. The algorithms were evaluated in face identification and verification using a standard face database-AT and T and a locally collected database-CBE. Each database consists of 400 images and 320 images respectively. Results: LDA-based methods outperform PCA for both face identification and verification. For face identification, PCA achieves accuracy of 91.9% (AT and T) and 76.7% (CBE) while LDA 94.2% (AT and T) and 83.1% (CBE). For face verification, PCA achieves Equal Error Rate (EER) of 1.15% (AT and T), 7.3% (CBE) while LDA 0.78% (AT and T) and 5.81% (CBE). Conclusion/Recommendations: This study had proved that, when given sufficient training samples, LDA is able to provide better discriminant ability in feature extraction for face biometrics.

Nonparametric Discriminant Analysis for Face Recognition

In this paper, we develop a new framework for face recognition based on nonparametric discriminant analysis (NDA) and multi-classifier integration. Traditional LDA-based methods suffer a fundamental limitation originating from the parametric nature of scatter matrices, which are based on the Gaussian distribution assumption. The performance of these methods notably degrades when the actual distribution is Non-Gaussian. To address this problem, we propose a new formulation of scatter matrices to extend the two-class nonparametric discriminant analysis to multi-class cases. Then, we develop two more improved multi-class NDA-based algorithms (NSA and NFA) with each one having two complementary methods based on the principal space and the null space of the intra-class scatter matrix respectively. Comparing to the NSA, the NFA is more effective in the utilization of the classification boundary information. In order to exploit the complementary nature of the two kinds of NFA (PNFA and NNFA), we finally develop a dual NFA-based multi-classifier fusion framework by employing the over complete Gabor representation to boost the recognition performance. We show the improvements of the developed new algorithms over the traditional subspace methods through comparative experiments on two challenging face databases, Purdue AR database and XM2VTS database.

WAVELET-FACE BASED SUBSPACE LDA METHOD TO SOLVE SMALL SAMPLE SIZE PROBLEM IN FACE RECOGNITION

Linear Discriminant Analysis (LDA) is a popular statistical method for both feature extraction and dimensionality reduction in face recognition. The major drawback of LDA is the so-called small sample size (3S) problem. This problem always occurs when the total number of training samples is smaller than the dimension of feature space. Under this situation, the within-class scatter matrix Sw becomes singular and LDA approach cannot be implemented directly. To overcome the 3S problem, this paper proposes a novel wavelet-face based subspace LDA algorithm. Wavelet-face feature extraction and dimensionality reduction are based on two-level D4-filter wavelet transform and discarding the null space of total class scatter matrix St. It is shown that our obtained projection matrix satisfies the uncorrelated constraint conditions. Hence in the sense of statistical uncorrelation, this projection matrix is optimal. The proposed method for face recognition has been evaluated with two public available databases, namely ORL and FERET databases. Comparing with existing LDA-based methods to solve the 3S problem, our method gives the best performance.