Extension Of Linear Discriminant Analysis Research Articles

Looking back: Selected contributions by C. R. Rao to multivariate analysis

AbstractStatistician C. R. Rao made many contributions to multivariate analysis over the span of his career. Some of his earliest contributions continue to be used and built upon almost 80 years later, while his more recent contributions spur new avenues of research. The present article discusses these contributions, how they helped shape multivariate analysis as we see it today, and what we may learn from reviewing his works. Topics include his extension of linear discriminant analysis, Rao's perimeter test, Rao's U statistic, his asymptotic expansion of Wilks' statistic, canonical factor analysis, functional principal component analysis, redundancy analysis, canonical coordinates, and correspondence analysis. The examination of his works shows that interdisciplinary collaboration and the utilization of real datasets were crucial in almost all of Rao's impactful contributions.

Quadratic Multilinear Discriminant Analysis for Tensorial Data Classification

Over the past decades, there has been an increase of attention to adapting machine learning methods to fully exploit the higher order structure of tensorial data. One problem of great interest is tensor classification, and in particular the extension of linear discriminant analysis to the multilinear setting. We propose a novel method for multilinear discriminant analysis that is radically different from the ones considered so far, and it is the first extension to tensors of quadratic discriminant analysis. Our proposed approach uses invariant theory to extend the nearest Mahalanobis distance classifier to the higher-order setting, and to formulate a well-behaved optimization problem. We extensively test our method on a variety of synthetic data, outperforming previously proposed MDA techniques. We also show how to leverage multi-lead ECG data by constructing tensors via taut string, and use our method to classify healthy signals versus unhealthy ones; our method outperforms state-of-the-art MDA methods, especially after adding significant levels of noise to the signals. Our approach reached an AUC of 0.95(0.03) on clean signals—where the second best method reached 0.91(0.03)—and an AUC of 0.89(0.03) after adding noise to the signals (with a signal-to-noise-ratio of −30)—where the second best method reached 0.85(0.05). Our approach is fundamentally different than previous work in this direction, and proves to be faster, more stable, and more accurate on the tests we performed.

Incremental and compressible kernel null discriminant analysis

Kernel discriminant analysis (KDA), the nonlinear extension of linear Discriminant Analysis (LDA), is a popular tool for learning one or multiple categories in nonlinear data sets. However, in most modern pattern recognition applications such as video surveillance, data are collected in flow and require sequential processing. In this context, KDA is faced two critical issues: an original formulation unsuited to the dynamic nature of the data and an increasing memory requirement for the kernel matrix storage. Motivated by the state-of-the-art performance reported by the null KDA, we propose in this paper a new solution to solve the null KDA (NKDA) in the context of data streams. Compared to previous works, our contribution is based on three points: first, we develop an exact incremental scheme which guarantees accurate solutions. Secondly, we develop a compression mechanism based on the following observation: rger the size of the training data set more the distances in the null space contract This property of the null space leads to formulate an indicator of redundancy in the training data set. This criterion is the cornerstone of our incremental KNDA because it authorizes incremental learning on large-scale data sets. Third, the problem of novelty detection in multi-class and one-class scenarios is addressed. More precisely, the fact that distances in the null space change over the training period leads us to define adjustable novelty thresholds. Lastly, numerous experiments based on various publicly available data sets and state-of-the-art classifiers show that the proposed method is effective both for multi-class and one-class real applications.

Time-Variant Linear Discriminant Analysis Improves Hand Gesture and Finger Movement Decoding for Invasive Brain-Computer Interfaces.

Invasive brain-computer interfaces yield remarkable performance in a multitude of applications. For classification experiments, high-gamma bandpower features and linear discriminant analysis (LDA) are commonly used due to simplicity and robustness. However, LDA is inherently static and not suited to account for transient information that is typically present in high-gamma features. To resolve this issue, we here present an extension of LDA to the time-variant feature space. We call this method time-variant linear discriminant analysis (TVLDA). It intrinsically provides a feature reduction stage, which makes external approaches thereto obsolete, such as feature selection techniques or common spatial patterns (CSPs). As well, we propose a time-domain whitening stage which equalizes the pronounced 1/f-shape of the typical brain-wave spectrum. We evaluated our proposed architecture based on recordings from 15 epilepsy patients with temporarily implanted subdural grids, who participated in additional research experiments besides clinical treatment. The experiments featured two different motor tasks involving three high-level gestures and individual finger movement. We used log-transformed bandpower features from the high-gamma band (50–300 Hz, excluding power-line harmonics) for classification. On average, whitening improved the classification performance by about 11%. On whitened data, TVLDA outperformed LDA with feature selection by 11.8%, LDA with CSPs by 13.9%, and regularized LDA with vectorized features by 16.4%. At the same time, TVLDA only required one or two internal features to achieve this. TVLDA provides stable results even if very few trials are available. It is easy to implement, fully automatic and deterministic. Due to its low complexity, TVLDA is suited for real-time brain-computer interfaces. Training is done in less than a second. TVLDA performed particularly well in experiments with data from high-density electrode arrays. For example, the three high-level gestures were correctly identified at a rate of 99% over all subjects. Similarly, the decoding accuracy of individual fingers was 96% on average over all subjects. To our knowledge, these mean accuracies are the highest ever reported for three-class and five-class motor-control BCIs.

Unsupervised Linear Discriminant Analysis for Jointly Clustering and Subspace Learning

Linear discriminant analysis (LDA) is one of commonly used supervised subspace learning methods. However, LDA will be powerless faced with the no-label situation. In this paper, the unsupervised LDA (Un-LDA) is proposed and first formulated as a seamlessly unified objective optimization which guarantees convergence during the iteratively alternative solving process. The objective optimization is in both the ratio trace and the trace ratio forms, forming a complete framework of a new approach to jointly clustering and unsupervised subspace learning. The extension of LDA into Un-LDA enables to not only complete unsupervised subspace learning via the explicitly presented subspace projection matrix but also simultaneously finish clustering and even clustering out-of-sample data via the explicitly presented transformation matrix. To overcome the difficulty in solving the non-convex objective optimization, we mathematically prove that the Un-LDA optimization in both forms can be transformed into the simple K-means clustering optimization when the subspace is determined. The Un-LDA optimization is eventually completed by alternatively optimizing the clusters using K-means and the subspace using the supervised LDA methods and iterating this whole process until convergence or stopping criterion. The experiments demonstrate that our proposed Un-LDA algorithms are comparable or even much superior to the counterparts.

Graph-based sparse linear discriminant analysis for high-dimensional classification

Linear discriminant analysis (LDA) is a well-known classification technique that enjoyed great success in practical applications. Despite its effectiveness for traditional low-dimensional problems, extensions of LDA are necessary in order to classify high-dimensional data. Many variants of LDA have been proposed in the literature. However, most of these methods do not fully incorporate the structure information among predictors when such information is available. In this paper, we introduce a new high-dimensional LDA technique, namely graph-based sparse LDA (GSLDA), that utilizes the graph structure among the features. In particular, we use the regularized regression formulation for penalized LDA techniques, and propose to impose a structure-based sparse penalty on the discriminant vector β. The graph structure can be either given or estimated from the training data. Moreover, we explore the relationship between the within-class feature structure and the overall feature structure. Based on this relationship, we further propose a variant of our proposed GSLDA to utilize effectively unlabeled data, which can be abundant in the semi-supervised learning setting. With the new regularization, we can obtain a sparse estimate of β and more accurate and interpretable classifiers than many existing methods. Both the selection consistency of β estimation and the convergence rate of the classifier are established, and the resulting classifier has an asymptotic Bayes error rate. Finally, we demonstrate the competitive performance of the proposed GSLDA on both simulated and real data studies.

LDA Extension via Oblique Projection

The classical linear discriminant analysis (LDA) was previously modified by orthogonal projection into null space LDA (N_LDA) and direct LDA (D_LDA) for solving small sample size (SSS) problem. In this paper, the author proposes an extension of LDA by oblique projection, wherein N_LDA and D_LDA are included as special cases, to reduce discriminative information loss resulted from single N_LDA or D_LDA. The effectiveness of the proposed algorithm is tested by image forensics and face recognition.

Higher order spectral regression discriminant analysis (HOSRDA): A tensor feature reduction method for ERP detection

Tensors are valuable tools to represent Electroencephalogram (EEG) data. Tucker decomposition is the most used tensor decomposition in multidimensional discriminant analysis and tensor extension of Linear Discriminant Analysis (LDA), called Higher Order Discriminant Analysis (HODA), is a popular tensor discriminant method used for analyzing Event Related Potentials (ERP). In this paper, we introduce a new tensor-based feature reduction technique, named Higher Order Spectral Regression Discriminant Analysis (HOSRDA), for use in a classification framework for ERP detection. The proposed method (HOSRDA) is a tensor extension of Spectral Regression Discriminant Analysis (SRDA) and casts the eigenproblem of HODA to a regression problem. The formulation of HOSRDA can open a new framework for adding different regularization constraints in higher order feature reduction problem. Additionally, when the dimension and number of samples is very large, the regression problem can be solved via efficient iterative algorithms. We applied HOSRDA on data of a P300 speller from BCI competition III and reached average character detection accuracy of 96.5% for the two subjects. HOSRDA outperforms almost all of other reported methods on this dataset. Additionally, the results of our method are fairly comparable with those of other methods when 5 and 10 repetitions are used in the P300 speller paradigm.

Separability-Oriented Subclass Discriminant Analysis.

Linear discriminant analysis (LDA) is a classical method for discriminative dimensionality reduction. The original LDA may degrade in its performance for non-Gaussian data, and may be unable to extract sufficient features to satisfactorily explain the data when the number of classes is small. Two prominent extensions to address these problems are subclass discriminant analysis (SDA) and mixture subclass discriminant analysis (MSDA). They divide every class into subclasses and re-define the within-class and between-class scatter matrices on the basis of subclass. In this paper we study the issue of how to obtain subclasses more effectively in order to achieve higher class separation. We observe that there is significant overlap between models of the subclasses, which we hypothesise is undesirable. In order to reduce their overlap we propose an extension of LDA, separability oriented subclass discriminant analysis (SSDA), which employs hierarchical clustering to divide a class into subclasses using a separability oriented criterion, before applying LDA optimisation using re-defined scatter matrices. Extensive experiments have shown that SSDA has better performance than LDA, SDA and MSDA in most cases. Additional experiments have further shown that SSDA can project data into LDA space that has higher class separation than LDA, SDA and MSDA in most cases.

Phone classification via manifold learning based dimensionality reduction algorithms

Mechanical limitations imposed on the articulators during speech production lead to a limitation of the intrinsic dimensionality of speech signals. This limitation leads to a specific neighborhood structure of speech sounds when they are represented in a high-dimensional feature space. We investigate whether phone classification can be improved by exploiting this neighborhood structure, by means of extended variants of the conventional Linear Discriminant Analysis (LDA) based on manifold learning.In this extended LDA approach, the within-class and between-class scatter matrices are defined in terms of adjacency graphs. We compare extensions of LDA that use either a full adjacency graph or an adjacency graph defined in the neighborhood of the training observations. In addition, we apply different kernels for weighing the distances in the graphs via different kernels, of which the Adaptive Kernel is proposed in this paper.Experiments with TIMIT show that while LDA algorithms that use the full adjacency graph do not outperform traditional LDA, the algorithms that exploit only local information provide significantly better results than traditional LDA. These improvements are not uniform across different broad phonetic classes, which suggests that the added value of the neighborhood structure is phone class dependent. The structure is represented by locally different densities in the neighborhood of feature vectors that are representative of a specific phone in a specific context.

Classification of Multi-Frequency Polarimetric SAR Images Based on Multi-Linear Subspace Learning of Tensor Objects

One key problem for the classification of multi-frequency polarimetric SAR images is to extract target features simultaneously in the aspects of frequency, polarization and spatial texture. This paper proposes a new classification method for multi-frequency polarimetric SAR data based on tensor representation and multi-linear subspace learning (MLS). Firstly, each cell of the SAR images is represented by a third-order tensor in the frequency, polarization and spatial domains, with each order of tensor corresponding to one domain. Then, two main MLS methods, i.e., multi-linear principal component analysis (MPCA) and multi-linear extension of linear discriminant analysis (MLDA), are used to learn the third-order tensors. MPCA is used to analyze the principal component of the tensors. MLDA is applied to improve the discrimination between different land covers. Finally, the lower dimension subtensor features extracted by the MPCA and MLDA algorithms are classified with a neural network (NN) classifier. The classification scheme is accessed using multi-band polarimetric SAR images (C-, L- and P-band) acquired by the Airborne Synthetic Aperture Radar (AIRSAR) sensor of the Jet Propulsion Laboratory (JPL) over the Flevoland area. Experimental results demonstrate that the proposed method has good classification performance in comparison with the classic multi-band Wishart classifier. The overall classification accuracy is close to 99%, even when the number of training samples is small.

Pareto models for discriminative multiclass linear dimensionality reduction

We address the class masking problem in multiclass linear discriminant analysis (LDA). In the multiclass setting, LDA does not maximize each pairwise distance between classes, but rather maximizes the sum of all pairwise distances. This results in serious overlaps between classes that are close to each other in the input space, and degrades classification performance. Our research proposes Pareto Discriminant Analysis (PARDA); an approach for multiclass discriminative analysis that builds over multiobjective optimizing models. PARDA decomposes the multiclass problem to a set of objective functions, each representing the distance between every pair of classes. Unlike existing LDA extensions that maximize the sum of all distances, PARDA maximizes each pairwise distance to maximally separate all class means, while minimizing the class overlap in the lower dimensional space. Experimental results on various data sets show consistent and promising performance of PARDA when compared with well-known multiclass LDA extensions.

Multiple-instance discriminant analysis

Multiple-instance discriminant analysis (MIDA) is proposed to cope with the feature extraction problem in multiple-instance learning. Similar to MidLABS, MIDA is also derived from linear discriminant analysis (LDA), and both algorithms can be treated as multiple-instance extensions of LDA. Different from MidLABS which learns from the bag level, MIDA is designed from the instance level. MIDA consists of two versions, i.e., binary-class MIDA (B-MIDA) and multi-class MIDA (M-MIDA), which are utilized to cope with binary-class (standard) and multi-class multiple-instance learning tasks, respectively. The block coordinate ascent approach, by which we seek positive prototypes (the most positive instance in a positive bag is termed as the positive prototype of this bag) and projection vectors alternatively and iteratively, is proposed to optimize B-MIDA and M-MIDA to obtain lower dimensional transformation subspaces. Extensive experiments empirically demonstrate the effectiveness of B-MIDA and M-MIDA in extracting discriminative components and weakening class-label ambiguities for instances in positive bags.

Using geovisual analytics to compare the performance of geographically weighted discriminant analysis versus its global counterpart, linear discriminant analysis

Geographically weighted spatial statistical methods are a family of spatial statistical methods developed to address the presence of non-stationarity in geographical processes, the so-called spatial heterogeneity. While these methods have recently become popular for analysis of spatial data, one of their characteristics is that they produce outputs that in themselves form complex multi-dimensional spatial data sets. Interpretation of these outputs is therefore not easy, but is of high importance, since spatial and non-spatial patterns in the results of these methods contain clues to causes of underlying non-stationarity. In this article, we focus on one of the geographically weighted methods, the geographically weighted discriminant analysis (GWDA), which is a method for prediction and analysis of categorical spatial data. It is an extension of linear discriminant analysis (LDA) that allows the relationship between the predictor variables and the categories to vary spatially. This produces a very complex data set of GWDA results, which include on top of the already complex discriminant analysis outputs (e.g. classifications and posterior probabilities) also spatially varying outputs (e.g. classification function parameters). In this article, we suggest using geovisual analytics to visualise results from LDA and GWDA to facilitate comparison between the global and local method results. For this, we develop a bespoke visual methodology that allows us to examine the performance of global and local classification method in terms of quality of classification. Furthermore, we are also interested in identifying the presence (or absence) of non-stationarity through comparison of the outputs of both methods. We do this in two ways. First, we visually explore spatial autocorrelation in both LDA and GWDA misclassifications. Second, we focus on relationships between the classification result and the independent variables and how they vary over space. We describe our visual analytic system for exploration of LDA and GWDA outputs and demonstrate our approach on a case study using a data set linking election results with a selection of socio-economic variables.

Simple tiered classifiers

In this paper we propose simple, general tiered classifiers for relatively complex data. Empirical studies on real and simulated data show that three two-tier classifiers, which are respective extensions of linear discriminant analysis, linear logistic regression and support vector machines, can reduce noticeably the relatively high misclassification error of their original single-tier counterparts, without significantly increasing computational labour. Copyright 2013, Oxford University Press.

라그랑지 기법을 쓴 영 공간 기반 선형 판별 분석법의 변형 기법

Due to the singularity of the within-class scatter, linear discriminant analysis (LDA) becomes ill-posed for small sample size (SSS) problems. An extension of LDA, the null space-based LDA (NLDA) provides good discriminant performances for SSS problems. In this paper, by applying the Lagrange technique, the procedure of transforming the problem of finding the feature extractor of NLDA into a linear equation problem is derived.

Fuzzy Linear Discriminant Analysis-guided maximum entropy fuzzy clustering algorithm

Linear Discriminant Analysis (LDA) is a classical statistical approach for supervised feature extraction and dimensionality reduction, hard c-means (HCM) is a classical unsupervised learning algorithm for clustering. Based on the analysis of the relationship between LDA and HCM, Linear Discriminant Analysis-guided adaptive subspace hard c-means clustering algorithm (LDA–HCM) had been proposed. LDA–HCM combines LDA and HCM into a coherent framework and can adaptively reduce the dimension of data while performing data clustering simultaneously. Seeing that LDA–HCM is still a hard clustering algorithm, we consider the fuzzy extension version of LDA–HCM in this paper. To this end, firstly, we propose a new optimization criterion of Fuzzy Linear Discriminant Analysis (FLDA) by extending the value of membership function in classical LDA from binary 0 or 1 into closed interval [0, 1]. In the meantime, we present an efficient algorithm for the proposed FLDA. Secondly, we show the close relationship between FLDA and Maximum Entropy Fuzzy Clustering Algorithm (MEFCA): they both are maximizing fuzzy between-class scatter and minimizing within-class scatter simultaneously. Finally, based on the above analysis, combining FLDA and MEFCA into a joint framework, we propose fuzzy Linear Discriminant Analysis-guided maximum entropy fuzzy clustering algorithm (FLDA–MEFCA). LDA–MEFCA is a natural and effective fuzzy extension of LDA–HCM. Due to the introduction of soft decision strategy, FLDA–MEFCA can yield fuzzy partition of data set and is more flexible than LDA–HCM. We also give the convergence proof of FLDA–MEFCA. Extensive experiments on a collection of benchmark data sets are presented to show the effectiveness of the proposed algorithm.

Modified Fuzzy Linear Discriminant Analysis for Threshold Selection

Otsu’s thresholding method is a popular and efficient method for image segmentation. However, its performance is greatly affected by noise and the population size of object and background. In this paper, a novel thresholding method is proposed based on modified fuzzy linear discriminant analysis (MFLDA). MFLDA is an extension of linear discriminant analysis to fuzzy domain, where the between-class variance is modified as the distance between the centers of background and object. The optimal threshold is selected such that the MFLDA criterion is maximized. Some images are used to test the performance of the proposed thresholding method and results reveal that the proposed method is less affected by noise, the population size of objects and background, and better segmentation results are obtained than Otsu’s method and other classical thresholding methods.

Complexity-reduced scheme for feature extraction with linear discriminant analysis.

Owing to the singularity of the within-class scatter, linear discriminant analysis (LDA) becomes ill-posed for small sample size (SSS) problems. Null-space-based LDA (NLDA), which is an extension of LDA, provides good discriminant performances for SSS problems. Yet, as the original scheme for the feature extractor (FE) of NLDA suffers from a complexity burden, a few modified schemes have since been proposed for complexity reduction. In this brief, by transforming the problem of finding the FE of NLDA into a linear equation problem, a novel scheme is derived, offering a further reduction of the complexity.

Multi-Task Learning in Heterogeneous Feature Spaces

Multi-task learning aims at improving the generalization performance of a learning task with the help of some other related tasks. Although many multi-task learning methods have been proposed, they are all based on the assumption that all tasks share the same data representation. This assumption is too restrictive for general applications. In this paper, we propose a multi-task extension of linear discriminant analysis (LDA), called multi-task discriminant analysis (MTDA), which can deal with learning tasks with different data representations. For each task, MTDA learns a separate transformation which consists of two parts, one specific to the task and one common to all tasks. A by-product of MTDA is that it can alleviate the labeled data deficiency problem of LDA. Moreover, unlike many existing multi-task learning methods, MTDA can handle binary and multi-class problems for each task in a generic way. Experimental results on face recognition show that MTDA consistently outperforms related methods.