Discriminative Subspace Research Articles

A subspace co-training framework for multi-view clustering

This paper addresses the problem of unsupervised clustering with multi-view data of high dimensionality. We propose a new algorithm which learns discriminative subspaces in an unsupervised fashion based upon the assumption that a reliable clustering should assign same-class samples to the same cluster in each view. The framework combines the simplicity of k-means clustering and Linear Discriminant Analysis (LDA) within a co-training scheme which exploits labels learned automatically in one view to learn discriminative subspaces in another. The effectiveness of the proposed algorithm is demonstrated empirically under scenarios where the conditional independence assumption is either fully satisfied (audio-visual speaker clustering) or only partially satisfied (handwritten digit clustering and document clustering). Significant improvements over alternative multi-view clustering approaches are reported in both cases. The new algorithm is flexible and can be readily adapted to use different distance measures, semi-supervised learning, and non-linear problems.

Fault diagnosis of rolling element bearings via discriminative subspace learning: Visualization and classification

Rolling element bearings play an important role in ensuring the availability of industrial machines. Unexpected bearing failures in such machines during field operation can lead to machine breakdown, which may have some pretty severe implications. To address such concern, we extend our algorithm for solving trace ratio problem in linear discriminant analysis to diagnose faulty bearings in this paper. Our algorithm is validated by comparison with other state-of art methods based on a UCI data set, and then be extended to rolling element bearing data. Through the construction of feature data set from sensor-based vibration signals of bearing, the fault diagnosis problem is solved as a pattern classification and recognition way. The two-dimensional visualization and classification accuracy of bearing data show that our algorithm is able to recognize different bearing fault categories effectively. Thus, it can be considered as a promising method for fault diagnosis.

Supervised orthogonal discriminant subspace projects learning for face recognition

In this paper, a new linear dimension reduction method called supervised orthogonal discriminant subspace projection (SODSP) is proposed, which addresses high-dimensionality of data and the small sample size problem. More specifically, given a set of data points in the ambient space, a novel weight matrix that describes the relationship between the data points is first built. And in order to model the manifold structure, the class information is incorporated into the weight matrix. Based on the novel weight matrix, the local scatter matrix as well as non-local scatter matrix is defined such that the neighborhood structure can be preserved. In order to enhance the recognition ability, we impose an orthogonal constraint into a graph-based maximum margin analysis, seeking to find a projection that maximizes the difference, rather than the ratio between the non-local scatter and the local scatter. In this way, SODSP naturally avoids the singularity problem. Further, we develop an efficient and stable algorithm for implementing SODSP, especially, on high-dimensional data set. Moreover, the theoretical analysis shows that LPP is a special instance of SODSP by imposing some constraints. Experiments on the ORL, Yale, Extended Yale face database B and FERET face database are performed to test and evaluate the proposed algorithm. The results demonstrate the effectiveness of SODSP.

Multidimensional Sequence Classification Based on Fuzzy Distances and Discriminant Analysis

In this paper, we present a novel method aiming at multidimensional sequence classification. We propose a novel sequence representation, based on its fuzzy distances from optimal representative signal instances, called statemes. We also propose a novel modified clustering discriminant analysis algorithm minimizing the adopted criterion with respect to both the data projection matrix and the class representation, leading to the optimal discriminant sequence class representation in a low-dimensional space, respectively. Based on this representation, simple classification algorithms, such as the nearest subclass centroid, provide high classification accuracy. A three step iterative optimization procedure for choosing statemes, optimal discriminant subspace and optimal sequence class representation in the final decision space is proposed. The classification procedure is fast and accurate. The proposed method has been tested on a wide variety of multidimensional sequence classification problems, including handwritten character recognition, time series classification and human activity recognition, providing very satisfactory classification results.

Efficient Mining of Discriminative Co-clusters from Gene Expression Data.

Discriminative models are used to analyze the differences between two classes and to identify class-specific patterns. Most of the existing discriminative models depend on using the entire feature space to compute the discriminative patterns for each class. Co-clustering has been proposed to capture the patterns that are correlated in a subset of features, but it cannot handle discriminative patterns in labeled datasets. In certain biological applications such as gene expression analysis, it is critical to consider the discriminative patterns that are correlated only in a subset of the feature space. The objective of this paper is two-fold: first, it presents an algorithm to efficiently find arbitrarily positioned co-clusters from complex data. Second, it extends this co-clustering algorithm to discover discriminative co-clusters by incorporating the class information into the co-cluster search process. In addition, we also characterize the discriminative co-clusters and propose three novel measures that can be used to evaluate the performance of any discriminative subspace pattern mining algorithm. We evaluated the proposed algorithms on several synthetic and real gene expression datasets, and our experimental results showed that the proposed algorithms outperformed several existing algorithms available in the literature.

Discriminative subspace learning with sparse representation view-based model for robust visual tracking

In this paper, we propose a robust tracking algorithm to handle drifting problem. This algorithm consists of two parts: the first part is the G&D part that combines Generative model and Discriminative model for tracking, and the second part is the View-Based model for target appearance that corrects the result of the G&D part if necessary. In G&D part, we use the Maximum Margin Projection (MMP) to construct a graph model to preserve both local geometrical and discriminant structures of the data manifold in low dimensions. Therefore, such discriminative subspace combined with traditional generative subspace can benefit from both models. In addition, we address the problem of learning maximum margin projection under the Spectral Regression (SR) which results in significant savings in computational time. To further solve the drift, an online learned sparsely represented view-based model of the target is complementary to the G&D part. When the result of G&D part is unreliable, the view-based model can rectify the result in order to avoid drifting. Experimental results on several challenging video sequences demonstrate the effectiveness and robustness of our approach.

Discriminative variable selection for clustering with the sparse Fisher-EM algorithm

The interest in variable selection for clustering has increased recently due to the growing need in clustering high-dimensional data. Variable selection allows in particular to ease both the clustering and the interpretation of the results. Existing approaches have demonstrated the importance of variable selection for clustering but turn out to be either very time consuming or not sparse enough in high-dimensional spaces. This work proposes to perform a selection of the discriminative variables by introducing sparsity in the loading matrix of the Fisher-EM algorithm. This clustering method has been recently proposed for the simultaneous visualization and clustering of high-dimensional data. It is based on a latent mixture model which fits the data into a low-dimensional discriminative subspace. Three different approaches are proposed in this work to introduce sparsity in the orientation matrix of the discriminative subspace through $$\ell _{1}$$ l 1 -type penalizations. Experimental comparisons with existing approaches on simulated and real-world data sets demonstrate the interest of the proposed methodology. An application to the segmentation of hyperspectral images of the planet Mars is also presented.

Incremental min–max projection analysis for classification

For data classification, the standard implementation of projection algorithms do not scale well with large dataset size. It makes the computation of large samples infeasible. In this paper, we utilize a block optimization strategy to propose a new locally discriminant projection algorithm termed min–max projection analysis (MMPA). The algorithm takes into account both intra-class and interclass geometries and also possesses the orthogonality property. Furthermore, an incremental MMPA is proposed to learn the local discriminant subspace with newly inserted data by employing the idea of singular value decomposition updating algorithm. Moreover, we extend MMPA to the semi-supervised case and nonlinear case, namely, semi-supervised MMPA and kernel MMPA. The experimental results on image database, hand written digit database, and face database demonstrate the effectiveness of those proposed algorithms.

Heterogeneous Face Recognition Using Kernel Prototype Similarities

Heterogeneous face recognition (HFR) involves matching two face images from alternate imaging modalities, such as an infrared image to a photograph or a sketch to a photograph. Accurate HFR systems are of great value in various applications (e.g., forensics and surveillance), where the gallery databases are populated with photographs (e.g., mug shot or passport photographs) but the probe images are often limited to some alternate modality. A generic HFR framework is proposed in which both probe and gallery images are represented in terms of nonlinear similarities to a collection of prototype face images. The prototype subjects (i.e., the training set) have an image in each modality (probe and gallery), and the similarity of an image is measured against the prototype images from the corresponding modality. The accuracy of this nonlinear prototype representation is improved by projecting the features into a linear discriminant subspace. Random sampling is introduced into the HFR framework to better handle challenges arising from the small sample size problem. The merits of the proposed approach, called prototype random subspace (P-RS), are demonstrated on four different heterogeneous scenarios: 1) near infrared (NIR) to photograph, 2) thermal to photograph, 3) viewed sketch to photograph, and 4) forensic sketch to photograph.

Sparse tensor discriminant analysis

The classical linear discriminant analysis has undergone great development and has recently been extended to different cases. In this paper, a novel discriminant subspace learning method called sparse tensor discriminant analysis (STDA) is proposed, which further extends the recently presented multilinear discriminant analysis to a sparse case. Through introducing the L1 and L2 norms into the objective function of STDA, we can obtain multiple interrelated sparse discriminant subspaces for feature extraction. As there are no closed-form solutions, k-mode optimization technique and the L1 norm sparse regression are combined to iteratively learn the optimal sparse discriminant subspace along different modes of the tensors. Moreover, each non-zero element in each subspace is selected from the most important variables/factors, and thus STDA has the potential to perform better than other discriminant subspace methods. Extensive experiments on face databases (Yale, FERET, and CMU PIE face databases) and the Weizmann action database show that the proposed STDA algorithm demonstrates the most competitive performance against the compared tensor-based methods, particularly in small sample sizes.

HDLSS Discrimination With Adaptive Data Piling

We propose new discrimination methods for classification of high dimension, low sample size (HDLSS) data that regularize the degree of data piling. The within-class scatter of the HDLSS data, when projected onto a low-dimensional discriminant subspace, can be selected to be arbitrarily small. Using this fact, we develop two different ways of tuning the amount of within-class scatter, or equivalently, the degree of data piling. In the first approach, we consider a linear path connecting the maximal data piling and the least data piling directions. We also formulate a problem of finding the optimal classifier under a constraint on data piling. The data piling regularization methods are extended to multicategory problems. Simulated and real data examples show competitive performances of the proposed classification methods. Supplementary materials for this article are available online on the journal web site.

An improvement of incremental recursive fisher linear discriminant for online feature extraction

This paper proposes a new online feature extraction method called the Incremental Recursive Fisher Linear Discriminant (IRFLD), whose batch learning algorithm, referred to as RFLD, was proposed by Xiang and colleagues. In the conventional Linear Discriminant Analysis (LDA), the number of discriminant vectors is limited to the number of classes minus one due to the rank of the between-class covariance matrix. However, RFLD and the proposed IRFLD can break this limit; that is, an arbitrary number of discriminant vectors can be obtained. In the proposed IRFLD, the Incremental Linear Discriminant Analysis (ILDA) of Pang and colleagues is extended in such a way that effective discriminant vectors are recursively searched for in the complementary space of a conventional discriminant subspace. In addition, to estimate a suitable number of effective discriminant vectors, the classification accuracy is evaluated using the cross-validation method in an online manner. For this purpose, validation data are obtained by performing k-means clustering on incoming training data and the previous validation data. The performance of IRFLD is evaluated for 16 benchmark data sets. The experimental results show that the final classification accuracies of IRFLD are always better than those of ILDA. We also show that this performance improvement is attained by adding discriminant vectors in a complementary LDA subspace. © 2013 Wiley Periodicals, Inc. Electron Comm Jpn, 96(4): 29–40, 2013; Published online in Wiley Online Library (wileyonlinelibrary.com). DOI 10.1002/ecj.10430

Cost-Sensitive Subspace Analysis and Extensions for Face Recognition

Conventional subspace-based face recognition methods seek low-dimensional feature subspaces to achieve high classification accuracy and assume the same loss from different types of misclassification. This assumption, however, may not hold in many practical face recognition systems as different types of misclassification could lead to different losses. Motivated by this concern, this paper proposes a cost-sensitive subspace analysis approach for face recognition. Our approach uses a cost matrix specifying different costs corresponding to different types of misclassifications, into two popular and widely used discriminative subspace analysis methods and devises the cost-sensitive linear discriminant analysis (CSLDA) and cost-sensitive marginal fisher analysis (CSMFA) methods, to achieve a minimum overall recognition loss by performing recognition in these learned low-dimensional subspaces. To better exploit the complementary information from multiple features for improved face recognition, we further propose a multiview cost-sensitive subspace analysis approach by seeking a common feature subspace to fuse multiple face features to improve the recognition performance. Extensive experimental results demonstrate the effectiveness of our proposed methods.

Robust spectral regression for face recognition

Spectral regression has been an efficient and powerful tool for face recognition. However, spectral regression is sensitive to the errors incurred by inaccurate annotation and occlusion. This paper studies robust spectral regression based discriminant subspace learning from correntropy and spatially smooth structure of facial subspace. First, we formulate the robust discriminant subspace learning problem as a maximum correntropy problem, which finds the most correlation solution between spectral targets and predictions. Second, total variation (TV) regularization is imposed on the correntropy objective to learn a spatially smooth face structure. Lastly, based on the additive form of half-quadratic optimization, we cast the maximum correntropy problem into a compound regularization model, which can be efficiently optimized via an accelerated proximal gradient algorithm. Compared with iteratively reweighted least squares based methods, the proposed method can not only improve recognition rates but also reduce computational cost. Experimental results on a couple of face recognition datasets demonstrate the robustness and effectiveness of our method against inaccurate annotation and occlusion.

Face Recognition and Micro-expression Recognition Based on Discriminant Tensor Subspace Analysis Plus Extreme Learning Machine

In this paper, a novel recognition algorithm based on discriminant tensor subspace analysis (DTSA) and extreme learning machine (ELM) is introduced. DTSA treats a gray facial image as a second order tensor and adopts two-sided transformations to reduce dimensionality. One of the many advantages of DTSA is its ability to preserve the spatial structure information of the images. In order to deal with micro-expression video clips, we extend DTSA to a high-order tensor. Discriminative features are generated using DTSA to further enhance the classification performance of ELM classifier. Another notable contribution of the proposed method includes significant improvements in face and micro-expression recognition accuracy. The experimental results on the ORL, Yale, YaleB facial databases and CASME micro-expression database show the effectiveness of the proposed method.

Discriminant subspace learning constrained by locally statistical uncorrelation for face recognition

High-dimensionality of data and the small sample size problem are two significant limitations for applying subspace methods which are favored by face recognition. In this paper, a new linear dimension reduction method called locally uncorrelated discriminant projections (LUDP) is proposed, which addresses the two problems from a new aspect. More specifically, we propose a locally uncorrelated criterion, which aims to decorrelate learned discriminant factors over data locally rather than globally. It has been shown that the statistical uncorrelation criterion is an important property for reducing dimension and learning robust discriminant projection as well. However, data are always locally distributed, so it is more important to explore locally statistical uncorrelated discriminant information over data. We impose this new constraint into a graph-based maximum margin analysis, so that LUDP also characterizes the local scatter as well as nonlocal scatter, seeking to find a projection that maximizes the difference, rather than the ratio between the nonlocal scatter and the local scatter. Experiments on ORL, Yale, Extended Yale face database B and FERET face database demonstrate the effectiveness of our proposed method.

EEG-fMRI fusion on the cortical surface using Coupled Tensor-Matrix Factorization: A simulation study

EEG-fMRI fusion on the cortical surface using Coupled Tensor-Matrix Factorization: A simulation study

Mixture subclass discriminant analysis link to restricted Gaussian model and other generalizations.

In this paper, a theoretical link between mixture subclass discriminant analysis (MSDA) and a restricted Gaussian model is first presented. Then, two further discriminant analysis (DA) methods, i.e., fractional step MSDA (FSMSDA) and kernel MSDA (KMSDA) are proposed. Linking MSDA to an appropriate Gaussian model allows the derivation of a new DA method under the expectation maximization (EM) framework (EM-MSDA), which simultaneously derives the discriminant subspace and the maximum likelihood estimates. The two other proposed methods generalize MSDA in order to solve problems inherited from conventional DA. FSMSDA solves the subclass separation problem, that is, the situation in which the dimensionality of the discriminant subspace is strictly smaller than the rank of the inter-between-subclass scatter matrix. This is done by an appropriate weighting scheme and the utilization of an iterative algorithm for preserving useful discriminant directions. On the other hand, KMSDA uses the kernel trick to separate data with nonlinearly separable subclass structure. Extensive experimentation shows that the proposed methods outperform conventional MSDA and other linear discriminant analysis variants.

Automatic prostate MR image segmentation with sparse label propagation and domain-specific manifold regularization.

Automatic prostate segmentation in MR images plays an important role in prostate cancer diagnosis. However, there are two main challenges: (1) Large inter-subject prostate shape variations; (2) Inhomogeneous prostate appearance. To address these challenges, we propose a new hierarchical prostate MR segmentation method, with the main contributions lying in the following aspects: First, the most salient features are learnt from atlases based on a subclass discriminant analysis (SDA) method, which aims to find a discriminant feature subspace by simultaneously maximizing the inter-class distance and minimizing the intra-class variations. The projected features, instead of only voxel-wise intensity, will be served as anatomical signature of each voxel. Second, based on the projected features, a new multi-atlases sparse label fusion framework is proposed to estimate the prostate likelihood of each voxel in the target image from the coarse level. Third, a domain-specific semi-supervised manifold regularization method is proposed to incorporate the most reliable patient-specific information identified by the prostate likelihood map to refine the segmentation result from the fine level. Our method is evaluated on a T2 weighted prostate MR image dataset consisting of 66 patients and compared with two state-of-the-art segmentation methods. Experimental results show that our method consistently achieves the highest segmentation accuracies than other methods under comparison.

Fuzzy Linear Discriminant Analysis-guided maximum entropy fuzzy clustering algorithm

Linear Discriminant Analysis (LDA) is a classical statistical approach for supervised feature extraction and dimensionality reduction, hard c-means (HCM) is a classical unsupervised learning algorithm for clustering. Based on the analysis of the relationship between LDA and HCM, Linear Discriminant Analysis-guided adaptive subspace hard c-means clustering algorithm (LDA–HCM) had been proposed. LDA–HCM combines LDA and HCM into a coherent framework and can adaptively reduce the dimension of data while performing data clustering simultaneously. Seeing that LDA–HCM is still a hard clustering algorithm, we consider the fuzzy extension version of LDA–HCM in this paper. To this end, firstly, we propose a new optimization criterion of Fuzzy Linear Discriminant Analysis (FLDA) by extending the value of membership function in classical LDA from binary 0 or 1 into closed interval [0, 1]. In the meantime, we present an efficient algorithm for the proposed FLDA. Secondly, we show the close relationship between FLDA and Maximum Entropy Fuzzy Clustering Algorithm (MEFCA): they both are maximizing fuzzy between-class scatter and minimizing within-class scatter simultaneously. Finally, based on the above analysis, combining FLDA and MEFCA into a joint framework, we propose fuzzy Linear Discriminant Analysis-guided maximum entropy fuzzy clustering algorithm (FLDA–MEFCA). LDA–MEFCA is a natural and effective fuzzy extension of LDA–HCM. Due to the introduction of soft decision strategy, FLDA–MEFCA can yield fuzzy partition of data set and is more flexible than LDA–HCM. We also give the convergence proof of FLDA–MEFCA. Extensive experiments on a collection of benchmark data sets are presented to show the effectiveness of the proposed algorithm.