Discriminative Subspace Research Articles

Multi-Branch Tensor Network Structure for Tensor-Train Discriminant Analysis.

Higher-order data with high dimensionality arise in a diverse set of application areas such as computer vision, video analytics and medical imaging. Tensors provide a natural tool for representing these types of data. Two major challenges that confound current tensor based supervised learning algorithms are storage complexity and computational efficiency. In this paper, we address these problems by introducing a multi-branch tensor network structure. The multi-branch structure is a general tensor decomposition that includes Tucker and tensor-train (TT) as special cases and takes advantage of the flexibility of the tensor network to provide a better balance between storage and computational complexity. We then introduce a supervised discriminative tensor-train subspace learning approach referred to as tensor-train discriminant analysis (TTDA), and its implementations using the multi-branch tensor network structure. Multi-branch implementations of TTDA are shown to achieve lower storage and computational complexity while providing improved classification performance with respect to both Tucker and TT based supervised learning methods.

Distributed Adaptive Trace Ratio Optimization in Wireless Sensor Networks

The trace ratio optimization (TRO) problem consists of finding an orthonormal basis for the discriminative subspace that maximizes the ratio of two trace operators on two covariance matrices corresponding to two distinctive classes or signal components. The TRO problem is encountered in various signal processing problems such as dimensionality reduction, signal enhancement, and discriminative analysis. In this paper, we propose a distributed and adaptive algorithm for solving the TRO problem in the context of wireless sensor networks (WSNs), where the two matrices involved in the trace ratio operators correspond to the (unknown) spatial correlation of the sensor signals across the nodes in the network. We first focus on fully-connected networks where every node can communicate with each other, but only compressed signals observations can be shared to reduce the communication cost. After showing convergence, we modify the algorithm to operate in WSNs with more general topologies. Simulation results are provided to validate and complement the theoretical results.

Multilinear Discriminative Spatial Patterns for Movement-Related Cortical Potential Based on EEG Classification with Tensor Representation.

The discriminative spatial patterns (DSP) algorithm is a classical and effective feature extraction technique for decoding of voluntary finger premovements from electroencephalography (EEG). As a purely data-driven subspace learning algorithm, DSP essentially is a spatial-domain filter and does not make full use of the information in frequency domain. The paper presents multilinear discriminative spatial patterns (MDSP) to derive multiple interrelated lower dimensional discriminative subspaces of low frequency movement-related cortical potential (MRCP). Experimental results on two finger movement tasks' EEG datasets demonstrate the effectiveness of the proposed MDSP method.

Unsupervised Domain Adaptation Based on Discriminative Subspace Learning for Cross-Project Defect Prediction

Unsupervised Domain Adaptation Based on Discriminative Subspace Learning for Cross-Project Defect Prediction

A novel acoustic scene classification model using the late fusion of convolutional neural networks and different ensemble classifiers

Recent evidence suggests that convolutional neural networks (CNNs) can model acoustic scene classification (ASC) with high accuracy. Ensemble classifiers have also shown high accuracy in different machine learning areas. However, little is known about fusion models between CNNs and different ensemble classifiers for ASC. This study presents an enhanced CNN classification model using the late fusion between CNNs and ensemble classifiers to predict different classes of acoustic scenes. A CNN model was first built to classify fifteen acoustic scene environments. Different ensemble classifier models were then used for this classification problem. Late fusion of CNN and ensemble classifier models was then applied. The results showed that late fusion models have higher classification accuracy, as compared to individual CNN or ensemble classifier models. The best model was obtained by fusion of the CNN and discriminant random subspace classifier with an increase in the average accuracy of 10% as compared to the average accuracy of the CNN model. When compared with previous research on ASC, the late fusion model between CNN and ensemble classifiers showed higher accuracy. Therefore, this method has robust applicability for future ASC problems.

Spatio-temporal multi-factor model for individual identification from biological motion

Despite the promising progress made in recent years, individual identification from biological motion remains a challenging task due to the drastic changes in human poses produced by various covariate factors. A variety of algorithms have been proposed to solve this challenging problem by extracting the invariant features of individuals. There exist two main bottlenecks for these methods. One is the large variability within each class; the other is that few methods have taken the spatio-temporal structure of biological motion into consideration. To address these problems, we propose a novel multi-factor spatio-temporal paradigm for individual identification that capture the entire spatio-temporal structure of the 3D biological motion. The first proposed approach, namely spatio-temporal multi-factor discriminant analysis (ST-MFDA), learns a set of paired factor-specific spatio-temporal projections to map motion features from different factors into a common discriminant subspace. Then, in the common subspace, features from the same individual are united and those from different individuals are separated by Fisher Discriminant Criterion. We prove that spatio-temporal projections of various factors can be jointly learnt by solving an iterative generalized eigenvalue problem. Beyond that, a bilinear classifier strategy is also incorporated into the proposed paradigm to resolve the influence of covariant factors. Similar to ST-MFDA, the multi-factor bilinear classifier can map the features from different factors into a common subspace, so that an unbiased classifier can be learned in the common space. Experimental results from four public databases demonstrate the effectiveness of the proposed approaches, and we find that the spatio-temporal projections trained in this way lead to significant improvements in identification accuracy.

Multi-view Incremental Discriminant Analysis

In recent years, the use of multi-view data has attracted much attention resulting in many multi-view batch learning algorithms. However, these algorithms prove expensive in terms of training time and memory when used on the incremental data. In this paper, we propose Multi-view Incremental Discriminant Analysis (MvIDA), which updates the trained model to incorporate new data samples. MvIDA requires only the old model and newly added data to update the model. Depending on the nature of the increments, MvIDA is presented as two cases, sequential MvIDA and chunk MvIDA. We have compared the proposed method against the batch Multi-view Discriminant Analysis (MvDA) for its discriminability, order independence, the effect of the number of views, training time, and memory requirements. We have also compared our method with single-view Incremental Linear Discriminant Analysis (ILDA) for accuracy and training time. The experiments are conducted on four datasets with a wide range of dimensions per view. The results show that through order independence and faster construction of the optimal discriminant subspace, MvIDA addresses the issues faced by the batch multi-view algorithms in the incremental setting.

Improving the performance of drifted/shifted electronic nose systems by cross-domain transfer using common transfer samples

Sensor drift/shift is a challenging and high-profile issue in the field of sensors and measurements. Because of the time variability and unpredictable properties of drift/shift, traditional compensation methods are costly and laborious. Considering the distribution alignments of different domains at both the feature and decision level, a unified double-level drift/shift compensation framework is presented to address sensor drift/shift. The framework consists of cross-domain feature subspace transfer learning coupled with a cross-domain adaptation extreme learning machine (CFST-AELM), which learns a low-dimensional discriminant subspace and a robust classifier using small numbers of labeled samples selected from the target domain. In general, the unified framework has the following advantages: (1) implementing anti-drift/shift at both the feature and decision level by cross-domain transfer learning which takes full advantage of transfer samples; (2) the subspace learning is a joint distribution adaptation algorithm and can be easily realized by eigenvalue decomposition; (3) more discriminative information of the source and target datasets remain after linear mapping; and (4) there is enhancement of the generalization performance of the model by adding the prediction error of transfer samples as a constraint of extreme learning machines (ELM) classification. The results suggest that compared with several state-of-the-art strategies, the CFST-AELM framework has superior performance in terms of classification accuracy.

Bio-Inspired Structure Representation Based Cross-View Discriminative Subspace Learning via Simultaneous Local and Global Alignment

Recently, cross-view feature learning has been a hot topic in machine learning due to the wide applications of multiview data. Nevertheless, the distribution discrepancy between cross-views leads to the fact that instances of the different views from same class are farther than those within the same view but from different classes. To address this problem, in this paper, we develop a novel cross-view discriminative feature subspace learning method inspired by layered visual perception from human. Firstly, the proposed method utilizes a separable low-rank self-representation model to disentangle the class and view structure layers, respectively. Secondly, a local alignment is constructed with two designed graphs to guide the subspace decomposition in a pairwise way. Finally, the global discriminative constraint on distribution center in each view is designed for further alignment improvement. Extensive cross-view classification experiments on several public datasets prove that our proposed method is more effective than other existing feature learning methods.

DMIB: Dual-Correlated Multivariate Information Bottleneck for Multiview Clustering.

Multiview clustering (MVC) has recently been the focus of much attention due to its ability to partition data from multiple views via view correlations. However, most MVC methods only learn either interfeature correlations or intercluster correlations, which may lead to unsatisfactory clustering performance. To address this issue, we propose a novel dual-correlated multivariate information bottleneck (DMIB) method for MVC. DMIB is able to explore both interfeature correlations (the relationship among multiple distinct feature representations from different views) and intercluster correlations (the close agreement among clustering results obtained from individual views). For the former, we integrate both view-shared feature correlations discovered by learning a shared discriminative feature subspace and view-specific feature information to fully explore the interfeature correlation. This allows us to attain multiple reliable local clustering results of different views. Following this, we explore the intercluster correlations by learning the shared mutual information over different local clusterings for an improved global partition. By integrating both correlations, we formulate the problem as a unified information maximization function and further design a two-step method for optimization. Moreover, we theoretically prove the convergence of the proposed algorithm, and discuss the relationships between our method and several existing clustering paradigms. The experimental results on multiple datasets demonstrate the superiority of DMIB compared to several state-of-the-art clustering methods.

Complex Chemical Data Classification and Discrimination Using Locality Preserving Partial Least Squares Discriminant Analysis.

Partial least squares discriminant analysis (PLS-DA) is a well-known technique for feature extraction and discriminant analysis in chemometrics. Despite its popularity, it has been observed that PLS-DA does not automatically lead to extraction of relevant features. Feature learning and extraction depends on how well the discriminant subspace is captured. In this paper, discriminant subspace learning of chemical data is discussed from the perspective of PLS-DA and a recent extension of PLS-DA, which is known as the locality preserving partial least squares discriminant analysis (LPPLS-DA). The objective is twofold: (a) to introduce the LPPLS-DA algorithm to the chemometrics community and (b) to demonstrate the superior discrimination capabilities of LPPLS-DA and how it can be a powerful alternative to PLS-DA. Four chemical data sets are used: three spectroscopic data sets and one that contains compositional data. Comparative performances are measured based on discrimination and classification of these data sets. To compare the classification performances, the data samples are projected onto the PLS-DA and LPPLS-DA subspaces, and classification of the projected samples into one of the different groups (classes) is done using the nearest-neighbor classifier. We also compare the two techniques in data visualization (discrimination) task. The ability of LPPLS-DA to group samples from the same class while at the same time maximizing the between-class separation is clearly shown in our results. In comparison with PLS-DA, separation of data in the projected LPPLS-DA subspace is more well defined.

Trace Ratio Optimization for High-Dimensional Multi-Class Discrimination

In multi-class discrimination with high-dimensional data, identifying a lower-dimensional subspace with maximum class separation is crucial. We propose a new optimization criterion for finding such a discriminant subspace, which is the ratio of two traces: the trace of between-class scatter matrix and the trace of within-class scatter matrix. Since this problem is not well-defined for high-dimensional data, we propose to regularize the within trace and maximize the between trace. A careful investigation reveals that this optimization has an innate connection to the eigenvalue decomposition of an indefinite matrix. For the sake of better interpretability of the classifier, we also consider a sparse estimation via a group-wise soft-thresholding. Interesting relationships between the proposed method and some classical methods such as Fisher’s linear discriminant analysis and its variants are discussed. Empirical examples with simulated and real datasets suggest that the proposed method works well and is often better than some existing approaches in a wide range of problems, with respect to both variable selectivity and classification accuracy. Supplementary files for this article are available online.

Infinite Bayesian Max-Margin Discriminant Projection.

In this article, considering the supervised dimensionality reduction, we first propose a model, called infinite Bayesian max-margin linear discriminant projection (iMMLDP), by assembling a set of local regions, where we make use of Bayesian nonparametric priors to handle the model selection problem, for example, the underlying number of local regions. In each local region, our model jointly learns a discriminative subspace and the corresponding classifier. Under this framework, iMMLDP combines dimensionality reduction, clustering, and classification in a principled way. Moreover, to deal with more complex data, for example, a local nonlinear separable structure, we extend the linear projection to a nonlinear case based on the kernel trick and develop an infinite kernel max-margin discriminant projection (iKMMDP) model. Thanks to the conjugate property, the parameters in these two models can be inferred efficiently via the Gibbs sampler. Finally, we implement our models on synthesized and real-world data, including multimodally distributed datasets and measured radar image data, to validate their efficiency and effectiveness.

Discriminative subspace matrix factorization for multiview data clustering

In a real-world scenario, an object is easily considered as features combined by multiple views in reality. Thus, multiview features can be encoded into a unified and discriminative framework to achieve satisfactory clustering performance. An increasing number of algorithms have been proposed for multiview data clustering. However, existing multiview methods have several drawbacks. First, most multiview algorithms focus only on origin data in high dimension directly without the intrinsic structure in the relative low-dimensional subspace. Spectral and manifold-based methods ignore pseudo-information that can be extracted from the optimization process. Thus, we design an unsupervised nonnegative matrix factorization (NMF)-based method called discriminative multiview subspace matrix factorization (DMSMF) for clustering. We provide the following contributions. (1) We extend linear discriminant analysis and NMF to a multiview version and connect them to a unified framework to learn in the discriminant subspace. (2) We propose a multiview manifold regularization term and discriminant multiview manifold regularization term that instruct the regularization term to discriminate different classes and obtain the geometry st ructure from the low-dimensional subspace. (3) We design an effective optimization algorithm with proven convergence to obtain an optimal solution procedure for the complex model. Adequate experiments are conducted on multiple benchmark datasets. Finally, we demonstrate that our model is superior to other comparable multiview data clustering algorithms.

Analysis of Regression Methods for Face Recognition System

Face verification has recently drawn a lot of attention as one of the best applications of image assessment and perception, particularly over the course of the last significant amount of time. Two factors, namely the wide range of corporate and legal approved applications and the openness of accessible headways after 30 years of research, speak to this example. Despite the fact that existing machine affirmation structures have made some progress, their success is still limited by the requirements of many real-world applications. For instance, the verification of facial images captured in an outside setting with variations in lighting or possibly position isstill a very difficult problem. This paper present a analysis for face recognition method in comparison with PCA and LDA and suggests that LCDRC finds a discriminant subspace by maximizing collaborative between-class reconstruction error and minimizing the within-class reconstruction error simultaneously.

Spectrum-aware discriminative deep feature learning for multi-spectral face recognition

One primary challenge of face recognition is that the performance is seriously affected by varying illumination. Multi-spectral imaging can capture face images in the visible spectrum and beyond, which is deemed to be an effective technology in response to this challenge. For current multi-spectral imaging-based face recognition methods, how to fully explore the discriminant and correlation features from both the intra-spectrum and inter-spectrum aspects with only a limited number of multi-spectral samples for model training has not been well studied. To address this problem, in this paper, we propose a novel face recognition approach named Spectrum-aware Discriminative Deep Learning (SDDL). To take full advantage of the multi-spectral training samples, we build a discriminative multi-spectral network (DMN) and take face sample pairs as the input of the network. By jointly considering the spectrum and the class label information, SDDL trains the network for projecting samples pairs into a discriminant feature subspace, on which the intrinsic relationship including the intra- and inter-spectrum discrimination and the inter-spectrum correlation among face samples is well discovered. The proposed approach is evaluated on three widely used datasets HK PolyU, CMU, and UWA. Extensive experimental results demonstrate the superiority of SDDL over state-of-the-art competing methods.

Supervised discriminative dimensionality reduction by learning multiple transformation operators

Analyzing and learning from high dimensional data have always been challenging in machine learning tasks, causing serious computational complexities and poor learning performances. Supervised dimensionality reduction is a popular technique to address such challenges in supervised learning tasks, where data are accompanied with labels. Traditionally, such techniques mostly learn one single transformation to project data into a low-dimensional discriminative subspace. However, learning only one transformation for the whole data could be dominated by one or several classes, and the rest of classes receive less discrimination in the reduced space. That is to say, learning one transformation is insufficient to properly discriminate classes of data in the reduced space because they may have complex and completely dissimilar distributions. This insufficiency becomes even more serious if the number of classes increases, leading to poor discrimination and lessening the learning performance in the reduced space. To overcome this limitation, we propose a novel supervised dimensionality reduction method, which learns per-class transformations by optimizing a newly designed and efficient objective function. The proposed method captures more discriminative information from each single class of data compared to the case of one single transformation. Moreover, the proposed objective function enjoys several desirable properties: (1) maximizing margins between the transformed classes of data, (2) having a closed-form solution, (3) being easily kernelized in the case of nonlinear data,(4) preventing overfitting, and (5) ensuring the transformations are sparse in rows so that discriminative features are learned in the reduced space. Experimental results verify that the proposed method is superior to the related state-of-the-art methods and promising in generating discriminative embeddings.

Semi-Supervised Subspace Learning for Pattern Classification via Robust Low Rank Constraint

Feature subspace learning is a crucial issue in pattern analysis. However, it remains challenging when partial samples are unlabeled, which will cause weak discrimination. In this paper, we present a novel semi-supervised learning model that is capable of utilizing labeled and unlabeled training data simultaneously to learn discriminative feature subspace while preserving their locality. To achieve this goal, we joint learning feature subspace and completed labels. In the framework, low rank representation model is firstly exploited to explore the similarity relationship among all training samples, including labeled and unlabeled data. Then, the learned representation coefficients are used to generate a dynamic neighbor graph for designing the locality preservation constraints on both of label propagation and feature subspace. Finally, the prediction and true label are used to enforce the discrimination of feature subspace in the semantic space. Extensive experiment indicates that our proposed approach is more competitive than other comparison methods, while the model shows more robustness when training datasets are contaminated with noise.

Fast Multi-View Outlier Detection via Deep Encoder

Multi-view outlier detection has a wide range of applications and has been well investigated in recent years. However, 1) most existing state-of-the-art methods cannot efficiently handle outlier detection problem for large-scale multi-view data, since exploring pairwise constraints among different views causes highly-computational cost; 2) the data collected from original heterogeneous feature spaces further increases the consistent difficulty of multi-view outlier detection. To address these issues, we present a fast multi-view outlier detection model via learning a low-rank latent subspace representation with deep encoder architecture, which can not only efficiently identify the outliers for large-scale data even with numerous data views, but also exploit a discriminative common latent subspace shared by all the views. First, we learn a set of orthogonal bases as view-specific dictionaries from a small dataset, which is randomly sampled from the original dataset. Benefitting from view-specific dictionaries, the sampled data is projected and decomposed as a shared and discriminative latent subspace representations, which correspond to the view-consistent and view-specific components across multiple views, respectively. Then, the obtained discriminative latent representations are applied to train the view-specific deep encoders, which can efficiently compute the abnormal score for the remaining instances. Our proposed model can cost-effectively identify the outliers in large-scale datasets from numerous data views with less computational complexity. Experiments conducted on eight real datasets and a synthesis dataset show that our proposed model outperforms the existing ones on effectiveness and efficiency.

Discriminant Geometrical and Statistical Alignment With Density Peaks for Domain Adaptation.

Unsupervised domain adaptation (DA) aims to perform classification tasks on the target domain by leveraging rich labeled data in the existing source domain. The key insight of DA is to reduce domain divergence by learning domain-invariant features or transferable instances. Despite its rapid development, there still exist several challenges to explore. At the feature level, aligning both domains only in a single way (i.e., geometrical or statistical) has limited ability to reduce the domain divergence. At the instance level, interfering instances often obstruct learning a discriminant subspace when performing the geometrical alignment. At the classifier level, only minimizing the empirical risk on the source domain may result in a negative transfer. To tackle these challenges, this article proposes a novel DA method, called discriminant geometrical and statistical alignment (DGSA). DGSA first aligns the geometrical structure of both domains by projecting original space into a Grassmann manifold, then matches the statistical distributions of both domains by minimizing their maximum mean discrepancy on the manifold. In the former step, DGSA only selects the density peaks to learn the Grassmann manifold and so to reduce the influences of interfering instances. In addition, DGSA exploits the high-confidence soft labels of target landmarks to learn a more discriminant manifold. In the latter step, a structural risk minimization (SRM) classifier is learned to match the distributions (both marginal and conditional) and predict the target labels at the same time. Extensive experiments on objection recognition and human activity recognition tasks demonstrate that DGSA can achieve better performance than the comparison methods.