Discriminative Subspace Research Articles

Multiview nonlinear discriminant structure learning for emotion recognition

Multiview subspace learning (MSL) has been widely used in various practical applications including emotion recognition. Despite the recent progress in MSL, two challenges remain to address. First, most existing MSL methods indiscriminately utilize both helpful and defective information contained in different views. Second, the most recent methods are linear approaches that do not perform well on emotion datasets with weak linear separability. Therefore, in this study, we introduce a framework for emotion recognition: multiview nonlinear discriminant structure learning (MNDSL). MNDSL fully exploits useful information in each input through local information preservation and discriminant reconstruction (LPDR) and obtains latent subspaces using multiview discriminant latent subspace learning (MDLSL). In addition, an out-of-sample extension was introduced to satisfy the requirements of large-scale applications and obtain the projections of new samples. The proposed framework constructs interviews and intra-view-weighted connections to explore discriminant structures and preserve locality under complementarity and correlation principles. The results demonstrate the superiority of the proposed framework compared with state-of-the-art methods.

DISA tool: Discriminative and informative subspace assessment with categorical and numerical outcomes.

Pattern discovery and subspace clustering play a central role in the biological domain, supporting for instance putative regulatory module discovery from omics data for both descriptive and predictive ends. In the presence of target variables (e.g. phenotypes), regulatory patterns should further satisfy delineate discriminative power properties, well-established in the presence of categorical outcomes, yet largely disregarded for numerical outcomes, such as risk profiles and quantitative phenotypes. DISA (Discriminative and Informative Subspace Assessment), a Python software package, is proposed to evaluate patterns in the presence of numerical outcomes using well-established measures together with a novel principle able to statistically assess the correlation gain of the subspace against the overall space. Results confirm the possibility to soundly extend discriminative criteria towards numerical outcomes without the drawbacks well-associated with discretization procedures. Results from four case studies confirm the validity and relevance of the proposed methods, further unveiling critical directions for research on biotechnology and biomedicine. Availability: DISA is freely available at https://github.com/JupitersMight/DISA under the MIT license.

Generalized One-Class Learning Using Pairs of Complementary Classifiers.

One-class learning is the classic problem of fitting a model to the data for which annotations are available only for a single class. In this paper, we explore novel objectives for one-class learning, which we collectively refer to as Generalized One-class Discriminative Subspaces (GODS). Our key idea is to learn a pair of complementary classifiers to flexibly bound the one-class data distribution, where the data belongs to the positive half-space of one of the classifiers in the complementary pair and to the negative half-space of the other. To avoid redundancy while allowing non-linearity in the classifier decision surfaces, we propose to design each classifier as an orthonormal frame and seek to learn these frames via jointly optimizing for two conflicting objectives, namely: i) to minimize the distance between the two frames, and ii) to maximize the margin between the frames and the data. The learned orthonormal frames will thus characterize a piecewise linear decision surface that allows for efficient inference, while our objectives seek to bound the data within a minimal volume that maximizes the decision margin, thereby robustly capturing the data distribution. We explore several variants of our formulation under different constraints on the constituent classifiers, including kernelized feature maps. We demonstrate the empirical benefits of our approach via experiments on data from several applications in computer vision, such as anomaly detection in video sequences, human poses, and human activities. We also explore the generality and effectiveness of GODS for non-vision tasks via experiments on several UCI datasets, demonstrating state-of-the-art results.

Saliency-Based Multilabel Linear Discriminant Analysis.

Linear discriminant analysis (LDA) is a classical statistical machine-learning method, which aims to find a linear data transformation increasing class discrimination in an optimal discriminant subspace. Traditional LDA sets assumptions related to the Gaussian class distributions and single-label data annotations. In this article, we propose a new variant of LDA to be used in multilabel classification tasks for dimensionality reduction on original data to enhance the subsequent performance of any multilabel classifier. A probabilistic class saliency estimation approach is introduced for computing saliency-based weights for all instances. We use the weights to redefine the between-class and within-class scatter matrices needed for calculating the projection matrix. We formulate six different variants of the proposed saliency-based multilabel LDA (SMLDA) based on different prior information on the importance of each instance for their class(es) extracted from labels and features. Our experiments show that the proposed SMLDA leads to performance improvements in various multilabel classification problems compared to several competing dimensionality reduction methods.

A robust spike sorting method based on the joint optimization of linear discrimination analysis and density peaks

Spike sorting is a fundamental step in extracting single-unit activity from neural ensemble recordings, which play an important role in basic neuroscience and neurotechnologies. A few algorithms have been applied in spike sorting. However, when noise level or waveform similarity becomes relatively high, their robustness still faces a big challenge. In this study, we propose a spike sorting method combining Linear Discriminant Analysis (LDA) and Density Peaks (DP) for feature extraction and clustering. Relying on the joint optimization of LDA and DP: DP provides more accurate classification labels for LDA, LDA extracts more discriminative features to cluster for DP, and the algorithm achieves high performance after iteration. We first compared the proposed LDA-DP algorithm with several algorithms on one publicly available simulated dataset and one real rodent neural dataset with different noise levels. We further demonstrated the performance of the LDA-DP method on a real neural dataset from non-human primates with more complex distribution characteristics. The results show that our LDA-DP algorithm extracts a more discriminative feature subspace and achieves better cluster quality than previously established methods in both simulated and real data. Especially in the neural recordings with high noise levels or waveform similarity, the LDA-DP still yields a robust performance with automatic detection of the number of clusters. The proposed LDA-DP algorithm achieved high sorting accuracy and robustness to noise, which offers a promising tool for spike sorting and facilitates the following analysis of neural population activity.

Simultaneous label inference and discriminant projection estimation through adaptive self-taught graphs

Processing structured data has become an interesting topic in recent years. The development of graph-based semi-supervised learning models has attracted much attention from machine learning researchers. In this paper, we present a novel approach for graph-based semi-supervised learning. We provide an effective method for simultaneous label recovery and linear transformation estimation. The targeted linear transformation is to obtain a discriminant subspace. The most important factor in this work to improve the semi-supervised learning is to exploit the data structure and soft labels of the available unlabeled samples. In the iterative optimization scheme used, the prior estimation of the labels increases the monitoring information in an indirect way through an introduced label-graph, avoiding the use of confidence-based hard decisions as used in self-supervised methods. It also enforces label smoothing and projected data smoothing through the use of hybrid graphs. For each smoothing type, the hybrid graph is an adaptive fusion of the two graphs encoding the similarity of the data information and the similarity of the label information. The proposed method leads to an improved discriminant linear transformation. Several experimental results on real image datasets confirm the effectiveness of the proposed method. This work also shows superior performance compared to semi-supervised methods that use simultaneous embedding and inference of labels.

Discriminative aging subspace learning for age estimation

Human age estimation from facial images has become an active research topic in computer vision field because of various real-world applications. Temporal property of facial aging display sequential patterns that lie on the low-dimensional aging manifold. In this paper, we propose hidden factor analysis (HFA) model-based discriminative manifold learning method for age estimation. The hidden factor analysis decomposes facial features into independent age factor and identity factor. Various age invariant face recognition systems in the literature utilize identity factor for face recognition; however, the age factor remains unutilized. The age component of the hidden factor analysis model depends on the subject’s age. Thus it carries significant age-related information. In this paper, we demonstrate that such aging patterns can be effectively extracted from the HFA-based discriminant subspace learning algorithm. Next, we have applied multiple regression methods on low-dimensional aging features learned from the HFA model. Effect of reduced dimensionality on the accuracy has been evaluated by extensive experiments and compared with the state-of-the-art methods. Effectiveness and robustness in terms of MAE and CS of the proposed framework are demonstrated using experimental analysis on a large-scale aging database MORPH II. The accuracy of our method is found superior to the current state-of-the-art methods.

Unsupervised classification of CIGALE galaxy spectra

Aims. Our study aims at providing deeper insight into the power and limitation of an unsupervised classification algorithm (called Fisher-EM) on spectra of galaxies. This algorithm uses a Gaussian mixture in a discriminative latent subspace. To this end, we investigate the capacity of this algorithm to segregate the physical parameters used to generate mock spectra and the influence of the noise on the classification. Methods. With the code CIGALE and different values for nine input parameters characterising the stellar population, we simulated a sample of 11 475 optical spectra of galaxies containing 496 monochromatic fluxes. The statistical model and the optimum number of clusters are given in Fisher-EM by the integrated completed likelihood (ICL) criterion. We repeated the analyses several times to assess the robustness of the results. Results. Two distinct classifications can be distinguished in the case of the noiseless spectra. The classification with more than 13 clusters disappears when noise is added, while the classification with 12 clusters is very robust against noise down to a signal-to-noise ratio (S/N) of 3. At S/N = 1, the optimum is 5 clusters, but the classification is still compatible with the previous classification. The distribution of the parameters used for the simulation shows an excellent discrimination between classes. A higher dispersion both in the spectra within each class and in the parameter distribution leads us to conclude that despite a much higher ICL, the classification with more than 13 clusters in the noiseless case is not physically relevant. Conclusions. This study yields two conclusions that are valid at least for the Fisher-EM algorithm. Firstly, the unsupervised classification of spectra of galaxies is both reliable and robust to noise. Secondly, such analyses are able to extract the useful physical information contained in the spectra and to build highly meaningful classifications. In an epoch of data-driven astrophysics, it is important to trust unsupervised machine-learning approaches that do not require training samples that are unavoidably biased.

Unsupervised robust discriminative subspace representation based on discriminative approximate isometric embedding

Subspace learning has shown a tremendous potential in the fields of machine learning and computer vision due to its effectiveness. Subspace representation is a key subspace learning method that encodes subspace membership information. To effectively encode the subspace memberships of data, some structured prior constraints are imposed on the subspace representation, such as low-rank, sparse, and so on. To handle various noises, existing methods tend to separate a specific type of noise using a specific way to obtain robust subspace representation. When encountering diversified noises, their subspace-preserving property may not be guaranteed. To address this issue, we propose a novel unsupervised robust discriminative subspace representation to mitigate the impacts of diversified noises via discriminative approximate isometric embedding, rather than directly separating noises from the high-dimensional space, as done like the existing methods. To ensure the performance of our approach, we provide a crucial theorem, termed as noisy Johnson–Lindenstrauss theorem. Meanwhile, Laplacian rank constraint is imposed on the discriminative subspace representation to uncover the ground truth subspace memberships of noisy data and improve the graph connectivity of subspaces. Extensive experiments on several benchmark datasets and two large-scale datasets validate the effectiveness and robustness of our approach with respect to diversified noises.

Discriminant Subspace Low-Rank Representation Algorithm for Electroencephalography-Based Alzheimer's Disease Recognition.

Alzheimer’s disease (AD) is a chronic progressive neurodegenerative disease that often occurs in the elderly. Electroencephalography (EEG) signals have a strong correlation with neuropsychological test results and brain structural changes. It has become an effective aid in the early diagnosis of AD by exploiting abnormal brain activity. Because the original EEG has the characteristics of weak amplitude, strong background noise and randomness, the research on intelligent AD recognition based on machine learning is still in the exploratory stage. This paper proposes the discriminant subspace low-rank representation (DSLRR) algorithm for EEG-based AD and mild cognitive impairment (MCI) recognition. The subspace learning and low-rank representation are flexibly integrated into a feature representation model. On the one hand, based on the low-rank representation, the graph discriminant embedding is introduced to constrain the representation coefficients, so that the robust representation coefficients can preserve the local manifold structure of the EEG data. On the other hand, the least squares regression, principle component analysis, and global graph embedding are introduced into the subspace learning, to make the model more discriminative. The objective function of DSLRR is solved by the inexact augmented Lagrange multiplier method. The experimental results show that the DSLRR algorithm has good classification performance, which is helpful for in-depth research on AD and MCI recognition.

Neighborhood Preserving and Weighted Subspace Learning Method for Drift Compensation in Gas Sensor

This article presents a novel discriminative subspace-learning-based unsupervised domain adaptation (DA) method for the gas sensor drift problem. Many existing subspace learning approaches assume that the gas sensor data follow a certain distribution such as Gaussian, which often does not exist in real-world applications. In this article, we address this issue by proposing a novel discriminative subspace learning method for DA with neighborhood preserving (DANP). We introduce two novel terms, including the intraclass graph term and the interclass graph term, to embed the graphs into DA. Besides, most existing methods ignore the influence of the subspace learning on the classifier design. To tackle this issue, we present a novel classifier design method (DANP+) that incorporates the DA ability of the subspace into the learning of the classifier. The weighting function is introduced to assign different weights to different dimensions of the subspace. We have verified the effectiveness of the proposed methods by conducting experiments on two public gas sensor datasets in comparison with the state-of-the-art DA methods.

Center-Aware Adversarial Autoencoder for Anomaly Detection.

Anomaly detection based on subspace learning has attracted much attention, in which the compactness of subspace is commonly considered as the core concern. Most related studies directly optimize the distance from the subspace representation to the fixed center, and the influence of the anomaly level of each normal sample is not considered to adjust the normal concentrated areas. In such cases, it is difficult to isolate the normal areas from the anomaly ones by making the subspace compact. To this end, we propose a center-aware adversarial autoencoder (CA-AAE) method, which detects anomaly samples by acquiring more compact and discriminative subspace representations. To fully exploit the subspace information to improve the compactness, anomaly-level description and feature learning are novelly integrated herein by dividing the output space of the encoder into presubspace and postsubspace. In presubspace, the toward-center prior distribution is imposed by the adversarial learning mechanism, and the anomaly level of normal samples can be described from a probabilistic perspective. In postsubspace, a novel center-aware strategy is established to enhance the compactness of the postsubspace, which achieves adaptive adjustment of the normal areas by constructing a weighted center based on the anomaly level. Then, a flexible anomaly score function is constructed in the testing stage, in which both the toward-center loss and the reconstruction loss are combined to balance the information in the learned subspace and the original space. Compared to other related methods, the proposed CA-AAE shows the effectiveness and advantages in numerical experiments.

A Cross-Media Advertising Design and Communication Model Based on Feature Subspace Learning.

This paper uses feature subspace learning and cross-media retrieval analysis to construct an advertising design and communication model. To address the problems of the traditional feature subspace learning model and make the samples effectively maintain their local structure and discriminative properties after projection into the feature space, this paper proposes a discriminative feature subspace learning model based on Low-Rank Representation (LRR), which explores the local structure of samples through Low-Rank Representation and uses the representation coefficients as similarity constraints of samples in the projection space so that the projection subspace can better maintain the local nearest-neighbor relationship of samples. Based on the common subspace learning, this paper uses the extreme learning machine method to improve the cross-modal retrieval accuracy, mining deeper data features and maximizing the correlation between different modalities, so that the learned shared subspace is more discriminative; meanwhile, it proposes realizing cross-modal retrieval by the deep convolutional generative adversarial network, using unlabeled samples to further explore the correlation of different modal data and improve the cross-modal performance. The clustering quality of images and audios is corrected in the feature subspace obtained by dimensionality reduction through an optimization algorithm based on similarity transfer. Three active learning strategies are designed to calculate the conditional probability of unannotated samples around user-annotated samples in the correlation feedback process, thus improving the efficiency of cross-media retrieval in the case of limited feedback samples. The experimental results show that the method accurately measures the cross-media relevance and effectively achieves mutual retrieval between image and audio data. Through the study of cross-media advertising design and communication models based on feature subspace learning, it is of positive significance to advance commercial advertising design by guiding designers and artists to better utilize digital media technology for artistic design activities at the level of theoretical research and applied practice.

Learning robust graph for clustering

Graph is a popular technique to explore the structure of data. Many related algorithms directly construct graphs based on the original data. Actually, the samples collected in real life usually contain noise. Besides, some unimportant features probably exist in high-dimensional data. Therefore, this way cannot assure a high-quality graph and furthermore brings some adverse influence to the following tasks. In this paper, we incorporate robust graph learning and dimensionality reduction into a unified framework which also seamlessly integrates the clustering task. On the basis of the framework, Euclidean distance-based robust graph (EDBRG) and self-expressiveness-based robust graph (SEBRG) are presented. Both EDBRG and SEBRG contain clustering information from which the clustering results can be obtained directly. By projecting the original data into a discriminative subspace where the negative effect of redundant features and noise is removed, EDBRG and SEBRG are informative, robust, and sparse. During the whole mapping process, the main energy of data is preserved. Finally, some data sets are adopted to test the performances of EDBRG and SEBRG. Extensive experiments illustrate that the proposed methods have many advantages for the task of clustering, comparing with the state-of-the-art algorithms.

Feature reduction based transfer structural subspace learning for small-footprint cross-domain keyword spotting via linear discriminant analysis

Small-footprint keyword spotting has received considerable attention in recent years, which is often conducted on the assumption that the predefined keywords in training and testing data are obtained under the same condition. However, in practical situations, this assumption does not hold due to the widespread existence of various speech scenarios or voice-embedded devices. To tackle this problem, in this paper, we propose a new transfer subspace learning method called feature reduction based transfer structural subspace learning (FRTSSL) for small-footprint cross-domain keyword spotting. FRTSSL aims to learn a domain-invariant and discriminative subspace by which (1) feature reduction is used to high dimensional features to avoid unnecessary computation; and (2) transfer structural subspace learning jointly exploits the statistical properties and geometric structure to reduce the distribution discrepancy of the source and target domains based on a joint linear discriminant analysis (LDA) framework; and (3) the feedback term is constructed for improving the discrimination of the subspace based on source labels and pseudo-target labels. To preserve the intrinsic geometric structure of samples in the projection subspace, we first preserve the global subspace structure by imposing the reconstruction constraints on the reconstruction coefficient matrix, and then we preserve the space relationship of samples using a graph regularization method. Furthermore, we formulate a minimization problem that integrates marginal and conditional distribution alignment, reconstruction constraints, and graph regularization into the joint LDA framework, giving an effective optimization algorithm. Experimental results on four cross-domain keyword datasets show that our method outperforms some state-of-the-art conventional transfer learning methods and no transfer learning methods.

Distribution probability‐based self‐adaption metric learning for person re‐identification

Person re-identification addresses the problem of pedestrian image matching in a non-overlapped surveillance network. Traditional metric learning methods try to learn a fixed pedestrian images matching metric model. However, existing metric learning-based methods have the problem of overfitting the training data. In order to solve this problem, a sample-specific metric learning-based method is proposed. In the perspective of probability distribution, the over-fitting problem is attributed to the problem that the generalisation ability of projection features is weak. Firstly, the proposed method learns a metric subspace, which projects the raw feature to a discriminative subspace. Then, a projection feature selection method is established based on the probability distribution of positive image pairs. According to the proposed method, the probability that the test samples following the training data distribution is closely related to the adaptability. The proposed projection feature selection method selects different projection features for each individual's similarity distance measure. Finally, extensive experiments on four published datasets verify the effectiveness of the proposed metric learning method. It performs favourably against the comparison methods, especially on the rank-1 rate.

Learning task-specific discriminative embeddings for few-shot image classification

Recently, few-shot learning has attracted more and more attention. Generally, the fine-tuning-based few-shot learning framework contains two stages: i) In the pre-training stage, using base data to train the feature extractor; ii) In the meta-testing stage, using a well-trained feature extractor to extract embedding features of novel data and designing a base learner to predict the labels. Due to the diverse categories of base and novel data, it is challenging for the feature extractor trained in the pre-training stage to adapt to novel data, which will result in an embedding-mismatch problem. This paper proposes Task-specific Discriminative Embeddings for Few-shot Learning (TDE-FSL) to solve the embedding-mismatch problem. Specifically, we embed the dictionary learning method into the few-shot learning framework to map the feature embeddings to a more discriminative subspace to adapt to the specific task. Moreover, we extend the self-training framework to our approach to fully utilize the unlabeled data. Finally, we evaluate the TDE-FSL on five benchmark image datasets, such as mini-Imagenet, tiered-Imagenet, CIFAR-FS, FC100, and CUB dataset. The experimental results show that the performance of our proposed TDE-FSL achieves a significant improvement.

Knowledge-based quality control of organ delineations in radiation therapy.

To reduce the likelihood of errors in organ delineations used for radiotherapy treatment planning, a knowledge-based quality control (KBQC) system, which discriminates between valid and anomalous delineations isdeveloped. The KBQC is comprised of a group-wise inference system and anomaly detection modules trained using historical priors from 296 locally advanced lung and prostate cancer patient computational tomographies (CTs). The inference system discriminates different organs based on shape, relational, and intensity features. For a given delineated image set, the inference system solves a combinatorial optimization problem that results in an organ group whose relational features follow those of the training set considering the posterior probabilities obtained from support vector machine (SVM), discriminant subspace ensemble (DSE), and artificial neural network (ANN) classifiers. These classifiers are trained on nonrelational features with a 10-fold cross-validation scheme. The anomaly detection module is a bank of ANN autoencoders, each corresponding with an organ, trained on nonrelational features. A heuristic rule detects anomalous organs that exceed predefined organ-specific tolerances for the feature reconstruction error and the classifier's posterior probabilities. Independent data sets with anomalous delineations were used to test the overall performance of the KBQC system. The anomalous delineations were manually manipulated, computer-generated, or propagated based on a transformation obtained by imperfect registrations. Both peer-review-based scoring system and shape similarity coefficient (DSC) were used to label regions of interest (ROIs) as normal or anomalous in two independent test cohorts. The accuracy of the classifiers was 99.8%, and the minimum per-class F1-scores were 0.99, 0.99, and 0.98 for SVM, DSE, and ANN, respectively. The group-wise inference system reduced the miss-classification likelihood for the test data set with anomalous delineations compared to each individual classifier and a fused classifier that used the average posterior probability of all classifiers. For 15 independent locally advanced lung patients, the system detected 79% of the anomalous ROIs. For 1320 auto-segmented abdominopelvic organs, the anomaly detection system identified anomalous delineations, which also had low Dice similarity coefficient values with respect to manually delineated organs in the training dataset. The KBQC system detected anomalous delineations with superior accuracy compared to classification methods that judge only based on posterior probabilities.

DS4L: Deep Semisupervised Shared Subspace Learning for Hyperspectral Image Classification

Hyperspectral image (HSI) classification is essential in remote sensing image analysis. The classification methods based on deep learning have attracted more and more attention. However, classification accuracy is seriously affected by the quantity of labeled data and redundant information. Therefore, a deep semisupervised shared subspace learning (DS <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">4</sup> L) model is developed to overcome these problems in this article. DS <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">4</sup> L is composed of two parts. First, the basic feature extraction (BFE) network is constructed to preliminary extract high-dimensional-space features of multiscale data and fusion them to one shared subspace. Then, a deep shared subspace learning (DSSL) network is proposed to obtain a deeper and more representative low-dimensional subspace. Moreover, to obtain a more representative subspace and alleviate dependence on labeled samples, the regular, irregular constraint, and cross-entropy (CE) loss are integrated into the model. The regular constraint is adopted to reconstruct the multiscale patches to ensure the quality of the subspace in an unsupervised manner. The irregular constraint can well embed labeled and unlabeled samples into the procedure of subspace learning (SL). Then, the CE loss is used to extract more discriminative subspace using the limited labeled samples. Finally, we perform experiments on three widely used HSI datasets. Compared with the basic SL model, the DS <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">4</sup> L’s classification accuracy on the popular Salinas, Indian Pines, and PaviaU datasets are increased by 4.68%, 4.25%, and 1.57%, respectively.

Locality-Aware Discriminative Subspace Learning for Image Classification

Projection learning is an effective and widely used technique for extracting discriminative features for pattern recognition and classification. In projection learning, it is essential to preserve the global and local structure of the data while extracting discriminative features. However, transforming the source data directly to a target i.e., the strict binary label matrix, using a projection matrix may result in the loss of some intrinsic information. We propose a locality-aware discriminative subspace learning (LADSL) method to address these limitations. In LADSL, the original data is transformed into a latent space instead of a restrictive label space. The latent space seamlessly integrates the original visual features and the class labels to improve the classification performance. The projection matrix and classification parameters are jointly optimized to supervise the discriminative subspace learning. Additionally, LADSL exploits the adaptive local structure to preserve the nearest neighbor relationship among the data samples while learning more projections to achieve superior classification performance. Experiments have been carried out on various data sets for face and object recognition, and the results achieved are compared with the state-of-the-art methods to validate the effectiveness of the proposed LADSL method.