Semi-supervised Dimensionality Reduction Method Research Articles

Semi-supervised t-SNE with multi-scale neighborhood preservation

Unsupervised dimensionality reduction (DR) aims to preserve input data structure in a low-dimensional (LD) space based on neighborhood information. In contrast, supervised DR intends to improve the learning performance, i.e., classification and regression, in an LD representation. Unfortunately, obtaining the complete label outputs of a data set for real-world applications is hard. Here, we introduce a novel DR framework coupling both available class labels and input feature similarities to extend the well-known t-distributed Stochastic Neighbor Embedding (SNE) for semi-supervised scenarios. Our proposal, termed Semi-Supervised t-SNE (SS.t-SNE), properly fixes the widths of Gaussian neighborhoods to reveal the salient local and global data structures in an LD space. Indeed, our approach is presented as a generalization of unsupervised and supervised versions of t-SNE. SS.t-SNE outperforms other semi-supervised DR methods in data visualization and classification tasks in LD embeddings.

ScSemiAE: a deep model with semi-supervised learning for single-cell transcriptomics.

BackgroundWith the development of modern sequencing technology, hundreds of thousands of single-cell RNA-sequencing (scRNA-seq) profiles allow to explore the heterogeneity in the cell level, but it faces the challenges of high dimensions and high sparsity. Dimensionality reduction is essential for downstream analysis, such as clustering to identify cell subpopulations. Usually, dimensionality reduction follows unsupervised approach.ResultsIn this paper, we introduce a semi-supervised dimensionality reduction method named scSemiAE, which is based on an autoencoder model. It transfers the information contained in available datasets with cell subpopulation labels to guide the search of better low-dimensional representations, which can ease further analysis.ConclusionsExperiments on five public datasets show that, scSemiAE outperforms both unsupervised and semi-supervised baselines whether the transferred information embodied in the number of labeled cells and labeled cell subpopulations is much or less.Supplementary InformationThe online version contains supplementary material available at 10.1186/s12859-022-04703-0.

Adaptive Local Embedding Learning for Semi-Supervised Dimensionality Reduction

Semi-supervised learning as one of most attractive problems in machine learning research field has aroused broad attentions in recent years. In this paper, we propose a novel locality preserved dimensionality reduction framework, named Semi-supervised Adaptive Local Embedding learning (SALE), which learns a local discriminative embedding by constructing a <inline-formula><tex-math notation="LaTeX">$k_1$</tex-math></inline-formula> Nearest Neighbors ( <inline-formula><tex-math notation="LaTeX">$k_1$</tex-math></inline-formula> NN) graph on labeled data, so as to explore the intrinsic structure, i.e., sub-manifolds from non-Gaussian labeled data. Then, mapping all samples into learned embedding and constructing another <inline-formula><tex-math notation="LaTeX">$k_2$</tex-math></inline-formula> NN graph on all embedded data to explore the global structure of all samples. Therefore, the unlabeled data and their corresponding labeled neighbors can be clustered into same sub-manifold, so as to improve the discriminative power of embedded data. Furthermore, we propose two semi-supervised dimensionality reduction methods with orthogonal and whitening constraints based on proposed SALE framework. An efficient alternatively iterative optimization algorithm is developed to solve the NP-hard problem in our models. Extensive experiments conducted on several synthetic and real-world data sets demonstrate the superiorities of our methods on local structure exploration and classification task.

Semisupervised Linear Discriminant Analysis Based on Pairwise Constraint Propagation for Hyperspectral Images

As a classical supervised dimensionality reduction (DR) method, linear discriminant analysis (LDA) has been developed for many variants. However, it is not applicable to the case that labeled samples are scarce and unlabeled samples are in large quantity, which always happens in the real world. In this letter, we propose a novel technique format termed semisupervised LDA based on pairwise constraint propagation (SLDA-PCP) for hyperspectral images (HSIs). The basic idea of this method is to use a specially designed PCP technique to propagate label information from the labeled samples to the unlabeled samples. In addition, an extended LDA format to learn the optimal projection vectors according to the newly obtained label information is also created. Comprehensive experiments on two HSIs show that our SLDA-PCP performs better than some state-of-the-art semisupervised DR methods.

NetAE: semi-supervised dimensionality reduction of single-cell RNA sequencing to facilitate cell labeling.

Single-cell RNA sequencing allows us to study cell heterogeneity at an unprecedented cell-level resolution and identify known and new cell populations. Current cell labeling pipeline uses unsupervised clustering and assigns labels to clusters by manual inspection. However, this pipeline does not utilize available gold-standard labels because there are usually too few of them to be useful to most computational methods. This article aims to facilitate cell labeling with a semi-supervised method in an alternative pipeline, in which a few gold-standard labels are first identified and then extended to the rest of the cells computationally. We built a semi-supervised dimensionality reduction method, a network-enhanced autoencoder (netAE). Tested on three public datasets, netAE outperforms various dimensionality reduction baselines and achieves satisfactory classification accuracy even when the labeled set is very small, without disrupting the similarity structure of the original space. The code of netAE is available on GitHub: https://github.com/LeoZDong/netAE. Supplementary data are available at Bioinformatics online.

Semisupervised Manifold Joint Hypergraphs for Dimensionality Reduction of Hyperspectral Image

In this letter, a new semisupervised dimensionality reduction (DR) method, termed geodesic-based manifold joint hypergraphs (GMJHs), is proposed for hyperspectral image (HSI). This method first builds a geodesic-based reconstruction model to discover the nonlinear similarity between two manifold reconstruction neighborhoods. Then, it implies the probabilistic relationship between unlabeled samples and each class via the geodesic-based reconstruction distance. With the probabilistic class relationship, a supervised hypergraph and an unsupervised hypergraph are constructed to represent the multivariate manifold relationship of samples. Finally, the supervised and unsupervised hypergraphs are jointed for learning optimal projection matrix and enhancing the intraclass compactness in low-dimensional embedding space. Experiments on two HSI data sets show that the proposed GMJH algorithm performs better performance than some state-of-the-art DR methods.

Semi-supervised partial least squares

Traditional supervised dimensionality reduction methods can establish a better model often under the premise of a large number of samples. However, in real-world applications where labeled data are scarce, traditional methods tend to perform poorly because of overfitting. In such cases, unlabeled samples could be useful in improving the performance. In this paper, we propose a semi-supervised dimensionality reduction method by using partial least squares (PLS) which we call semi-supervised partial least squares (S2PLS). To combine the labeled and unlabeled samples into a S2PLS model, we first apply the PLS algorithm to unsupervised dimensionality reduction. Then, the final S2PLS model is established by ensembling the supervised PLS model and the unsupervised PLS model which using the basic idea of principal model analysis (PMA) method. Compared with unsupervised or supervised dimensionality reduction algorithms, S2PLS not only can improve the prediction accuracy of the samples but also enhance the generalization ability of the model. Meanwhile, it can obtain better results even there are only a few or no labeled samples. Experimental results on five UCI data sets also confirmed the above properties of S2PLS algorithm.

Transductive non-linear semantic embedding for multi-class classification

This paper presents a method for semi-supervised multi-class classification based on matrix factorization. The method called semi-supervised online kernel matrix factorization (SS-OKMF) performs a semantic embedding by finding a non-linear mapping to a low-dimensional semantic space. An important characteristic of the SS-OKMF method is that the new low-dimensional semantic representation can be learned in a semi-supervised fashion. Therefore, the annotated instances can be used to maximize the discrimination between classes, but also, the non-annotated instances can be exploited to estimate the intrinsic manifold structure of the data. The non-linear modeling is based on kernel methods with a learning-in-a-budget strategy that allows keeping low memory requirements. This strategy, along with an online formulation based on stochastic gradient descent, reduces the computation time and keeps low computational requirements in large-scale problems. According to the experimental evaluation performed on several datasets of different nature (i.e. images, biosignals, and synthetic data), SS-OKMF, in comparison with several non-linear supervised, unsupervised and semi-supervised dimensionality reduction methods, presents a competitive performance in classification tasks under a transductive learning setup preserving a lower computational cost.

Noisy multi-label semi-supervised dimensionality reduction

Noisy labeled data represent a rich source of information that often are easily accessible and cheap to obtain, but label noise might also have many negative consequences if not accounted for. How to fully utilize noisy labels has been studied extensively within the framework of standard supervised machine learning over a period of several decades. However, very little research has been conducted on solving the challenge posed by noisy labels in non-standard settings. This includes situations where only a fraction of the samples are labeled (semi-supervised) and each high-dimensional sample is associated with multiple labels. In this work, we present a novel semi-supervised and multi-label dimensionality reduction method that effectively utilizes information from both noisy multi-labels and unlabeled data. With the proposed Noisy multi-label semi-supervised dimensionality reduction (NMLSDR) method, the noisy multi-labels are denoised and unlabeled data are labeled simultaneously via a specially designed label propagation algorithm. NMLSDR then learns a projection matrix for reducing the dimensionality by maximizing the dependence between the enlarged and denoised multi-label space and the features in the projected space. Extensive experiments on synthetic data, benchmark datasets, as well as a real-world case study, demonstrate the effectiveness of the proposed algorithm and show that it outperforms state-of-the-art multi-label feature extraction algorithms.

Semisupervised dimensionality reduction for hyperspectral images based on the combination of semisupervised learning and metric learning

ABSTRACTThis paper presents a semisupervised dimensionality reduction (DR) method based on the combination of semisupervised learning (SSL) and metric learning (ML) (CSSLML-DR) in order to overcome some existing limitations in HSIs analysis. Specifically, CSSML focuses on the difficulties of high dimensionality of hyperspectral images (HSIs) data, the insufficient number of labelled samples and inappropriate distance metric. CSSLML aims to learn a local metrics under which the similar samples are pushed as close as possible, and simultaneously, the different samples are pulled away as far as possible. CSSLML constructs two local-reweighted dynamic graphs in an iterative two-steps approach: L-step and V-step. In L-step, the local between-class and within-class graphs are updated. In V-step, the transformation matrix and the reduced space are updated. The algorithm is repeated until a stopping criterion is satisfied. Experimental results on two well-known hyperspectral image data sets demonstrate the superiority of CSSLML algorithm compared to some traditional DR methods.

A noise-robust semi-supervised dimensionality reduction method for face recognition

Face recognition (FR) is a fundamental problem in a biometric identification system and has attracted much attention in pattern recognition and computer vision fields. Since human face images have high dimensionality, dimensionality reduction (DR) is often adapted for FR and a number of relevant methods are proposed, such as principal component analysis (PCA), linear discriminant analysis (LDA). However, face images are generally hard to be labeled and corrupted by noise in the collection phase. To address the problem, we present a novel semi-supervised DR method which is more robust to noise than the traditional DR methods. Our basic idea is that collaborative representation-based classification (CRC) can achieve better performance than the classifier trained by Euclidean distance when the face images are corrupted by noise. In our algorithm, we firstly employ CRC to compute a representation coefficient for each face image. We then construct a reconstruction error-based regularization term through the obtained coefficient vector. Finally, we extend LDA to the semi-supervised framework by embedding the regularization term into LDA. To evaluate the effectiveness of our algorithm, we use four well-known face databases to conduct several experiments by comparing to unsupervised, supervised and semi-supervised DR methods. The results illustrate that our algorithm can always achieve the best performance as the noise corruption percent of noise images increases.

Global and local scatter based semi-supervised dimensionality reduction with active constraints selection in ensemble subspaces

Semi-supervised dimensionality reduction has attracted an increasing amount of attention in this big-data era. Many algorithms have been developed with a small number of pairwise constraints to achieve performances comparable to those of fully su pervised methods. However, one challenging problem with semi-supervised approaches is the appropriate choice of the constraint set, including the cardinality and the composition of the constraint set which, to a large extent, affects the performance of the resulting algorithm. In this work, we address the problem by incorporating ensemble subspaces and active learning into dimensionality reduction and propose a new global and local scatter based semi-supervised dimensionality reduction method with active constraints selection. Unlike traditional methods that select the supervised information in one subspace, we pick up pairwise constraints in ensemble subspaces, where a novel active learning algorithm is designed with both exploration and filtering to generate informative pairwise constraints. The automatic constraint selection approach proposed in this paper can be generalized to be used with all constraint-based semi-supervised learning algorithms. Comparative experiments are conducted on four face database and the results validate the effectiveness of the proposed method.

Adaptive semi-supervised dimensionality reduction with sparse representation using pairwise constraints

With the rapid accumulation of high dimensional data, dimensionality reduction plays a more and more important role in practical data processing and learning tasks. This paper studies semi-supervised dimensionality reduction using pairwise constraints. In this setting, domain knowledge is given in the form of pairwise constraints, which specifies whether a pair of instances belong to the same class (must-link constraint) or different classes (cannot-link constraint). In this paper, a novel semi-supervised dimensionality reduction method called Adaptive Semi-Supervised Dimensionality Reduction with Sparse Representation (ASSDR-SR) is proposed, which can get the optimized low dimensional representation of the original data by adaptively adjusting the weights of the pairwise constraints and simultaneously optimizing the graph construction using the ℓ1 graph of sparse representation. Experiments on clustering and classification tasks show that ASSDR-SR is superior to some existing dimensionality reduction methods.

Learning from normalized local and global discriminative information for semi-supervised regression and dimensionality reduction

Semi-supervised dimensionality reduction is one of the important topics in pattern recognition and machine learning. During the past decade, Laplacian Regularized Least Square (LapRLS) and Semi-supervised Discriminant Analysis (SDA) are the two widely-used semi-supervised dimensionality reduction methods. In this paper, we show that SDA and LapRLS can be unified into a constrained manifold regularized least square framework. The manifold term, however, cannot fully utilize the underlying discriminative information. We thus introduce a new and effective semi-supervised dimensionality reduction method, called Learning from Local and Global Information (LLGDI), to solve the problem. The proposed LLGDI method adopts a set of local classification functions to preserve both local geometrical and discriminative information of dataset. It also adopts a global classification function to preserve the global discriminative information, and an uncorrelated constraint to calculate the projection matrix for simultaneously solving regression and dimensionality reduction problem. As a result, the LLGDI method is able to preserve local discriminative, manifold information as well as the global discriminative information. Theoretical analysis and extensive simulations presented in the paper show the effectiveness of the LLGDI algorithm. The results also demonstrate LLGDI can achieve superior performance compared with other existing methods.

Adaptive semi-supervised dimensionality reduction based on pairwise constraints weighting and graph optimizing

With the rapid growth of high dimensional data, dimensionality reduction is playing a more and more important role in practical data processing and analysing tasks. This paper studies semi-supervised dimensionality reduction using pairwise constraints. In this setting, domain knowledge is given in the form of pairwise constraints, which specifies whether a pair of instances belong to the same class (must-link constraint) or different classes (cannot-link constraint). In this paper, a novel semi-supervised dimensionality reduction method called adaptive semi-supervised dimensionality reduction (ASSDR) is proposed, which can get the optimized low dimensional representation of the original data by adaptively adjusting the weights of the pairwise constraints and simultaneously optimizing the graph construction. Experiments on UCI classification and image recognition show that ASSDR is superior to many existing dimensionality reduction methods.

Image classification via least square semi-supervised discriminant analysis with flexible kernel regression for out-of-sample extension

Semi-supervised dimensionality reduction is an important research topic in many pattern recognition and machine learning applications. Among all the methods for semi-supervised dimensionality reduction, SDA and LapRLS/L are two popular ones. Though the two methods are actually the extensions of different supervised methods, we show in this paper that both SDA and Lap-RLS/L can be unified under a regularized least square framework. In this paper, we propose a new effective semi-supervised dimensionality reduction method for better cope with data sampled from nonlinear manifold. In addition, the proposed method can both handle the regression as well as the subspace learning problem. Theoretical analysis and extensive simulations show the effectiveness of our algorithm. The results in simulations demonstrate that our proposed algorithm can achieve great superiority compared with other existing methods.

A Weighted Semi-Supervised Dimensionality Reduction Method Based Locally Linear Graph

A Weighted Semi-Supervised Dimensionality Reduction Method Based Locally Linear Graph

Semi-Supervised Hyperspectral Subspace Learning Based on a Generalized Eigenvalue Problem for Regression and Dimensionality Reduction

Manifold learning for the hyperspectral data structure of intra-class variation provides useful information for investigating the intrinsic coordinates corresponding to the quantitative properties inherent in the class. However, in the high-dimensional feature space, it is unfeasible to acquire a statistically sufficient number of labeled data to estimate the coordinates. In this paper, we propose semi-supervised regression and dimensionality reduction methods for hyperspectral subspace learning that utilize abundant unlabeled data and a small number of labeled data. The quantitative target variables for regression and the order constraints for dimensionality reduction are embedded in matrices representing data relations, i.e., a set of between-class scatter matrices, within-class scatter matrices, and supervised local attraction matrices. The optimal projection matrices are estimated by generalized eigenvalue problems based on the matrices. The proposed methods are applied to synthetic linear regression problems and dimensionality reduction problems based on a time-series of hyperspectral data for a deciduous broad-leaved forest to extract local coordinates related to phenological changes. The order consistency of the projections is assessed by evaluating an index based on the Mann-Kendall test statistics. The proposed methods demonstrate much better performances in terms of both regression and dimensionality reduction than the alternative supervised and unsupervised methods.

Visualization of graphical data in a user-specified 2D space using a weighted Isomap method

We present a visualization system for exploring the high-dimensional graphical data, such as textures or 3D models, in 2D space using the dimensionality reduction method. To arrange high-dimensional data in a meaningful 2D space, we develop a novel semi-supervised dimensionality reduction method that can embed data of high dimension in a user-defined 2D coordinate system that is meaningful in terms of the properties of the data. This is achieved by modifying the Isomap method by weighting the data so that the resulting coordinates have no degeneracies and are orthogonal.

Semisupervised Dimensionality Reduction of Hyperspectral Images via Local Scaling Cut Criterion

Hyperspectral images (HSIs) provide a vast amount of geometrical, radiation, and spectral information about a scene. However, high-dimensional data make HSI classification complex and time consuming. It is important to reduce the dimensionality and find a low-dimensional representation of the high-dimensional data. Since the labels of HSI data are really difficult to collect while the unlabeled data are abundant and easy to obtain, in this letter, a semisupervised dimensionality reduction method using both limited labeled samples and a large number of unlabeled samples based on a local scaling cut (LSC) criterion is proposed. LSC is similar to linear discriminant analysis (LDA), but it can handle the heteroscedastic and multimodal data for which LDA fails. The framework of our proposed method contains two terms: 1) a discrimination term based on the labeled samples and 2) a regularization term based on the prior knowledge provided by both labeled and unlabeled samples. Experimental results show that our proposed algorithm provides a relatively promising performance compared with other methods. Moreover, the algorithm is stable and insensitive to parameters.