Semi-supervised Dimensionality Reduction Research Articles

Semisupervised Dimension Reduction Based on Pairwise Constraint Propagation for Hyperspectral Images

This letter presents a semisupervised dimension reduction method based on pairwise constraint propagation (SSDR-PCP) for hyperspectral images (HSIs). SSDR-PCP first utilizes pairwise constraint propagation, which is based on the labeled samples and k-nearest neighbor graphs to obtain more similarity information. Then SSDR-PCP applies the obtained weak supervised information of the entire training data set to construct a new similarity matrix. At last, we embed the similarity matrix to local preserving projection to achieve dimension reduction by finding the optimal transformation matrix for HSIs. The experimental results demonstrate that SSDR-PCP achieves better performance than the previous methods on two HSIs.

Recursively global and local discriminant analysis for semi-supervised and unsupervised dimension reduction with image analysis

Semi-supervised discriminant analysis (SDA) is a recently-developed semi-supervised dimension reduction method for improving the performance of Fisher linear discriminant analysis (FDA), which attempts to mine the local structures of both labeled and unlabeled data. In this paper, we develop new semi-supervised and unsupervised discriminant analysis techniques. Our semi-supervised method, referred as to as recursively global and local discriminant analysis (RGLDA), is modeled based on the characterizations of “locality” and “non-locality”, such that the manifold regularization in the formulation has a more direct connection to classification. The objective of RGLDA is a “concave-convex” programming problem based on the hinge loss. Its solution follows from solving multiple related SVM-type problems. In addition, we also propose a simple version (called URGLDA) for unsupervised dimension reduction. The experiments tried out on several image databases show the effectiveness of RGLDA and URGLDA.

Global and local scatter based semi-supervised dimensionality reduction with active constraints selection in ensemble subspaces

Semi-supervised dimensionality reduction has attracted an increasing amount of attention in this big-data era. Many algorithms have been developed with a small number of pairwise constraints to achieve performances comparable to those of fully su pervised methods. However, one challenging problem with semi-supervised approaches is the appropriate choice of the constraint set, including the cardinality and the composition of the constraint set which, to a large extent, affects the performance of the resulting algorithm. In this work, we address the problem by incorporating ensemble subspaces and active learning into dimensionality reduction and propose a new global and local scatter based semi-supervised dimensionality reduction method with active constraints selection. Unlike traditional methods that select the supervised information in one subspace, we pick up pairwise constraints in ensemble subspaces, where a novel active learning algorithm is designed with both exploration and filtering to generate informative pairwise constraints. The automatic constraint selection approach proposed in this paper can be generalized to be used with all constraint-based semi-supervised learning algorithms. Comparative experiments are conducted on four face database and the results validate the effectiveness of the proposed method.

A Novel Semi-Supervised Dimensionality Reduction Framework for Multi-Manifold Learning

In pattern recognition, when a data set contains multiple classes and the structures of the classes are different, single manifold assumption can hardly guarantee the best classification performance. It is more reasonable to assume each class lies on a separate manifold. In this paper, we propose a novel framework of semi-supervised dimensionality reduction for multi-manifold learning. To address the issue of label insufficiency under mulit-manifold assumption, we propose to solve three challenging problems: cluster unlabeled samples into different manifolds by sparse manifold clustering even when they are close to each other; predict the label of image sets instead of a single sample by calculating manifold to manifold distance; construct three kinds of graphs for each manifold to exploit more information from unlabeled samples. During the investigation, we are able to demonstrate that most existing dimension reduction methods based on manifold learning can be viewed as special cases of the proposed. Experimental results verify the advantages and effectiveness of this new framework.

A non-negative sparse semi-supervised dimensionality reduction algorithm for hyperspectral data

A non-negative sparse semi-supervised dimensionality reduction algorithm is proposed for hyperspectral data by making adequate use of a few labeled samples and a large number of unlabeled samples. The objective function of the proposed algorithm consists of two terms: (1) a discriminant item is designed to analyze a few labeled samples from the global viewpoint, which can assess the separability between surface objects; (2) a regularization term is used to build a non-negative sparse representation graph based on the unlabeled samples, which can adaptively find an adjacency graph for each sample and then find valuable samples with huge information volume from the original hyperspectral data. Based on the objective function and the maximum margin criterion, a dimensionality reduction algorithm, the non-negative sparse semi-supervised maximum margin algorithm, is proposed. Experimental results on the ROSIS University and AVIRIS 92AV3C hyperspectral data sets show that the proposed algorithm can effectively utilize the unlabeled samples to achieve higher overall classification accuracy and Kappa coefficient when compared with some representative supervised, unsupervised and semi-supervised dimensionality reduction algorithms.

Adaptive semi-supervised dimensionality reduction with sparse representation using pairwise constraints

With the rapid accumulation of high dimensional data, dimensionality reduction plays a more and more important role in practical data processing and learning tasks. This paper studies semi-supervised dimensionality reduction using pairwise constraints. In this setting, domain knowledge is given in the form of pairwise constraints, which specifies whether a pair of instances belong to the same class (must-link constraint) or different classes (cannot-link constraint). In this paper, a novel semi-supervised dimensionality reduction method called Adaptive Semi-Supervised Dimensionality Reduction with Sparse Representation (ASSDR-SR) is proposed, which can get the optimized low dimensional representation of the original data by adaptively adjusting the weights of the pairwise constraints and simultaneously optimizing the graph construction using the ℓ1 graph of sparse representation. Experiments on clustering and classification tasks show that ASSDR-SR is superior to some existing dimensionality reduction methods.

Can the Virtual Labels Obtained by Traditional LP Approaches Be Well Encoded in WLR?

Semisupervised dimension reduction via virtual label regression first derives the virtual labels of unlabeled data by employing a newly designed label propagation (LP) approach (called Special random walk (SRW)) and then encodes them in a weighted linear regression model. Nie et al. (2011) highlighted two important characteristics of SRW nonexistent in the previous LP approaches: outlier detection and probability value output, which guarantee the elegant encoding of the resultant virtual labels in the weighted label regression. However, in this brief, we show that the relationship between the SRW and the previous work on LP is very close. Naturally, a problem deserving investigation is whether traditional LP approaches are indeed unable to share the above two characteristics of SRW. We aim to address this problem.

Erratum to: Relative manifold based semi-supervised dimensionality reduction

Figure 8 of this article shows YaleB and CMU PIE with incorrect legend titles: YaleB (Tr=1900, Te=514, NOC=100) should be YaleB (Tr=1900, Te=514, d=100) (Fig. 8(a)); TIE (Tr=1200, Te=2880, d=100) should be PIE (Tr=1200, Te=2880, d=100) (Fig. 8(b)). In Fig. 9, the legend keys and the legend texts are mismatched. The correct figure is illustrated as follows.

Active constraints selection based semi-supervised dimensionality in ensemble subspaces

Semi-supervised dimensionality reduction (SSDR) has attracted an increasing amount of attention in this big-data era. Many algorithms have been developed with a small number of pairwise constraints to achieve performances comparable to those of fully supervised methods. However, one challenging problem with semi-supervised approaches is the appropriate choice of the constraint set, including the cardinality and the composition of the constraint set, which to a large extent, affects the performance of the resulting algorithm. In this work, we address the problem by incorporating ensemble subspace and active learning into dimensionality reduction and propose a new algorithm, termed as global and local scatter based SSDR with active pairwise constraints selection in ensemble subspaces (SSGL-ESA). Unlike traditional methods that select the supervised information in one subspace, we pick up pairwise constraints in ensemble subspace, where a novel active learning algorithm is designed with both exploration and filtering to generate informative pairwise constraints. The automatic constraint selection approach proposed in this paper can be generalized to be used with all constraint-based semi-supervised learning algorithms. Comparative experiments are conducted on two face database and the results validate the effectiveness of the proposed method.

Recursive Dimension Reduction for semisupervised learning

Semisupervised Dimension Reduction (SDR) Using Trace Ratio Criterion (TR-FSDA) is an effective iterative SDR algorithm, which introduces a flexible regularization term ‖F−XTW‖2 to relax such a hard linear constraint in SDA that the low-dimensional representation F is constrained to lie in the linear subspace spanned by the data matrix X. We, however, observe that TR-FSDA may take some meaningless features in the iteration and cannot be always guaranteed to converge. In this paper, we propose a novel method for SDR, referred to as Recursive Dimension Reduction for Semisupervised Learning (RDS). Instead of solving the non-trivial TR problem using the iterative algorithm of TR-FSDA, we solve the objective function of TR-FSDA using a newly-developed recursive procedure. In each iteration, only a projection vector and a one-dimensional data representation are produced by solving a standard Rayleigh Quotient problem. Our algorithm escapes from the convergence guarantee, since it directly solves the objective and requires no any iterative strategy in finding each of the projection vectors. The experiments on four face databases, one object database, one shape image database, and one Handwritten Digit database demonstrate the effectiveness of RDS.

Learning from normalized local and global discriminative information for semi-supervised regression and dimensionality reduction

Semi-supervised dimensionality reduction is one of the important topics in pattern recognition and machine learning. During the past decade, Laplacian Regularized Least Square (LapRLS) and Semi-supervised Discriminant Analysis (SDA) are the two widely-used semi-supervised dimensionality reduction methods. In this paper, we show that SDA and LapRLS can be unified into a constrained manifold regularized least square framework. The manifold term, however, cannot fully utilize the underlying discriminative information. We thus introduce a new and effective semi-supervised dimensionality reduction method, called Learning from Local and Global Information (LLGDI), to solve the problem. The proposed LLGDI method adopts a set of local classification functions to preserve both local geometrical and discriminative information of dataset. It also adopts a global classification function to preserve the global discriminative information, and an uncorrelated constraint to calculate the projection matrix for simultaneously solving regression and dimensionality reduction problem. As a result, the LLGDI method is able to preserve local discriminative, manifold information as well as the global discriminative information. Theoretical analysis and extensive simulations presented in the paper show the effectiveness of the LLGDI algorithm. The results also demonstrate LLGDI can achieve superior performance compared with other existing methods.

Adaptive semi-supervised dimensionality reduction based on pairwise constraints weighting and graph optimizing

With the rapid growth of high dimensional data, dimensionality reduction is playing a more and more important role in practical data processing and analysing tasks. This paper studies semi-supervised dimensionality reduction using pairwise constraints. In this setting, domain knowledge is given in the form of pairwise constraints, which specifies whether a pair of instances belong to the same class (must-link constraint) or different classes (cannot-link constraint). In this paper, a novel semi-supervised dimensionality reduction method called adaptive semi-supervised dimensionality reduction (ASSDR) is proposed, which can get the optimized low dimensional representation of the original data by adaptively adjusting the weights of the pairwise constraints and simultaneously optimizing the graph construction. Experiments on UCI classification and image recognition show that ASSDR is superior to many existing dimensionality reduction methods.

A Reconstruction Error Based Framework for Multi-Label and Multi-View Learning

A significant challenge to make learning techniques more suitable for general purpose use is to move beyond i) complete supervision, ii) low dimensional data, iii) a single label and single view per instance. Solving these challenges allows working with complex learning problems that are typically high dimensional with multiple (but possibly incomplete) labelings and views. While other work has addressed each of these problems separately, in this paper we show how to address them together, namely <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">semi-supervised dimension reduction for multi-label and multi-view learning</i> (SSDR-MML), which performs optimization for dimension reduction and label inference in semi-supervised setting. The proposed framework is designed to handle both multi-label and multi-view learningsettings, and can be easily extended to many useful applications. Our formulation has a number of advantages. We explicitly model the information combining mechanism as a data structure (a weight/nearest-neighbor matrix) which allows investigating fundamentalquestions in multi-label and multi-view learning. We address one such question by presenting a general measure to quantify thesuccess of simultaneous learning of multiple labels or views. We empirically demonstrate the usefulness of our SSDR-MML approach, and show that it can outperform many state-of-the-art baseline methods.

Discriminative Spectral–Spatial Margin-Based Semisupervised Dimensionality Reduction of Hyperspectral Data

The past few years have witnessed prosperity of spectral-spatial processing of hyperspectral images. In this letter, in order to determine the optimal projection subspace of spectrums, we define discriminate spectral-spatial margins (DSSMs) to reveal the local information of hyperspectral pixels and explore the global structures of both labeled and unlabeled data via low-rank representation (LRR). Heterogeneous and homogeneous spectral-spatial neighbors of hyperspectral pixels are used to define DSSMs. By maximizing the DSSM of hyperspectral data and casting an LRR manifold regularizer on finding better projection, both the local and global information of hyperspectral data can be well explored to determine more discriminative features. Some experiments are taken on several real hyperspectral data sets, and the results exhibit its efficiency and superiority to the counterparts, when only a small number of labeled samples are available.

Constraint Sparse Neighborhood Preserving Embedding

Neighborhood Preserving Embedding (NPE) and extensions of NPE are hot research topics of data mining at present. An algorithm called Constraint Sparse Neighborhood Preserving Embedding (CSNPE) for dimensionality reduc- tion is proposed in the paper. The algorithm firstly creates the local sparse reconstructive relation information of samples; then, exacts the pairwise constrain information of samples. Finally, projections are obtained by infusing the two kinds of information with linear weighted way. In contrast to existing semi-supervised dimensionality reduction algorithms on NPE, CSNPE is available with the following characteristics: 1) Sparse reconstruction of local neighborhood of samples cost little because the number of them is limited. 2) CSNPE inherits the great robustness from sparse learning. 3) CSNPE infuses pairwise constrain information with weighted, preserving more discriminant information and local neighborhood sparse reconstruction information. Experiments conducted on real word facial databases demonstrate the effectiveness of the proposed algorithm.

Semi-Supervised Dimensionality Reduction of Hyperspectral Image Based on Sparse Multi-Manifold Learning

In this paper, we proposed a new semi-supervised multi-manifold learning method, called semi- supervised sparse multi-manifold embedding (S3MME), for dimensionality reduction of hyperspectral image data. S3MME exploits both the labeled and unlabeled data to adaptively find neighbors of each sample from the same manifold by using an optimization program based on sparse representation, and naturally gives relative importance to the labeled ones through a graph-based methodology. Then it tries to extract discriminative features on each manifold such that the data points in the same manifold become closer. The effectiveness of the proposed multi-manifold learning algorithm is demonstrated and compared through experiments on a real hyperspectral images.

Image classification via least square semi-supervised discriminant analysis with flexible kernel regression for out-of-sample extension

Semi-supervised dimensionality reduction is an important research topic in many pattern recognition and machine learning applications. Among all the methods for semi-supervised dimensionality reduction, SDA and LapRLS/L are two popular ones. Though the two methods are actually the extensions of different supervised methods, we show in this paper that both SDA and Lap-RLS/L can be unified under a regularized least square framework. In this paper, we propose a new effective semi-supervised dimensionality reduction method for better cope with data sampled from nonlinear manifold. In addition, the proposed method can both handle the regression as well as the subspace learning problem. Theoretical analysis and extensive simulations show the effectiveness of our algorithm. The results in simulations demonstrate that our proposed algorithm can achieve great superiority compared with other existing methods.

Relative manifold based semi-supervised dimensionality reduction

A well-designed graph plays a fundamental role in graph-based semi-supervised learning; however, the topological structure of a constructed neighborhood is unstable in most current approaches, since they are very sensitive to the high dimensional, sparse and noisy data. This generally leads to dramatic performance degradation. To deal with this issue, we developed a relative manifold based semi-supervised dimensionality reduction (RMSSDR) approach by utilizing the relative manifold to construct a better neighborhood graph with fewer short-circuit edges. Based on the relative cognitive law and manifold distance, a relative transformation is used to construct the relative space and the relative manifold. A relative transformation can improve the ability to distinguish between data points and reduce the impact of noise such that it may be more intuitive, and the relative manifold can more truly reflect the manifold structure since data sets commonly exist in a nonlinear structure. Specifically, RMSSDR makes full use of pairwise constraints that can define the edge weights of the neighborhood graph by minimizing the local reconstruction error and can preserve the global and local geometric structures of the data set. The experimental results on face data sets demonstrate that RMSSDR is better than the current state of the art comparing methods in both performance of classification and robustness.

Pairwise Costs in Semisupervised Discriminant Analysis for Face Recognition

In recent years, face recognition is being recognized as a cost-sensitive learning problem. Many cost-sensitive classifiers have been proposed. However, no sufficient attention is paid to the research on cost-sensitive dimensionality reduction, especially on the cost-sensitive semisupervised dimensionality reduction. To the best of our knowledge, cost sensitive semisupervised discriminant analysis (CS3DA) may be the first work. CS3DA first uses the sparse representation to infer a soft label for unlabeled sample and then learns the projection direction by incorporating misclassification costs into both labeled and unlabeled data. Although CS3DA reduces the loss of misclassification, it has two major drawbacks: 1) the sparsity is not a feature of face recognition, and therefore sparse approximations may not deliver the robustness or performance desired and 2) CS3DA is not proven to satisfy the minimal misclassification loss criterion. In this paper, we embed pairwise costs in semisupervised discriminant analysis (PCSDA) for face recognition. PCSDA first uses a simple \(l_{2}\) approach to predict the label of unlabeled data, and then learns the projection direction by embedding pairwise costs in both labeled and unlabeled data. Compared with CS3DA, PCSDA has three major advantages: 1) \(l_{2}\) approach is more accurate and robust than sparse representation for face recognition; 2) we prove that CS3DA approximates the pairwise Bayesian risk only when the classes are balanced and without outliers in face data sets; and 3) PCSDA approximates the pairwise Bayesian risk considering the class imbalance problem and outliers in face recognition. Hence, the projection direction obtained by using PCSDA can be more discriminative, immunes to outliers and class imbalance problem. The experimental results on AR, PIE, ORL, and extended Yale B data sets demonstrate the effectiveness of PCSDA.

Feature Extraction from Hyperspectral Remote Sensing Imagery based on Semisupervised Dimensionality Reduction with Pairwise Constraints

Feature Extraction from Hyperspectral Remote Sensing Imagery based on Semisupervised Dimensionality Reduction with Pairwise Constraints