Semi-supervised Dimensionality Reduction Research Articles

Cost-Sensitive Semi-Supervised Discriminant Analysis for Face Recognition

This paper presents a cost-sensitive semi-supervised discriminant analysis method for face recognition. While a number of semi-supervised dimensionality reduction algorithms have been proposed in the literature and successfully applied to face recognition in recent years, most of them aim to seek low-dimensional feature representations to achieve low classification errors and assume the same loss from all misclassifications in the feature representation/extraction phase. In many real-world face recognition applications, however, this assumption may not hold as different misclassifications could lead to different losses. For example, it may cause inconvenience to a gallery person who is misrecognized as an impostor and not allowed to enter the room by a face recognition-based door locker, but it could result in a serious loss or damage if an impostor is misrecognized as a gallery person and allowed to enter the room. Motivated by this concern, we propose in this paper a new method to learn a discriminative feature subspace by making use of both labeled and unlabeled samples and exploring different cost information of all the training samples simultaneously. Experimental results are presented to demonstrate the efficacy of the proposed method.

앙상블 접근법을 이용한 반감독 차원 감소 방법

While LDA is a supervised dimension reduction method which finds projective directions to maximize separability between classes, the performance of LDA is severely degraded when the number of labeled data is small. Recently semi-supervised dimension reduction methods have been proposed which utilize abundant unlabeled data and overcome the shortage of labeled data. However, matrix computation usually used in statistical dimension reduction methods becomes hindrance to make the utilization of a large number of unlabeled data difficult, and moreover too much information from unlabeled data may not so helpful compared to the increase of its processing time. In order to solve these problems, we propose an ensemble approach for semi-supervised dimension reduction. Extensive experimental results in text classification demonstrates the effectiveness of the proposed method.

Semi-supervised dimension reduction using trace ratio criterion.

In this brief, we address the trace ratio (TR) problem for semi-supervised dimension reduction. We first reformulate the objective function of the recent work semi-supervised discriminant analysis (SDA) in a TR form. We also observe that in SDA the low-dimensional data representation F is constrained to be in the linear subspace spanned by the training data matrix X (i.e., F = X(T) W). In order to relax this hard constraint, we introduce a flexible regularizer ||F - X(T) W||(2) which models the regression residual into the reformulated objective function. With such relaxation, our method referred to as TR based flexible SDA (TR-FSDA) can better cope with data sampled from a certain type of nonlinear manifold that is somewhat close to a linear subspace. In order to address the non-trivial optimization problem in TR-FSDA, we further develop an iterative algorithm to simultaneously solve for the low-dimensional data representation F and the projection matrix W. Moreover, we theoretically prove that our iterative algorithm converges to the optimum based on the Newton-Raphson method. The experiments on two face databases, one shape image database and one webpage database demonstrate that TR-FSDA outperforms the existing semi-supervised dimension reduction methods.

Patch Distribution Compatible Semisupervised Dimension Reduction for Face and Human Gait Recognition

We propose a new semisupervised learning algorithm, referred to as patch distribution compatible semisupervised dimension reduction, for face and human gait recognition. Each image (a face image or an average human silhouette image) is first represented as a set of local patch features and it is further characterized as the corresponding patch distribution feature, which can be expressed as an image-specific Gaussian mixture model (GMM) adapted from the universal background model. Assuming that the individual components of the image-specific GMMs from all the training images reside on a submanifold, we assign a component-level prediction label matrix to each individual GMM component and introduce a new regularizer based on a set of local submanifold smoothness assumptions in our objective function. We also constrain each component-level prediction label matrix to be consistent with the image-level prediction label matrix , as well as enforce to be close to the given labels for the labeled samples. We further use a linear regression function to provide embeddings for the training data and the unseen test data. Inspired by the recent work flexible manifold embedding, we additionally integrate the regression residue in our objective function to measure the mismatch between and , such that our method can better cope with the data sampled from a nonlinear manifold. Finally, the optimal solutions of the component-level prediction label matrix , the image-level prediction label matrix , the projection matrix , and the bias term b can be simultaneously obtained. Comprehensive experiments on three benchmark face databases CMU PIE, FERET, and AR as well as the USF HumanID gait database clearly demonstrate the effectiveness of our algorithm over other state-of-the-art semisupervised dimension reduction methods.

Dimensionality reduction via compressive sensing

Compressive sensing is an emerging field predicated upon the fact that, if a signal has a sparse representation in some basis, then it can be almost exactly reconstructed from very few random measurements. Many signals and natural images, for example under the wavelet basis, have very sparse representations, thus those signals and images can be recovered from a small amount of measurements with very high accuracy. This paper is concerned with the dimensionality reduction problem based on the compressive assumptions. We propose novel unsupervised and semi-supervised dimensionality reduction algorithms by exploiting sparse data representations. The experiments show that the proposed approaches outperform state-of-the-art dimensionality reduction methods.

A General Model for Semi-Supervised Dimensionality Reduction

This paper focuses on semi-supervised dimensionality reduction. In this scenario, we present a general model for semi-supervised dimensionality reduction with pairwise constraints (SSPC). Through defining a discriminant adjacent matrix, SSPC learns a projection embedding the data from the original space to the low-dimensional space such that intra-cluster instances become even more nearby while extra-cluster instances become as far away from each other as possible. Experimental results on a collection of benchmark data sets show that SSPC is superior to many established dimensionality reduction methods.

Semi-Supervised Locality Discriminant Projection

In this paper, we consider the problem of semi-supervised dimensionality reduction. We focus on the local geometric structure of data and propose a novel method, called Semi-supervised Locality Discriminant Projections (SSLDP). It uses both labeled and unlabeled samples. Specifically, the labeled samples are used to explore the discriminating ainformation including both similarity and dissimilarity information, while the unlabeled samples are used to estimate the intrinsic geometric structure of data. Thus, SSLDP learns a discriminant projection which can best preserve both the discriminating structure and the local geometric structure of data. We evaluate SSLDP in the similarity measure which plays a key role in most of the information processing tasks. The experimental results show the effectiveness of our algorithm.

Linear Semi-Supervised Dimensionality Reduction with Pairwise Constraint for Multiple Subclasses

We propose a new method, called Subclass-oriented Dimensionality Reduction with Pairwise Constraints (SODRPaC), for dimensionality reduction. In a high dimensional space, it is common that a group of data points with one class may scatter in several different groups. Current linear semi-supervised dimensionality reduction methods would fail to achieve fair performances, as they assume two data points linked by a must-link constraint are close each other, while they are likely to be located in different groups. Inspired by the above observation, we classify the must-link constraint into two categories, which are the inter-subclass must-link constraint and the intra-subclass must-link constraint, respectively. We carefully generate cannot-link constraints by using must-link constraints, and then propose a new discriminant criterion by employing the cannot-link constraints and the compactness of shared nearest neighbors. The manifold regularization is also incorporated in our dimensionality reduction framework. Extensive experiments on both synthetic and practical data sets illustrate the effectiveness of our method.

Self-adaptive local Fisher discriminant analysis for semi-supervised image recognition

In this paper, we present a self-adaptive semi-supervised dimension reduction framework for image class recognition. Compared with the Semi-supervised Local Fisher Discriminant Analysis (SELF), whose classification performance is significantly influenced by some parameters - Neighbour size k of every labelled sample, Affinity Matrix A = {A

Locality Sensitive Semi-Supervised Dimensionality Reduction on Multimodal Data

A special kind of data is considered in this paper called multimodal data. It has the property that samples in a class are from several separate clusters. Locality Preserving Projection (LPP) can work well with multimodal data due to its locality preserving property. However, the label information is not used to improve the learning performance due to the unsupervised character of LPP. In this paper, we propose a method called Locality Sensitive Semi-Supervised Dimensionality Reduction (semi-LSDR). It takes both the discriminant information and geometry structure into account. Specifically, we construct a between-class graph on labeled samples and a nearest neighbor graph both from the perspective of locality. A directly mapping can be achieved by solving a generalized eigenvalue problem. Effectiveness of the proposed method is showed through simulations with benchmark data sets.

Trace ratio criterion based generalized discriminative learning for semi-supervised dimensionality reduction

Dealing with high-dimensional data has always been a major problem in many pattern recognition and machine learning applications. Trace ratio criterion is a criterion that can be applicable to many dimensionality reduction methods as it directly reflects Euclidean distance between data points of within or between classes. In this paper, we analyze the trace ratio problem and propose a new efficient algorithm to find the optimal solution. Based on the proposed algorithm, we are able to derive an orthogonal constrained semi-supervised learning framework. The new algorithm incorporates unlabeled data into training procedure so that it is able to preserve the discriminative structure as well as geometrical structure embedded in the original dataset. Under such a framework, many existing semi-supervised dimensionality reduction methods such as SDA, Lap-LDA, SSDR, SSMMC, can be improved using our proposed framework, which can also be used to formulate a corresponding kernel framework for handling nonlinear problems. Theoretical analysis indicates that there are certain relationships between linear and nonlinear methods. Finally, extensive simulations on synthetic dataset and real world dataset are presented to show the effectiveness of our algorithms. The results demonstrate that our proposed algorithm can achieve great superiority to other state-of-art algorithms.

Semi-supervised classification based on random subspace dimensionality reduction

Graph structure is vital to graph based semi-supervised learning. However, the problem of constructing a graph that reflects the underlying data distribution has been seldom investigated in semi-supervised learning, especially for high dimensional data. In this paper, we focus on graph construction for semi-supervised learning and propose a novel method called Semi-Supervised Classification based on Random Subspace Dimensionality Reduction, SSC-RSDR in short. Different from traditional methods that perform graph-based dimensionality reduction and classification in the original space, SSC-RSDR performs these tasks in subspaces. More specifically, SSC-RSDR generates several random subspaces of the original space and applies graph-based semi-supervised dimensionality reduction in these random subspaces. It then constructs graphs in these processed random subspaces and trains semi-supervised classifiers on the graphs. Finally, it combines the resulting base classifiers into an ensemble classifier. Experimental results on face recognition tasks demonstrate that SSC-RSDR not only has superior recognition performance with respect to competitive methods, but also is robust against a wide range of values of input parameters.

Semi-supervised dimensionality reduction for analyzing high-dimensional data with constraints

In this paper, we present a novel semi-supervised dimensionality reduction technique to address the problems of inefficient learning and costly computation in coping with high-dimensional data. Our method named the dual subspace projections (DSP) embeds high-dimensional data in an optimal low-dimensional space, which is learned with a few user-supplied constraints and the structure of input data. The method projects data into two different subspaces respectively the kernel space and the original input space. Each projection is designed to enforce one type of constraints and projections in the two subspaces interact with each other to satisfy constraints maximally and preserve the intrinsic data structure. Compared to existing techniques, our method has the following advantages: (1) it benefits from constraints even when only a few are available; (2) it is robust and free from overfitting; and (3) it handles nonlinearly separable data, but learns a linear data transformation. As a conclusion, our method can be easily generalized to new data points and is efficient in dealing with large datasets. An empirical study using real data validates our claims so that significant improvements in learning accuracy can be obtained after the DSP-based dimensionality reduction is applied to high-dimensional data.

Experimental Comparisons of Semi-Supervised Dimensional Reduction Methods

半监督学习是近年来机器学习领域中的研究热点之一,已从最初的半监督分类和半监督聚类拓展到半监督回归和半监督降维等领域.目前,有关半监督分类、聚类和回归等方面的工作已经有了很好的综述,如Zhu 的半监督学习文献综述.降维一直是机器学习和模式识别等相关领域的重要研究课题,近年来出现了很多将半监督思想用于降维,即半监督降维方面的工作.有鉴于此,试图对目前已有的一些半监督降维方法进行综述,然后在大量的标准数据集上对这些方法的性能进行实验比较,并据此得出了一些经验性的启示.;Semi-Supervised learning is one of the hottest research topics in the technological community, which has been developed from the original semi-supervised classification and semi-supervised clustering to the semi-supervised regression and semi-supervised dimensionality reduction, etc. At present, there have been several excellent surveys on semi-supervised classification: Semi-Supervised clustering and semi-supervised regression, e.g. Zhu’s semi-supervised learning literature survey. Dimensionality reduction is one of the key issues in machine learning, pattern recognition, and other related fields. Recently, a lot of research has been done to integrate the idea of semi-supervised learning into dimensionality reduction, i.e. semi-supervised dimensionality reduction. In this paper, the current semi-supervised dimensionality reduction methods are reviewed, and their performances are evaluated through extensive experiments on a large number of benchmark datasets, from which some empirical insights can be obtained.

DISEASE CLASSIFICATION AND PREDICTION VIA SEMI-SUPERVISED DIMENSIONALITY REDUCTION.

We present a new semi-supervised algorithm for dimensionality reduction which exploits information of unlabeled data in order to improve the accuracy of image-based disease classification based on medical images. We perform dimensionality reduction by adopting the formalism of constrained matrix decomposition of [1] to semi-supervised learning. In addition, we add a new regularization term to the objective function to better captur the affinity between labeled and unlabeled data. We apply our method to a data set consisting of medical scans of subjects classified as Normal Control (CN) and Alzheimer (AD). The unlabeled data are scans of subjects diagnosed with Mild Cognitive Impairment (MCI), which are at high risk to develop AD in the future. We measure the accuracy of our algorithm in classifying scans as AD and NC. In addition, we use the classifier to predict which subjects with MCI will converge to AD and compare those results to the diagnosis given at later follow ups. The experiments highlight that unlabeled data greatly improves the accuracy of our classifier.

Semisupervised Dimensionality Reduction With Pairwise Constraints for Hyperspectral Image Classification

Dimensionality reduction is an important task in the analysis of hyperspectral image data. While traditional dimensionality reduction methods use class labels as prior information, this letter presents a general semisupervised dimensionality reduction framework for hyperspectral image classification based on new prior information, i.e., pairwise constraints which specify whether a pair of examples belongs to the same class or not. The proposed semisupervised dimensionality reduction framework contains two terms: 1) a discrimination term that assesses the separability between classes; and 2) a regularization term that characterizes some property of the original data set. Furthermore, a novel semisupervised dimensionality reduction method is derived from the framework based on sparse representation. Experimental results on two hyperspectral image data sets show that the proposed algorithms are remarkably effective in comparison to traditional dimensionality reduction methods.

Local margin based semi-supervised discriminant embedding for visual recognition

Most manifold learning algorithms adopt the k nearest neighbors function to construct the adjacency graph. However, severe bias may be introduced in this case if the samples are not uniformly distributed in the ambient space. In this paper a semi-supervised dimensionality reduction method is proposed to alleviate this problem. Based on the notion of local margin, we simultaneously maximize the separability between different classes and estimate the intrinsic geometric structure of the data by both the labeled and unlabeled samples. For high-dimensional data, a discriminant subspace is derived via maximizing the cumulative local margins. Experimental results on high-dimensional classification tasks demonstrate the efficacy of our algorithm.

Mixture graph based semi-supervised dimensionality reduction

Graph structure is crucial to graph based dimensionality reduction. A mixture graph based semi-supervised dimensionality reduction (MGSSDR) method with pairwise constraints is proposed. MGSSDR first constructs multiple diverse graphs on different random subspaces of dataset, then it combines these graphs into a mixture graph and does dimensionality reduction on this mixture graph. MGSSDR can preserve the pairwise constraints and local structure of samples in the reduced subspace. Meanwhile, it is robust to noise and neighborhood size. Experimental results on facial images feature extraction demonstrate its effectiveness.

Semisupervised Dimensionality Reduction and Classification Through Virtual Label Regression

Semisupervised dimensionality reduction has been attracting much attention as it not only utilizes both labeled and unlabeled data simultaneously, but also works well in the situation of out-of-sample. This paper proposes an effective approach of semisupervised dimensionality reduction through label propagation and label regression. Different from previous efforts, the new approach propagates the label information from labeled to unlabeled data with a well-designed mechanism of random walks, in which outliers are effectively detected and the obtained virtual labels of unlabeled data can be well encoded in a weighted regression model. These virtual labels are thereafter regressed with a linear model to calculate the projection matrix for dimensionality reduction. By this means, when the manifold or the clustering assumption of data is satisfied, the labels of labeled data can be correctly propagated to the unlabeled data; and thus, the proposed approach utilizes the labeled and the unlabeled data more effectively than previous work. Experimental results are carried out upon several databases, and the advantage of the new approach is well demonstrated.

Distance metric learning guided adaptive subspace semi-supervised clustering

Most existing semi-supervised clustering algorithms are not designed for handling high-dimensional data. On the other hand, semi-supervised dimensionality reduction methods may not necessarily improve the clustering performance, due to the fact that the inherent relationship between subspace selection and clustering is ignored. In order to mitigate the above problems, we present a semi-supervised clustering algorithm using adaptive distance metric learning (SCADM) which performs semi-supervised clustering and distance metric learning simultaneously. SCADM applies the clustering results to learn a distance metric and then projects the data onto a low-dimensional space where the separability of the data is maximized. Experimental results on real-world data sets show that the proposed method can effectively deal with high-dimensional data and provides an appealing clustering performance.