Supervised Dimensionality Reduction Research Articles

Neighborhood discriminant tensor mapping

Linear dimensionality reduction (feature extraction) methods have been widely used in computer vision and pattern recognition. Two of the most representative methods are principal component analysis ( PCA) and linear discriminant analysis ( LDA). However, when dealing with a multidimensional dataset of dimension R m 1 ⊗ R m 1 ⊗ ⋯ ⊗ R m N (e.g. for images N = 2 , videos N = 3 ), these methods usually first transform the original data to high dimensional vectors in R m 1 × m 2 × ⋯ × m N , and then analyze the data in such a high dimensional space. In this paper, we propose a supervised dimensionality reduction method called neighborhood discriminative tensor mapping ( NDTM), which can directly process the multidimensional data as tensors. Moreover, NDTM can make use of the local information of the dataset to achieve a better classification result. Experimental results on face recognition show the superiority of our algorithm to traditional dimensionality reduction methods.

Kernel Uncorrelated and Regularized Discriminant Analysis: A Theoretical and Computational Study

Linear and kernel discriminant analyses are popular approaches for supervised dimensionality reduction. Uncorrelated and regularized discriminant analyses have been proposed to overcome the singularity problem encountered by classical discriminant analysis. In this paper, we study the properties of kernel uncorrelated and regularized discriminant analyses, called KUDA and KRDA, respectively. In particular, we show that under a mild condition, both linear and kernel uncorrelated discriminant analysis project samples in the same class to a common vector in the dimensionality-reduced space. This implies that uncorrelated discriminant analysis may suffer from the overfitting problem if there are a large number of samples in each class. We show that as the regularization parameter in KRDA tends to zero, KRDA approaches KUDA. This shows that KUDA is a special case of KRDA and that regularization can be applied to overcome the overfitting problem in uncorrelated discriminant analysis. As the performance of KRDA depends on the value of the regularization parameter, we show that the matrix computations involved in KRDA can be simplified, so that a large number of candidate values can be cross-validated efficiently. Finally, we conduct experiments to evaluate the proposed theories and algorithms.

Generalized Linear Discriminant Analysis: A Unified Framework and Efficient Model Selection

High-dimensional data are common in many domains, and dimensionality reduction is the key to cope with the curse-of-dimensionality. Linear discriminant analysis (LDA) is a well-known method for supervised dimensionality reduction. When dealing with high-dimensional and low sample size data, classical LDA suffers from the singularity problem. Over the years, many algorithms have been developed to overcome this problem, and they have been applied successfully in various applications. However, there is a lack of a systematic study of the commonalities and differences of these algorithms, as well as their intrinsic relationships. In this paper, a unified framework for generalized LDA is proposed, which elucidates the properties of various algorithms and their relationships. Based on the proposed framework, we show that the matrix computations involved in LDA-based algorithms can be simplified so that the cross-validation procedure for model selection can be performed efficiently. We conduct extensive experiments using a collection of high-dimensional data sets, including text documents, face images, gene expression data, and gene expression pattern images, to evaluate the proposed theories and algorithms.

Semantic Coding by Supervised Dimensionality Reduction

This paper addresses the problem of representing multimedia information under a compressed form that permits efficient classification. The semantic coding problem starts from a subspace method where dimensionality reduction is formulated as a matrix factorization problem. Data samples are jointly represented in a common subspace extracted from a redundant dictionary of basis functions. We first build on greedy pursuit algorithms for simultaneous sparse approximations to solve the dimensionality reduction problem. The method is extended into a supervised algorithm, which further encourages the class separability in the extraction of the most relevant features. The resulting supervised dimensionality reduction scheme provides an interesting tradeoff between approximation (or compression) and discriminant feature extraction (or classification). The algorithm provides a compressed signal representation that can directly be used for multimedia data mining. The application of the proposed algorithm to image recognition problems further demonstrates classification performances that are competitive with state-of-the-art solutions in handwritten digit or face recognition. Semantic coding certainly represents an interesting solution to the challenging problem of processing huge volumes of multidimensional data in modern multimedia systems, where compressed data have to be processed and analyzed with limited computational complexity.

Supervised dimensionality reduction via sequential semidefinite programming

Many dimensionality reduction problems end up with a trace quotient formulation. Since it is difficult to directly solve the trace quotient problem, traditionally the trace quotient cost function is replaced by an approximation such that the generalized eigenvalue decomposition can be applied. In contrast, we directly optimize the trace quotient in this work. It is reformulated as a quasi-linear semidefinite optimization problem, which can be solved globally and efficiently using standard off-the-shelf semidefinite programming solvers. Also this optimization strategy allows one to enforce additional constraints (for example, sparseness constraints) on the projection matrix. We apply this optimization framework to a novel dimensionality reduction algorithm. The performance of the proposed algorithm is demonstrated in experiments by several UCI machine learning benchmark examples, USPS handwritten digits as well as ORL and Yale face data.

Extracting the optimal dimensionality for local tensor discriminant analysis

Supervised dimensionality reduction with tensor representation has attracted great interest in recent years. It has been successfully applied to problems with tensor data, such as image and video recognition tasks. However, in the tensor-based methods, how to select the suitable dimensions is a very important problem. Since the number of possible dimension combinations exponentially increases with respect to the order of tensor, manually selecting the suitable dimensions becomes an impossible task in the case of high-order tensor. In this paper, we aim at solving this important problem and propose an algorithm to extract the optimal dimensionality for local tensor discriminant analysis. Experimental results on a toy example and real-world data validate the effectiveness of the proposed method.

Augmenting the bootstrap to analyze high dimensional genomic data

The data produced by high-throughput genomic techniques are often high dimensional and undersampled. In these settings, statistical analyses that require the inversion of covariance matrices, such as those pursuing supervised dimension reduction or the assessment of interdependence structures, are problematic. In this article we show how the idea of adding noise to the bootstrap, pioneered by Efron, and Silverman and Young, in the late seventies and eighties, can be used to overcome undersampling and effectively estimate the inverse covariance matrix for data sets in which the number of observations is small relative to the number of variables. We demonstrate the performance of this approach, which we call augmented bootstrap, on simulated data and on data derived from genomic DNA sequences and microarray experiments.

Face Recognition Using Classification-Based Linear Projections

Subspace methods have been successfully applied to face recognition tasks. In this study we propose a face recognition algorithm based on a linear subspace projection. The subspace is found via utilizing a variant of the neighbourhood component analysis (NCA) algorithm which is a supervised dimensionality reduction method that has been recently introduced. Unlike previously suggested supervised subspace methods, the algorithm explicitly utilizes the classification performance criterion to obtain the optimal linear projection. In addition to its feature extraction capabilities, the algorithm also finds the optimal distance-metric that should be used for face-image retrieval in the transformed space. The proposed face-recognition technique significantly outperforms traditional subspace-based approaches particulary in very low-dimensional representations. The method performance is demonstrated across a range of standard face databases.

Incremental Linear Discriminant Analysis for Face Recognition

Dimensionality reduction methods have been successfully employed for face recognition. Among the various dimensionality reduction algorithms, linear (Fisher) discriminant analysis (LDA) is one of the popular supervised dimensionality reduction methods, and many LDA-based face recognition algorithms/systems have been reported in the last decade. However, the LDA-based face recognition systems suffer from the scalability problem. To overcome this limitation, an incremental approach is a natural solution. The main difficulty in developing the incremental LDA (ILDA) is to handle the inverse of the within-class scatter matrix. In this paper, based on the generalized singular value decomposition LDA (LDA/GSVD), we develop a new ILDA algorithm called GSVD-ILDA. Different from the existing techniques in which the new projection matrix is found in a restricted subspace, the proposed GSVD-ILDA determines the projection matrix in full space. Extensive experiments are performed to compare the proposed GSVD-ILDA with the LDA/GSVD as well as the existing ILDA methods using the face recognition technology face database and the Carneggie Mellon University Pose, Illumination, and Expression face database. Experimental results show that the proposed GSVD-ILDA algorithm gives the same performance as the LDA/GSVD with much smaller computational complexity. The experimental results also show that the proposed GSVD-ILDA gives better classification performance than the other recently proposed ILDA algorithms.

Rank-One Projections With Adaptive Margins for Face Recognition

In supervised dimensionality reduction, tensor representations of images have recently been employed to enhance classification of high dimensional data with small training sets. Previous approaches for handling tensor data have been formulated with tight restrictions on projection directions that, along with convergence issues and the assumption of Gaussian-distributed class data, limit its face-recognition performance. To overcome these problems, we propose a method of rank-one projections with adaptive margins (RPAM) that gives a provably convergent solution for tensor data over a more general class of projections, while accounting for margins between samples of different classes. In contrast to previous margin-based works which determine margin sample pairs within the original high dimensional feature space, RPAM aims instead to maximize the margins defined in the expected lower dimensional feature sub-space by progressive margin refinement after each rank-one projection. In addition to handling tensor data, vector-based variants of RPAM are presented for linear mappings and for nonlinear mappings using kernel tricks. Comprehensive experimental results demonstrate that RPAM brings significant improvement in face recognition over previous subspace learning techniques.

Kernel logistic PLS: A tool for supervised nonlinear dimensionality reduction and binary classification

“Kernel logistic PLS” (KL-PLS) is a new tool for supervised nonlinear dimensionality reduction and binary classification. The principles of KL-PLS are based on both PLS latent variables construction and learning with kernels. The KL-PLS algorithm can be seen as a supervised dimensionality reduction (complexity control step) followed by a classification based on logistic regression. The algorithm is applied to 11 benchmark data sets for binary classification and to three medical problems. In all cases, KL-PLS proved its competitiveness with other state-of-the-art classification methods such as support vector machines. Moreover, due to successions of regressions and logistic regressions carried out on only a small number of uncorrelated variables, KL-PLS allows handling high-dimensional data. The proposed approach is simple and easy to implement. It provides an efficient complexity control by dimensionality reduction and allows the visual inspection of data segmentation.

Multilinear Discriminant Analysis for Face Recognition

There is a growing interest in subspace learning techniques for face recognition; however, the excessive dimension of the data space often brings the algorithms into the curse of dimensionality dilemma. In this paper, we present a novel approach to solve the supervised dimensionality reduction problem by encoding an image object as a general tensor of second or even higher order. First, we propose a discriminant tensor criterion, whereby multiple interrelated lower dimensional discriminative subspaces are derived for feature extraction. Then, a novel approach, called k-mode optimization, is presented to iteratively learn these subspaces by unfolding the tensor along different tensor directions. We call this algorithm multilinear discriminant analysis (MDA), which has the following characteristics: 1) multiple interrelated subspaces can collaborate to discriminate different classes, 2) for classification problems involving higher order tensors, the MDA algorithm can avoid the curse of dimensionality dilemma and alleviate the small sample size problem, and 3) the computational cost in the learning stage is reduced to a large extent owing to the reduced data dimensions in k-mode optimization. We provide extensive experiments on ORL, CMU PIE, and FERET databases by encoding face images as second- or third-order tensors to demonstrate that the proposed MDA algorithm based on higher order tensors has the potential to outperform the traditional vector-based subspace learning algorithms, especially in the cases with small sample sizes.

Computational and Theoretical Analysis of Null Space and Orthogonal Linear Discriminant Analysis

Dimensionality reduction is an important pre-processing step in many applications. Linear discriminant analysis (LDA) is a classical statistical approach for supervised dimensionality reduction. It...

Gene Expression Profile of Empirically Delineated Classes of Unexplained Chronic Fatigue

To identify the underlying gene expression profiles of unexplained chronic fatigue subjects classified into five or six class solutions by principal component (PCA) and latent class analyses (LCA). Microarray expression data were available for 15,315 genes and 111 female subjects enrolled from a population-based study on chronic fatigue syndrome. Algorithms were developed to assign gene scores and threshold values that signified the contribution of each gene to discriminate the multiclasses in each LCA solution. Unsupervised dimensionality reduction was first used to remove noise or otherwise uninformative gene combinations, followed by supervised dimensionality reduction to isolate gene combinations that best separate the classes. The authors' gene score and threshold algorithms identified 32 and 26 genes capable of discriminating the five and six multiclass solutions, respectively. Pair-wise comparisons suggested that some genes (zinc finger protein 350 [ZNF350], solute carrier family 1, member 6 [SLC1A6], F-box protein 7 [FBX07] and vacuole 14 protein homolog [VAC14]) distinguished most classes of fatigued subjects from healthy subjects, whereas others (patched homolog 2 [PTCH2] and T-cell leukemia/lymphoma [TCL1A]) differentiated specific fatigue classes. A computational approach was developed for general use to identify discriminatory genes in any multiclass problem. Using this approach, differences in gene expression were found to discriminate some classes of unexplained chronic fatigue, particularly one termed interoception.

Supervised dimension reduction of intrinsically low-dimensional data.

High-dimensional data generated by a system with limited degrees of freedom are often constrained in low-dimensional manifolds in the original space. In this article, we investigate dimension-reduction methods for such intrinsically low-dimensional data through linear projections that preserve the manifold structure of the data. For intrinsically one dimensional data, this implies projecting to a curve on the plane with as few intersections as possible. We are proposing a supervised projection pursuit method that can be regarded as an extension of the single-index model for nonparametric regression. We show results from a toy and two robotic applications.

Reduced rank vector generalized linear models for feature extraction

Supervised linear feature extraction can be achieved by fitting a reduced rank multivariate model. This paper studies rank penalized and rank constrained vector generalized linear models. From the perspective of thresholding rules, we build a framework for fitting singular value penalized models and use it for feature extraction. Through solving the rank constraint form of the problem, we propose progressive feature space reduction for fast computation in high dimensions with little performance loss. A novel projective cross-validation is proposed for parameter tuning in such nonconvex setups. Real data applications are given to show the power of the methodology in supervised dimension reduction and feature extraction.