Linear Dimensionality Reduction Methods Research Articles

Semi-supervised clustering of unknown expressions

Automated facial expressions recognition is an important area in computer vision and machine learning due to its important role in socially aware human-computer interaction and affective analysis. However, expression recognition is not trivial as it may vary across different genders, age groups and ethnicities, despite the universality of basic expressions. We propose here a semi-supervised learning algorithm with the motivation to cluster known, as well as unknown facial behavior. We detect and align faces and then reduce their dimensionality using various linear and non-linear, supervised and unsupervised methods. The transformed faces in new dimensions are then clustered with various methods, including but not limited to, consensus clustering, which is also used for automated class discovery in genetics. The linear dimensionality reduction methods, that we employ, include principal component analysis and linear discriminant analysis; while the non-linear methods include learning embeddings with Deep Convolution Neural Networks (CNNs). Deep CNNs can learn highly complicated non-linear transformations in a complex feature space. We get the best results using embeddings that we learn using deep convolution neural networks with consensus clustering. The novelty of our work is to show that we can cluster facial expressions, which were not included while learning the embeddings in the first place. This shows that we can generalize our work to non-standard expressions and can learn expression classes from the datasets themselves.

Data Visualization using Linear and Non-linear Dimensionality Reduction Methods

Data Visualization using Linear and Non-linear Dimensionality Reduction Methods

Graph-dual Laplacian principal component analysis

Principal component analysis is the most widely used method for linear dimensionality reduction, due to its effectiveness in exploring low-dimensional global geometric structures embedded in data. To preserve the intrinsic local geometrical structures of data, graph-Laplacian PCA (gLPCA) incorporates Laplacian embedding into PCA framework for learning local similarities between data points, which leads to significant performance improvement in clustering and classification. Some recent works showed that not only the high dimensional data reside on a low-dimensional manifold in the data space, but also the features lie on a manifold in feature space. However, both PCA and gLPCA overlook the local geometric information contained in the feature space. By considering the duality between data manifold and feature manifold, graph-dual Laplacian PCA (gDLPCA) is proposed, which incorporates data graph regularization and feature graph regularization into PCA framework to exploit local geometric structures of data manifold and feature manifold simultaneously. The experimental results on four benchmark data sets have confirmed its effectiveness and suggested that gDLPCA outperformed gLPCA on classification and clustering tasks.

Linear regression based projections for dimensionality reduction

In graph embedding based dimensionality reduction methods, the number of K-nearest neighbors is usually needed to be manually chosen in high dimensional space. Graph construction by different number of K-nearest neighbor changes dramatically and seriously affects the performance of graph embedding based dimensionality reduction. How to automatically construct a graph is very important. In this paper, first, a discriminative L2-graph is investigated. It computes the edge weights using the class-specific samples and weighted ridge regression, avoiding manually choosing the K-nearest neighbors in traditional graph construction. Second, a discriminative L2-graph based dimensionality reduction method is proposed, named Linear Regression based Projections (LRP). LRP minimizes the ratio between the local compactness information and the total separability information to seek the optimal projection matrix. LRP is much faster than its counterparts, Sparsity Preserving Projections (SPP) and Collaborative Representation based Projections (CRP), since LRP is supervised and computes edge weights using class-specific samples while SPP and CRP are unsupervised and compute edge weights using all samples. The experimental results on benchmark face image databases show that the proposed LRP outperforms many existing representative linear dimensionality reduction methods.

Chernoff information between Gaussian trees

In this paper, we deal with Gaussian graphical classifiaction problem. We aim to provide a systematic study of the relationship between Chernoff information and topological, as well as algebraic properties of the corresponding Gaussian graphs for the underlying graphical testing problems. We first show the relationship between Chernoff information and generalized eigenvalues of the associated covariance matrices. It is then proved that Chernoff information between two Gaussian trees sharing certain local subtree structures can be transformed into that of two smaller trees. In this way, we provide a sequence of equivalent Gaussian tree pairs in terms of Chernoff Information. Under our proposed grafting operations, bottleneck Gaussian trees, namely, Gaussian trees connected by one such operation, can thus be simplified into two 3-node Gaussian trees. Thereafter, we provide a thorough study about how Chernoff information changes when small differences are accumulated into bigger ones via concatenated grafting operations, as well as partial ordering. In the end, we propose an optimal linear dimension reduction method based on generalized eigenvalues for the purpose of classification, which is proved to achieve maximum Chernoff information among all linear transformations.

A label compression method for online multi-label classification

Many modern applications deal with multi-label data, such as functional categorizations of genes, image labeling and text categorization. Classification of such data with a large number of labels and latent dependencies among them is a challenging task, and it becomes even more challenging when the data is received online and in chunks. Many of the current multi-label classification methods require a lot of time and memory, which make them infeasible for practical real-world applications. In this paper, we propose a fast linear label space dimension reduction method that transforms the labels into a reduced encoded space and trains models on the obtained pseudo labels. Additionally, it provides an analytical method to update the decoding matrix which maps the labels into the original space and is used during the test phase. Experimental results show the effectiveness of this approach in terms of running times and the prediction performance over different measures.

SICA: subjectively interesting component analysis

The information in high-dimensional datasets is often too complex for human users to perceive directly. Hence, it may be helpful to use dimensionality reduction methods to construct lower dimensional representations that can be visualized. The natural question that arises is how do we construct a most informative low dimensional representation? We study this question from an information-theoretic perspective and introduce a new method for linear dimensionality reduction. The obtained model that quantifies the informativeness also allows us to flexibly account for prior knowledge a user may have about the data. This enables us to provide representations that are subjectively interesting. We title the method Subjectively Interesting Component Analysis (SICA) and expect it is mainly useful for iterative data mining. SICA is based on a model of a user’s belief state about the data. This belief state is used to search for surprising views. The initial state is chosen by the user (it may be empty up to the data format) and is updated automatically as the analysis progresses. We study several types of prior beliefs: if a user only knows the scale of the data, SICA yields the same cost function as Principal Component Analysis (PCA), while if a user expects the data to have outliers, we obtain a variant that we term t-PCA. Finally, scientifically more interesting variants are obtained when a user has more complicated beliefs, such as knowledge about similarities between data points. The experiments suggest that SICA enables users to find subjectively more interesting representations.

Scaled radial axes for interactive visual feature selection: A case study for analyzing chronic conditions

In statistics, machine learning, and related fields, feature selection is the process of choosing a smaller subset of features to work with. This is an important topic since selecting a subset of features can help analysts to interpret models and data, and to decrease computational runtimes. While many techniques are purely automatic, the data visualization community has produced a number of interactive approaches where users can make decisions taking into account their domain knowledge. In this paper we propose a new visualization technique based on radial axes that allows analysts to perform feature selection effectively, in contrast to previous radial axes methods. This is achieved by employing alternative scaled axes that provide insight regarding the features that have a smaller contribution to the visualizations. Therefore, analysts can use the technique to carry out interactive backwards feature elimination, by discarding the least relevant features according to the information on the plots and their expertise. Our approach can be coupled with any linear dimensionality reduction method, and can be used when performing analyses of cluster structure, correlations, class separability, etc. Specifically, in this paper we focus on combining the proposed technique with methods designed for classification. Lastly, we illustrate the effectiveness of our proposal through a case study analyzing high-dimensional medical chronic conditions data. In particular, clinicians have used the technique for determining the most important features that discriminate between patients with diabetes and high blood pressure.

An overview of incremental feature extraction methods based on linear subspaces

With the massive explosion of machine learning in our day-to-day life, incremental and adaptive learning has become a major topic, crucial to keep up-to-date and improve classification models and their corresponding feature extraction processes. This paper presents a categorized overview of incremental feature extraction based on linear subspace methods which aim at incorporating new information to the already acquired knowledge without accessing previous data. Specifically, this paper focuses on those linear dimensionality reduction methods with orthogonal matrix constraints based on global loss function, due to the extensive use of their batch approaches versus other linear alternatives. Thus, we cover the approaches derived from Principal Components Analysis, Linear Discriminative Analysis and Discriminative Common Vector methods. For each basic method, its incremental approaches are differentiated according to the subspace model and matrix decomposition involved in the updating process. Besides this categorization, several updating strategies are distinguished according to the amount of data used to update and to the fact of considering a static or dynamic number of classes. Moreover, the specific role of the size/dimension ratio in each method is considered. Finally, computational complexity, experimental setup and the accuracy rates according to published results are compiled and analyzed, and an empirical evaluation is done to compare the best approach of each kind.

Comparative study of description algorithms for complex-valued gradient fields of digital images using linear dimensionality reduction methods

The paper presents an analysis of various approaches to constructing descriptions for the gradient fields of digital images. The analyzed approaches are based on the well-known methods for data dimensionality reduction, such as Principal (PCA) and Independent (ICA) Component Analysis, Linear Discriminant Analysis (LDA). We apply these methods not to the original image, represented as a two-dimensional field of brightness (a halftone image), but to its secondary representation in the form of a two-dimensional gradient field, that is, a complex-valued image. In this case, approaches based on using both the entire gradient field and only its phase component are considered. In addition, two independent ways of forming the final description of the original object are considered: using expansion coefficients of the gradient field in a derived basis and using an original authors' design that is called model-oriented descriptors. With the latter, the number of real coefficients used in the description of the original object can be halved. The studies are conducted via solving a face recognition problem. The effectiveness of the analyzed methods is demonstrated by applying them to images from Extended Yale Face Database B. The comparison is made using a nearest neighbor's classifier.

Study of feature extraction method of multi-information source for continuous casting process parameters

The control of the roll gap of the segment is one of the key links to ensure the quality of cast billet. In this paper, the big data in traditional continuous casting production operations is studied through in-depth experimental comparative analysis of linear and nonlinear dimension reduction method. The method is suitable for continuous casting to obtain the data of the dimension reduction. The method of principal component analysis is improved by using standardized data increment method. A faster and more efficient method of dimension reduction is obtained when the unrelated data, training time and reconstruction error are removed. Actual data simulation results show that this method is more efficient and suitable for continuous casting than any other dimension reduction method.

Nonlinear Model for Condition Monitoring and Fault Detection Based on Nonlocal Kernel Orthogonal Preserving Embedding

The dimension reduction methods have been proved powerful and practical to extract latent features in the signal for process monitoring. A linear dimension reduction method called nonlocal orthogonal preserving embedding (NLOPE) and its nonlinear form named nonlocal kernel orthogonal preserving embedding (NLKOPE) are proposed and applied for condition monitoring and fault detection. Different from kernel orthogonal neighborhood preserving embedding (KONPE) and kernel principal component analysis (KPCA), the NLOPE and NLKOPE models aim at preserving global and local data structures simultaneously by constructing a dual‐objective optimization function. In order to adjust the trade‐off between global and local data structures, a weighted parameter is introduced to balance the objective function. Compared with KONPE and KPCA, NLKOPE combines both the advantages of KONPE and KPCA, and NLKOPE is also more powerful in extracting potential useful features in nonlinear data set than NLOPE. For the purpose of condition monitoring and fault detection, monitoring statistics are constructed in feature space. Finally, three case studies on the gearbox and bearing test rig are carried out to demonstrate the effectiveness of the proposed nonlinear fault detection method.

Low-rank preserving embedding

In this paper, we consider the problem of linear dimensionality reduction with the novel technique of low-rank representation, which is a promising tool of discovering subspace structures of given data. Existing approaches based on graph embedding usually capture structure of data via stacking the local structure of each datum, such as neighborhood graph, ℓ1-graph and ℓ2-graph. Yet they lack explicit discrimination between those local structures and suffer from corrupted samples. To this end, we propose a new linear dimensionality reduction method by virtue of the lowest rank representation (LRR) of data, which is dubbed low-rank preserving embedding (LRPE). Different from the traditional routes, LRPE achieves all data self-representations jointly and can thus extract the global structure of a data set as a whole. The global low-rank constraint explicitly enforces the LRR matrix to be block-diagonal form, so that the samples with a similar intrinsic structure, which are more likely to be from the same class, are described by a similar set of bases. Hence, LRPE is discriminative even if no class labels are provided. Benefiting from the robust LRR, LRPE is also robust to various noises and errors in data. Besides, we rewritten all related methods into a unified formulation, followed by a detailed solution and clear comparisons. Finally, we conduct extensive experiments on publicly available data sets for data visualization and classification. The inspiring experimental results show the effectiveness, the cheap computation and the robustness of the proposed method.

Facial Expression Recognition Method Based on Stacked Denoising Autoencoders and Feature Reduction

Based on the deep learning theory, a novel facial expression recognition method, which utilizes both Principal Component Analysis (PCA) and stacked denoising autoencoders (SDAE), is proposed in this paper. At first, PCA is used as a linear dimensionality reduction method on the expression features, and subsequently non-linear dimensionality is further learnt by a greed layer-wise method of stacked denoising autoencoders. Then, some meaningful and low-dimensioned expression features can be learnt and used to classify. The comparative experiment results show that the proposed method is more effective than some other expression recognition methods based on deep learning theories and it can also get higher expression recognition accuracy than traditional non-deep learning based expression recognition methods.

Unsupervised Large Graph Embedding

There are many successful spectral based unsupervised dimensionality reduction methods, including Laplacian Eigenmap (LE), Locality Preserving Projection (LPP), Spectral Regression (SR), etc. LPP and SR are two different linear spectral based methods, however, we discover that LPP and SR are equivalent, if the symmetric similarity matrix is doubly stochastic, Positive Semi-Definite (PSD) and with rank p, where p is the reduced dimension. The discovery promotes us to seek low-rank and doubly stochastic similarity matrix, we then propose an unsupervised linear dimensionality reduction method, called Unsupervised Large Graph Embedding (ULGE). ULGE starts with similar idea as LPP, it adopts an efficient approach to construct similarity matrix and then performs spectral analysis efficiently, the computational complexity can reduce to O(ndm), which is a significant improvement compared to conventional spectral based methods which need O(n^2d) at least, where n, d and m are the number of samples, dimensions and anchors, respectively. Extensive experiments on several public available data sets demonstrate the efficiency and effectiveness of the proposed method.

An online incremental orthogonal component analysis method for dimensionality reduction

In this paper, we introduce a fast linear dimensionality reduction method named incremental orthogonal component analysis (IOCA). IOCA is designed to automatically extract desired orthogonal components (OCs) in an online environment. The OCs and the low-dimensional representations of original data are obtained with only one pass through the entire dataset. Without solving matrix eigenproblem or matrix inversion problem, IOCA learns incrementally from continuous data stream with low computational cost. By proposing an adaptive threshold policy, IOCA is able to automatically determine the dimension of feature subspace. Meanwhile, the quality of the learned OCs is guaranteed. The analysis and experiments demonstrate that IOCA is simple, but efficient and effective.

Supervised bilateral two-dimensional locality preserving projection algorithm based on Gabor wavelet

Bilateral two-dimensional locality preserving projection (B2DLPP) is an effective method for unsupervised linear dimensionality reduction, which directly extracts face features from image matrices based on locality criterion. Motivated by B2DLPP, this paper proposes a supervised bilateral two-dimensional locality preserving projection (SB2DLPP). Different from B2DLPP, the proposed method takes into account the class information when constructing the similarity matrix. It increases inter-class distance in the projection space so that better right and left-projection matrices are obtained. Furthermore, a Gabor-based supervised bilateral two-dimensional locality preserving projection method is proposed for face recognition. Gabor wavelet representations are adopted for face images to make the proposed method robust to illumination variations and facial expression changes. Then, SB2DLPP is applied to reduce feature dimension. The performance of the proposed method is evaluated and compared with other traditional face recognition schemes on the FERET, Yale and JAFFE databases. The experiment results demonstrate the effectiveness and superiority of the proposed approach.

On Gradient Descriptor Distribution-based Dimensionality Reduction Analysis: A Case of Recognizing Plastic Surgery Sample Faces

Objectives: To analyze the influence of the sparseness distribution characteristics of gradient-based descriptor data on reduction of high-dimensional data, this paper presents experimental analysis on learned samples of gradient descriptor data. Method: In order to draw valid inferences, a single gradient descriptor, the Edge based Gabor Magnitude (EGM) facial descriptor, is used. The descriptor data is learned using various linear subspace dimensionality reduction methods. The subspace models are the Principle Component Analysis plus Linear Discriminant Analysis (PCA plus LDA), supervised Locality Preserving Projection (sLPP) and Locality Sensitive Discriminant Analysis (LSDA) under the LGE and OLGE general framework (which in the present is used to aid the characterization of the data geometric properties). Findings: Using the plastic surgery data set, the following observations were made. The global based linear subspace model (PCA plus LDA) which do not require complex neighborhood assignment performs favorably well in relation to the graph embedding models. This may be due to the fact that it only works on the basis of class information. The LSDA is observed to be more affected by the nature of the descriptor data influenced by the complexity of plastic surgery because in all its identification rates, a below 60% is achieved. On the other hand, the sLPP show to be a best fit model for the sparse nature of the descriptor data. This can be attributed to its data preserving property by which it is able to preserve the local structures of a sparse data (gradient-based) and so outperformed the PCA plus LDA and most importantly, the LSDA. Applications/Improvements: Understanding the best fit model for certain descriptor data is as important as optimizing recognition rates, an important observation for the face recognition research community.

Prediction of high-dimensional states subject to respiratory motion: a manifold learning approach

The development of high-dimensional imaging systems in image-guided radiotherapy provides important pathways to the ultimate goal of real-time full volumetric motion monitoring. Effective motion management during radiation treatment usually requires prediction to account for system latency and extra signal/image processing time. It is challenging to predict high-dimensional respiratory motion due to the complexity of the motion pattern combined with the curse of dimensionality. Linear dimension reduction methods such as PCA have been used to construct a linear subspace from the high-dimensional data, followed by efficient predictions on the lower-dimensional subspace. In this study, we extend such rationale to a more general manifold and propose a framework for high-dimensional motion prediction with manifold learning, which allows one to learn more descriptive features compared to linear methods with comparable dimensions. Specifically, a kernel PCA is used to construct a proper low-dimensional feature manifold, where accurate and efficient prediction can be performed. A fixed-point iterative pre-image estimation method is used to recover the predicted value in the original state space. We evaluated and compared the proposed method with a PCA-based approach on level-set surfaces reconstructed from point clouds captured by a 3D photogrammetry system. The prediction accuracy was evaluated in terms of root-mean-squared-error. Our proposed method achieved consistent higher prediction accuracy (sub-millimeter) for both 200 ms and 600 ms lookahead lengths compared to the PCA-based approach, and the performance gain was statistically significant.

Two-Dimensional Discriminant Locality Preserving Projection Based on ℓ1-norm Maximization

In this paper, a new linear dimensionality reduction method named Two-Dimensional Discriminant Locality Preserving Projection Based on ℓ1-norm Maximization (2DDLPP-L1) is proposed for preprocessing of image data. 2DDLPP-L1 makes full use of the robustness of ℓ1-norm to noises and outliers. Furthermore, 2DDLPP-L1 is a 2D-based method which extracts image features directly from image matrices, avoiding instability and high complexity of matrix computation. Two graphs, separation graph and cohesiveness graph, are constructed with feature vectors as vertices to represent the inter-class separation and intra-class cohesiveness. An iterative algorithm with proof of convergence is proposed to solve the optimal projection matrix. Experiments on several face image databases demonstrate that the performance and robustness of 2DDLPP-L1 are better than its related methods.