Dimensionality Reduction Framework Research Articles

Unsupervised and Semisupervised Projection With Graph Optimization.

Graph-based technique is widely used in projection, clustering, and classification tasks. In this article, we propose a novel and solid framework, named unsupervised projection with graph optimization (UPGO), for both dimensionality reduction and clustering. Different from the existing algorithms which treat graph construction and projection learning as two separate steps, UPGO unifies graph construction and projection learning into a general framework. It learns the graph similarity matrix adaptively based on the relationships among the low-dimensional representations. A constraint is introduced to the Laplacian matrix to learn a structured graph which contains the clustering structure, from which the clustering results can be obtained directly without requiring any postprocessing. The structured graph achieves the ideal neighbors assignment, based on which an optimal low-dimensional subspace can be learned. Moreover, we generalize UPGO to tackle the semisupervised case, namely semisupervised projection with graph optimization (SPGO), a framework for both dimensionality reduction and classification. An efficient algorithm is derived to optimize the proposed frameworks. We provide theoretical analysis about convergence analysis, computational complexity, and parameter determination. Experimental results on real-world data sets show the effectiveness of the proposed frameworks compared with the state-of-the-art algorithms. Results also confirm the generality of the proposed frameworks.

A multilevel features selection framework for skin lesion classification

Melanoma is considered to be one of the deadliest skin cancer types, whose occurring frequency elevated in the last few years; its earlier diagnosis, however, significantly increases the chances of patients’ survival. In the quest for the same, a few computer based methods, capable of diagnosing the skin lesion at initial stages, have been recently proposed. Despite some success, however, margin exists, due to which the machine learning community still considers this an outstanding research challenge. In this work, we come up with a novel framework for skin lesion classification, which integrates deep features information to generate most discriminant feature vector, with an advantage of preserving the original feature space. We utilize recent deep models for feature extraction, and by taking advantage of transfer learning. Initially, the dermoscopic images are segmented, and the lesion region is extracted, which is later subjected to retrain the selected deep models to generate fused feature vectors. In the second phase, a framework for most discriminant feature selection and dimensionality reduction is proposed, entropy-controlled neighborhood component analysis (ECNCA). This hierarchical framework optimizes fused features by selecting the principle components and extricating the redundant and irrelevant data. The effectiveness of our design is validated on four benchmark dermoscopic datasets; PH2, ISIC MSK, ISIC UDA, and ISBI-2017. To authenticate the proposed method, a fair comparison with the existing techniques is also provided. The simulation results clearly show that the proposed design is accurate enough to categorize the skin lesion with 98.8%, 99.2% and 97.1% and 95.9% accuracy with the selected classifiers on all four datasets, and by utilizing less than 3% features.

Graph-Assisted Inverse Regression for Count Data and Its Application to Sequencing Data

Multivariate count data, such as sequencing reads in genomics, are often connected to a clinical phenotype of interest. We develop a flexible framework for dimension reduction in regression, with predictors that are correlated counts, by modeling the conditional distribution of the predictors, given the response, using a pairwise Poisson graphical model. This new framework, called network-based inverse regression for counts, allows us to derive a sufficient reduction of the predictors, while adjusting for the dependence structure among them. We propose a regularized criterion for estimating both the reduction structure and the network structure. The estimation algorithm can be implemented efficiently on a parallel computer. We also introduce an adaptive version and a sparse variant of the proposed procedure. The methods are evaluated on simulated data and are applied to a gut microbiome sequencing dataset. Supplementary materials for this article are available online.

Online Data Dimensionality Reduction and Reconstruction Using Graph Filtering

Graphs have become a popular toolbox for capturing similarity relationships among data vectors in a variety of networks. Diffusion processes are used to model the data using the graph topology. A novel approach is put forth that exploits data similarity and diffusion models, in a graph, to improve upon the reconstruction mean-square error (MSE) performance of principal component analysis. The tasks of data dimensionality reduction and reconstruction are formulated as graph matrix filtering operations, that are utilized in a dimensionality reduction framework optimal in the MSE sense. The novel formulation is seeking MSE-optimal filter matrices that minimize the reconstruction MSE of all the graph data. Block coordinate descent techniques are employed in the graph spectral domain MSE cost to recursively determine the MSE-optimal dimensionality reducing and reconstruction filters. Online implementations are put forth to work with continuous streams of graph data, while almost sure convergence to a stationary point of the ensemble reconstruction MSE cost is established. Analysis of the reconstruction MSE reveals that the lowest MSE achieved by the novel approach is not larger than the standard PCA MSE, while there is a bound for the filter orders after which no more MSE reduction occurs. Extensive numerical tests using synthetic data, as well as real image datasets demonstrate the advantage of the novel graph-based framework over standard PCA and related methods.

Multiple Kernel Feature Line Embedding for Hyperspectral Image Classification

In this study, a novel multple kernel FLE (MKFLE) based on general nearest feature line embedding (FLE) transformation is proposed and applied to classify hyperspectral image (HSI) in which the advantage of multple kernel learning is considered. The FLE has successfully shown its discriminative capability in many applications. However, since the conventional linear-based principle component analysis (PCA) pre-processing method in FLE cannot effectively extract the nonlinear information, the multiple kernel PCA (MKPCA) based on the proposed multple kernel method was proposed to alleviate this problem. The proposed MKFLE dimension reduction framework was performed through two stages. In the first multple kernel PCA (MKPCA) stage, the multple kernel learning method based on between-class distance and support vector machine (SVM) was used to find the kernel weights. Based on these weights, a new weighted kernel function was constructed in a linear combination of some valid kernels. In the second FLE stage, the FLE method, which can preserve the nonlinear manifold structure, was applied for supervised dimension reduction using the kernel obtained in the first stage. The effectiveness of the proposed MKFLE algorithm was measured by comparing with various previous state-of-the-art works on three benchmark data sets. According to the experimental results: the performance of the proposed MKFLE is better than the other methods, and got the accuracy of 83.58%, 91.61%, and 97.68% in Indian Pines, Pavia University, and Pavia City datasets, respectively.

AutoGAN-based dimension reduction for privacy preservation

Protecting sensitive information against data exploiting attacks is an emerging research area in data mining. Over the past, several different methods have been introduced to protect individual privacy from such attacks while maximizing data-utility of the application. However, these existing techniques are not sufficient to effectively protect data owner privacy, especially in the scenarios that utilize visualizable data (e.g. images, videos) or the applications that require heavy computations for implementation. To address these problems, we propose a new dimension reduction-based method for privacy preservation. Our method generates dimension-reduced data for performing machine learning tasks and prevents a strong adversary from reconstructing the original data. We first introduce a theoretical approach to evaluate dimension reduction-based privacy preserving mechanisms, then propose a non-linear dimension reduction framework motivated by state-of-the-art neural network structures for privacy preservation. We conducted experiments over three different face image datasets (AT&T, YaleB, and CelebA), and the results show that when the number of dimensions is reduced to seven, we can achieve the accuracies of 79%, 80%, and 73% respectively and the reconstructed images are not recognizable to naked human eyes.

Deep learning for high-dimensional reliability analysis

High-dimensional reliability analysis remains a grand challenge since most of the existing methods suffer from the curse of dimensionality. This paper introduces a novel high-dimensional data abstraction (HDDA) framework for dimension reduction in reliability analysis. It first involves training of a failure-informed autoencoder network to reduce the dimensionality of the high-dimensional input space, aiming at creating a distinguishable failure surface in a low-dimensional latent space. Then a deep feedforward neural network is constructed to connect the high-dimensional input parameters with the low-dimensional latent variables. With the HDDA framework, the high-dimensional reliability can be estimated by capturing the limit state function in the latent space using Gaussian process regression. To manage the uncertainty due to lack of training data, a distance-based sampling strategy is developed for iteratively identifying critical training samples, which improves the accuracy of the high-dimensional reliability estimations. Three high-dimensional examples are used to demonstrate the effectiveness of the proposed approach.

A machine learning approach to circumventing the curse of dimensionality in discontinuous time series machine data

The growing interest in artificial intelligence has led to current data-driven predictive maintenance (PdM) relying on machine learning (ML) algorithms. Although ML algorithms are useful for data-intensive analysis, research shows that their performance and reliability are reduced when high-dimensional data is used for training and testing. Raw machine data can be high-dimensional due to multi-sensor measurements and discontinuous due to the wide ranges of parameter variations during continuous sensor measurements. While standard dimension reduction methods such as principal component analysis are often applied to circumvent high-dimensionality, they are often unreliable when the data is discontinuous. This paper presents a ML-based dimension reduction framework to circumvent the challenges of high-dimensional discontinuous machine data. This framework minimizes discontinuity by clustering observations based on the dataset’s modality. The modality is identified using a kernel density estimation parameterized using a heat diffusion solution to the approximate mean integrated squared error. Then, low-dimension representations of each cluster are learned using Laplacian eigenmaps embedding. Finally, the original time sequence of observations across the low-dimensional clusters is used to re-index the observations into a continuous low-dimension feature set. We demonstrate the framework’s utility on common ML-based PdM analysis using the Commercial Modular Aero-Propulsion System Simulation dataset.

A robust dimensionality reduction and matrix factorization framework for data clustering

Most existing Non-negative Matrix Factorization (NMF) related data clustering techniques directly decompose the original feature space while have not well considered the fact that the low dimensional feature space is always embedded in the high dimensional feature space and can better reveal the spatial distribution of data. In this letter, we propose a new matrix factorization model, which unites the objectives of clustering and dimensionality reduction simultaneously. In the proposed framework, the clustering based on matrix factorization is actually executed on the embedded subspace which may provide more accurate and reasonable solutions. Furthermore, we use the l2,1-norm instead of the conventional l2-norm to enhance the clustering results and make the clustering framework more robust to the noises and outliers. Meanwhile, in order to preserve as much as possible local similarity of the data, we have also employed an affinity matrix with special learning to introduce the manifold learning into the cluster indicator matrix. An optimization procedure based on Augmented Lagrangian Method (ALM) is devised to effectively solve the proposed problem and explicitly show the clustering results. Experimental results on the benchmark datasets with different proprieties exhibit the superior performance of the proposed method.

Learning to propagate labels on graphs: An iterative multitask regression framework for semi-supervised hyperspectral dimensionality reduction

Hyperspectral dimensionality reduction (HDR), an important preprocessing step prior to high-level data analysis, has been garnering growing attention in the remote sensing community. Although a variety of methods, both unsupervised and supervised models, have been proposed for this task, yet the discriminative ability in feature representation still remains limited due to the lack of a powerful tool that effectively exploits the labeled and unlabeled data in the HDR process. A semi-supervised HDR approach, called iterative multitask regression (IMR), is proposed in this paper to address this need. IMR aims at learning a low-dimensional subspace by jointly considering the labeled and unlabeled data, and also bridging the learned subspace with two regression tasks: labels and pseudo-labels initialized by a given classifier. More significantly, IMR dynamically propagates the labels on a learnable graph and progressively refines pseudo-labels, yielding a well-conditioned feedback system. Experiments conducted on three widely-used hyperspectral image datasets demonstrate that the dimension-reduced features learned by the proposed IMR framework with respect to classification or recognition accuracy are superior to those of related state-of-the-art HDR approaches.

A General Framework for Dimensionality Reduction of K-Means Clustering

Dimensionality reduction plays an important role in many machine learning and pattern recognition applications. Linear discriminant analysis (LDA) is the most popular supervised dimensionality reduction technique which searches for the projection matrix that makes the data points of different classes to be far from each other while requiring data points of the same class to be close to each other. In this paper, trace ratio LDA is combined with K-means clustering into a unified framework, in which K-means clustering is employed to generate class labels for unlabeled data and LDA is used to investigate low-dimensional representation of data. Therefore, by combining the subspace clustering with dimensionality reduction together, the optimal subspace can be obtained. Differing from other existing dimensionality reduction methods, our novel framework is suitable for different scenarios: supervised, semi-supervised, and unsupervised dimensionality reduction cases. Experimental results on benchmark datasets validate the effectiveness and superiority of our algorithm compared with other relevant techniques.

Generative framework for dimensionality reduction of large scale network of nonlinear dynamical systems driven by external input

Several studies have proposed constraints under which a low-dimensional representation can be derived from large-scale real-world networks exhibiting complex nonlinear dynamics. Typically, these representations are formulated under certain assumptions, such as when solutions converge to attractor states using linear stability analysis or using projections of large-scale dynamical data into a set of lower dimensional modes that are selected heuristically. Here, we propose a generative framework for selection of lower dimensional modes onto which the entire network dynamics can be projected based on the symmetry of the input distribution for a large-scale network driven by external inputs, thus relaxing the heuristic selection of modes made in the earlier reduction approaches. The proposed mode reduction technique is tractable analytically and applied to different kinds of real-world large-scale network scenarios with nodes comprising of (a) Van der Pol oscillators (b) Hindmarsh–Rose neurons. These two demonstrations elucidate how order parameter is conserved at original and reduced descriptions thus validating our proposition.

Submanifold-Preserving Discriminant Analysis With an Auto-Optimized Graph.

Due to the multimodality of non-Gaussian data, traditional globality-preserved dimensionality reduction (DR) methods, such as linear discriminant analysis (LDA) and principal component analysis (PCA) are difficult to deal with. In this paper, we present a novel local DR framework via auto-optimized graph embedding to extract the intrinsic submanifold structure of multimodal data. Specifically, the proposed model seeks to learn an embedding space which can preserve the local neighborhood structure by constructing a k -nearest neighbors ( k NNs) graph on data points. Different than previous works, our model employs the l0 -norm constraint and binary constraint on the similarity matrix to impose that there only be a k nonzero value in each row of the similarity matrix, which can ensure the k -connectivity in graph. More important, as the high-dimensional data probably contains some noises and redundant features, calculating the similarity matrix in the original space by using a kernel function is inaccurate. As a result, a mechanism of an auto-optimized graph is derived in the proposed model. Concretely, we learn the embedding space and similarity matrix simultaneously. In other words, the selection of neighbors is automatically executed in the optimal subspace rather than in the original space when the algorithm reaches convergence, which can alleviate the affect of noises and improve the robustness of the proposed model. In addition, four supervised and semisupervised local DR methods are derived by the proposed framework which can extract the discriminative features while preserving the submanifold structure of data. Last but not least, since two variables need to be optimized simultaneously in the proposed methods, and the constraints on the similarity matrix are difficult to satisfy, which is an NP-hard problem. Consequently, an efficient iterative optimization algorithm is introduced to solve the proposed problems. Extensive experiments conducted on synthetic data and several real-world datasets have demonstrated the advantages of the proposed methods in robustness and recognition accuracy.

Joint graph optimization and projection learning for dimensionality reduction

Nowadays, graph-based dimensionality reduction approaches have become more and more popular due to their successful utilization for classification and clustering tasks. In these approaches, how to establish an appropriate graph is critical. To address this issue, a novel graph-based dimensionality reduction framework termed joint graph optimization and projection learning (JGOPL) is proposed in this paper. Compared with existing dimensionality reduction approaches, there are three main advantages of JGOPL. First, through performing the graph optimization and low-dimensional feature learning simultaneously, our proposed approach can accomplish the tasks of graph construction and dimensionality reduction jointly. Second, the l21-norm based distance measurement is adopted in the loss function of our JGOPL so that its robustness to the negative influence caused by the outliers or variations of data can be improved. Third, in order to well exploit and preserve the local structure information of high-dimensional data, a locality constraint is introduced into the proposed JGOPL to discourage a sample from connecting with the distant samples during graph optimization. Extensive classification and clustering experiments are carried out on seven publicly available databases to demonstrate the effectiveness of our approach. At last, the locality constraint and graph optimization strategy proposed in this paper is not only limited to dimensionality reduction, but also can be incorporated into other relevant graph-based tasks (such as spectral clustering).

SLIC Superpixel-Based l2,1-Norm Robust Principal Component Analysis for Hyperspectral Image Classification.

Hyperspectral Images (HSIs) contain enriched information due to the presence of various bands, which have gained attention for the past few decades. However, explosive growth in HSIs’ scale and dimensions causes “Curse of dimensionality” and “Hughes phenomenon”. Dimensionality reduction has become an important means to overcome the “Curse of dimensionality”. In hyperspectral images, labeled samples are more difficult to collect because they require many labor and material resources. Semi-supervised dimensionality reduction is very important in mining high-dimensional data due to the lack of costly-labeled samples. The promotion of the supervised dimensionality reduction method to the semi-supervised method is mostly done by graph, which is a powerful tool for characterizing data relationships and manifold exploration. To take advantage of the spatial information of data, we put forward a novel graph construction method for semi-supervised learning, called SLIC Superpixel-based -norm Robust Principal Component Analysis (SURPCA2,1), which integrates superpixel segmentation method Simple Linear Iterative Clustering (SLIC) into Low-rank Decomposition. First, the SLIC algorithm is adopted to obtain the spatial homogeneous regions of HSI. Then, the -norm RPCA is exploited in each superpixel area, which captures the global information of homogeneous regions and preserves spectral subspace segmentation of HSIs very well. Therefore, we have explored the spatial and spectral information of hyperspectral image simultaneously by combining superpixel segmentation with RPCA. Finally, a semi-supervised dimensionality reduction framework based on SURPCA2,1 graph is used for feature extraction task. Extensive experiments on multiple HSIs showed that the proposed spectral-spatial SURPCA2,1 is always comparable to other compared graphs with few labeled samples.

Unsupervised robust discriminative manifold embedding with self-expressiveness

Dimensionality reduction has obtained increasing attention in the machine learning and computer vision communities due to the curse of dimensionality. Many manifold embedding methods have been proposed for dimensionality reduction. Many of them are supervised and based on graph regularization whose weight affinity is determined by original noiseless data. When data are noisy, their performance may degrade. To address this issue, we present a novel unsupervised robust discriminative manifold embedding approach called URDME, which aims to offer a joint framework of dimensionality reduction, discriminative subspace learning , robust affinity representation and discriminative manifold embedding. The learned robust affinity not only captures the global geometry and intrinsic structure of underlying high-dimensional data, but also satisfies the self-expressiveness property. In addition, the learned projection matrix owns discriminative ability in the low-dimensional subspace. Experimental results on several public benchmark datasets corroborate the effectiveness of our approach and show its competitive performance compared with the related methods.

Quantum Algorithm for Spectral Regression for Regularized Subspace Learning

In this paper, we propose an efficient quantum algorithm for spectral regression which is a dimensionality reduction framework based on the regression and spectral graph analysis. The quantum algorithm involves two core subroutines: the quantum principal eigenvectors analysis and the quantum ridge regression algorithm. The quantum principal eigenvectors analysis can be performed by an efficient sparse Hamiltonian simulation. For the ridge regression, we propose a quantum algorithm that is derived from the quantum singular value decomposition method. Our quantum ridge regression algorithm is more suitable for data matrices that are non-sparse and skewed. Our analysis demonstrates that the quantum subroutines can be implemented with an approximatively polynomial speedup on a quantum computer over their classical counterparts.

2D Dimensionality Reduction Methods without Loss

In this paper, several two-dimensional extensions of principal component analysis (PCA) and linear discriminant analysis (LDA) techniques has been applied in a lossless dimensionality reduction framework, for face recognition application. In this framework, the benefits of dimensionality reduction were used to improve the performance of its predictive model, which was a support vector machine (SVM) classifier. At the same time, the loss of the useful information was minimized using the projection penalty idea. The well-known face databases were used to train and evaluate the proposed methods. The experimental results indicated that the proposed methods had a higher average classification accuracy in general compared to the classification based on Euclidean distance, and also compared to the methods which first extracted features based on dimensionality reduction technics, and then used SVM classifier as the predictive model.

Impact of reduction in descriptor size on object detection and classification

Extraction of distinctive and robust features in image/video analysis and processing has attracted the attention of researchers in the recent years. Elimination of irrelevant and less important features reduces the computational complexity to a great extent at the cost of a very marginal reduction in accuracy. This paper presents a framework for dimensionality reduction of the binary features to obtain a low dimension feature vector for object detection. The process of identification and selection of the most relevant feature is performed in three steps: extraction of features using binary descriptors; Selection of best feature subset using Sequential Forward Selection (SFS) and Principal Component Analysis; classification using SVM classifier. The experimental results show that BRISK and LATCH descriptors perform better even when the dimensionality is reduced from 256, 128, and 64 to 32 bits with an acceptable classification accuracy and significant reduction in run time. However, there is slight decrease in the classification accuracy of LBP, FREAK, BRIEF, and ORB. A classification rate of 84.93% is obtained with LATCH descriptor for a descriptor size of 32 bits.

Interactive dimensionality reduction using similarity projections

Recent advances in machine learning allow us to analyze and describe the content of high-dimensional data like text, audio, images or other signals. In order to visualize that data in 2D or 3D, usually Dimensionality Reduction (DR) techniques are employed. Most of these techniques, e.g., PCA or t-SNE, produce static projections without taking into account corrections from humans or other data exploration scenarios. In this work, we propose the interactive Similarity Projection (iSP), a novel interactive DR framework based on similarity embeddings, where we form a differentiable objective based on the user interactions and perform learning using gradient descent, with an end-to-end trainable architecture. Two interaction scenarios are evaluated. First, a common methodology in multidimensional projection is to project a subset of data, arrange them in classes or clusters, and project the rest unseen dataset based on that manipulation, in a kind of semi-supervised interpolation. We report results that outperform competitive baselines in a wide range of metrics and datasets. Second, we explore the scenario of manipulating some classes, while enriching the optimization with high-dimensional neighbor information. Apart from improving classification precision and clustering on images and text documents, the new emerging structure of the projection unveils semantic manifolds. For example, on the Head Pose dataset, by just dragging the faces looking far left to the left and those looking far right to the right, all faces are re-arranged on a continuum even on the vertical axis (face up and down). This end-to-end framework can be used for fast, visual semi-supervised learning, manifold exploration, interactive domain adaptation of neural embeddings and transfer learning.