Nonlinear Dimensionality Reduction Method Research Articles

Comparative study of different dimensionality reduction methods in hyperspectral image classification

Considering the high-dimensional characteristics of hyperspectral image (HSI) data, researchers generally adopt the dimensionality reduction (DR) methods to reduce the complexity and computing time of subsequent classification or regression tasks while preserving the intrinsic structure information of the data. At present, the research on DR of HSIs data mostly focuses on the application performance of a single method in specific tasks and few studies have been conducted on the adaptability of different DR methods to HSIs data. From the perspective of spectral domain and spatial domain, this paper makes a comparative study on the performance of various linear and nonlinear DR methods in the task of HSI classification of aerial HSI of Matiwan Village in Xiongan New Area. Specifically, it includes principal component analysis, independent component analysis, isometric mapping, Laplacian eigenmaps, autoencoder, etc. The results show that the intrinsic structure in Xiongan HSI is mainly linear structure. Compared with the nonlinear DR methods, the linear DR methods can better preserve the intrinsic structure information of data at a lower time-consuming cost. For the subsequent classification tasks, the linear DR methods have better classification performance and are more suitable for HSI data.

Understand SLE heterogeneity in the era of omics, big data, and artificial intelligence

AbstractSystemic lupus erythematosus (SLE) is a systemic autoimmune disease characterized by extraordinary heterogeneity, due to the complex pathogenesis and diverse manifestations. Stratification of patients for therapy and prognosis represents a major challenge to manage SLE. Conventional biomarkers for disease diagnosis and activity assessment provide very limited insight into immunological pathogenesis and therapeutic response rates. The advancement of “omics” technologies including genomics, transcriptomics, proteomics, and metabolomics has constituted an unprecedented opportunity to characterize the immunopathological landscape in individual patients with SLE. Indeed, genomic studies reveal a subset of SLE patients carrying one or more functional single nucleotide polymorphisms (SNPs) underlying immune dysregulation while transcriptomic studies have revealed subgroups in SLE patients showing distinct signatures for Type I interferon (TI‐IFN) pathway activation or aberrant differentiation of B cells into plasma cells. This review will summarize results from the latest studies using omics technology to understand SLE heterogeneity. In addition, we propose that the application of artificial intelligence, such as by machine learning‐based nonlinear dimensionality reduction method uniform manifold approximation and projection (UMAP) can further strengthen the analysis of omics big data. The combination of new technology and novel analysis pipeline can lead to breakthroughs in stratifying SLE patients for a better monitoring of disease activity and more precise design of treatment regime, not only for conventional immunosuppression but also novel immunotherapies targeting B‐cell activating factor (BAFF), TI‐IFN, and interleukin 2 (IL‐2).

Dimensionality reduction and classification for hyperspectral image based on robust supervised ISOMAP

ABSTRACT This study contributes to classification and recognition of hyperspectral image in industrial sorting, mineral identification, geological exploration, environmental survey, and other industrial production fields. The finding eliminates the redundancy of hyperspectral data and improves the classification accuracy of hyperspectral image. Hyperspectral remote sensing data are inherently nonlinear, and traditional linear dimensionality reduction methods cannot effectively reveal the nonlinear structure contained in the data. Manifold learning is a nonlinear dimensionality reduction method which has been developed rapidly in recent years. The intrinsic structure of the data is preserved by Isometric Feature Mapping (ISOMAP), but it is sensitive to noise and is not conducive to final classification. Prior studies neglected a higher dimensional manifold real structure in hyperspectral data dimensionality reduction due to lack of consideration credibility information and class information of samples in calculating manifold distance. This study proposes a Robust Supervised ISOMAP method. Data sample credibility mode is constructed, and a data similarity measure named triple geodesic distance is defined by introducing samples’ credibility information, class information, and neighborhood information. Triple geodesic distance fitting is proven. Multidimensional scaling analysis (MDS) and generalized regression neural network are used to construct the embedded coordinates of training samples and test samples. The results show that the proposed method has higher anti-noise performance than the traditional one. Also, the runtime is feasible, and can effectively improve the classification precision of the hyperspectral images, in comparison to traditional ISOMAP methods. The conclusion is that Robust Supervised ISOMAP is an effective nonlinear dimensionality reduction method for hyperspectral remote sensing image.

An appearance-driven space to create new BRDFs

The search for more realism in renderings led to increased interest in tabular bidirectional reflectance distribution function (BRDF) samples captured from real-world materials. This representation provides a high degree of realism at the cost of a long acquisition time and high storage space. Because of those drawbacks, the literature presents few BRDF databases captured from real materials, and the available ones present a limited number of samples. Aiming to create new and realistic-looking materials, this work proposes an approach to generate new BRDFs from a tabular BRDF database. Our goal is to provide a pipeline where the user expresses appearance necessities by selecting materials from a BRDF database. We generate an appearance-driven space shaped by the user’s interests, in which it is feasible to navigate and retrieve new tabular BRDFs. This new space should enable smooth navigation and, to provide visual richness and realism, not rely only on the materials selected by the user. To this end, we first preprocess a tabular BRDF database by keeping BRDF samples with relevant reflectance features. A low-dimensional representation of this preprocessed database is obtained by applying a dimensionality reduction method, and a clustering algorithm is applied to obtain clusters of materials with similar features. Based on the materials selected by the user, we define one or more clusters of materials to build the appearance-driven space. Any point in this lower-dimensional space can be mapped to the original BRDFs space. Therefore, new BRDFs that present similar properties to the ones which the user selected are created. We compare the performance of linear (Multidimensional Scaling - MDS) and nonlinear (Isometric Feature Mapping - ISOMAP) dimensionality reduction methods in this task and analyze the obtained materials.

Nondestructive Detection of Keemun Black Tea Grade Based on Hyperspectral Imaging Technique

In order to evaluate the grade of tea products quickly and nondestructive, the near infrared (900~1700 nm) hyperspectral imaging technology was applied to classify six grades of black tea. Firstly, the hyperspectral data were visualized by linear and nonlinear dimension reduction methods. The visualization algorithms included the linear method of principal component analysis (PCA), multi-dimensional scaling (MDS), the nonlinear method of T-distributed stochastic embedding (T-SNE) and Sammon nonlinear mapping. Secondly, the classification model was established by using support vector machine (SVM) and extreme learning machine (ELM) to identify the different grades of Keemun black tea. Finally, SVM and ELM classification model were used to identify each pixel of hyperspectral image, and the prediction map was obtained. The results showed that T-SNE could divide the six grades of Keemun black tea into six different clusters, and the accuracy of SVM and ELM prediction set was 100% and 96.35%, respectively. The T-SNE visualization effect was the best, and the detection model of SVM could effectively identify the six grades of Keemun black tea. This paper provides an effective method for rapid and nondestructive testing of tea product grade, which had great significance for quality control, authenticity and adulteration detection of tea products.

Mixture factor analysis with distance metric constraint for dimensionality reduction

Dimensionality reduction (DR) is a key preprocessing stage in high-dimensional data classification. Traditional linear DR algorithms, e.g., Linear Discriminant Analysis, transform the original data into a low-dimensional subspace with a linear transformation matrix. However, these methods cannot handle complex nonlinearly separable data. Although some nonlinear DR methods, e.g., Locally Linear Embedding, are proposed to solve this problem, most of them are unsupervised, which only focus on the data structure hidden in the original high-dimensional space, rather than maximizing the inter-class separability of the transformed data, thus reducing the classification accuracy. To tackle this challenge, a novel supervised nonlinear DR algorithm, distance metric restricted mixture factor analysis (DMR-MFA), is proposed for high-dimensional data classification. In DMR-MFA, the original data is divided into several clusters, and the generation of original data in each cluster is described via a factor analysis model. Meanwhile, the distance metric constraint (DMC) is used for maximizing the separability of transformed low-dimensional data from different classes. Moreover, the optimal model parameters are learned via the joint optimization of log-likelihood function and DMC loss function, which makes the DMR-MFA possible to obtain the more separable low-dimensional embeddings while accurately describing the original data. Experimental results on synthetic data, benchmark datasets and high-resolution range profile data demonstrate that our method can handle nonlinearly separable data and improves the classification accuracy of data with high dimensionality.

Detection of microsleep states from the EEG: a comparison of feature reduction methods.

Microsleeps are brief lapses in consciousness with complete suspension of performance. They are the cause of fatal accidents in many transport sectors requiring sustained attention, especially driving. A microsleep-warning device, using wireless EEG electrodes, could be used to rouse a user from an imminent microsleep. High-dimensional datasets, especially in EEG-based classification, present challenges as there are often a large number of potentially useful features for detecting the phenomenon of interest. Thus, it is often important to reduce the dimension of the original data prior to training the classifier. In this study, linear dimensionality reduction methods-principal component analysis (PCA) and probabilistic PCA (PPCA)-were compared with eight non-linear dimensionality reduction methods (kernel PCA, classical multi-dimensional scaling, isometric mapping, nearest neighbour estimation, stochastic neighbourhood embedding, autoencoder, stochastic proximity embedding, and Laplacian eigenmaps) on previously collected behavioural and EEG data from eight healthy non-sleep-deprived volunteers performing a 1D-visuomotor tracking task for 1 h. The effectiveness of the feature reduction algorithms was evaluated by visual inspection of class separation on 3D scatterplots, by trustworthiness scores, and by microsleep detection performance on a stacked-generalisation-based linear discriminant analysis (LDA) system estimating the microsleep/responsive state at 1 Hz based on the reduced features. On trustworthiness, PPCA outperformed PCA, but PCA outperformed all of the non-linear techniques. The trustworthiness score for each feature reduction method also correlated strongly with microsleep-state detection performance, providing strong validation of the ability of trustworthiness to estimate the relative effectiveness of feature reduction approaches, in terms of predicting performance, and ability to do so independently of the gold standard. Graphical abstract Proposed microsleep detection system.

A novel method to recognize and classify based on an E-nose

The electronic nose (E-nose) has been widely reported as a new and convenient detection technology. While making continuous breakthroughs, the electronic nose is also facing a series of new problems and challenges. Especially in the actual application process, it faces high-dimensional and non-linear problems in the perception data of odor under non-ideal conditions. In this paper, a new method based on multiple optimal mapping is proposed, called superimposed mapping analysis (SMA). This method optimizes the loss function and mapping feature space of the model by making full use of the relationship between dependencies and mutual relationships within the class. The optimization method is used to construct the optimal loss function of the reconstruction matrix, which can achieve dimensionality reduction and feature extraction. The experimental results show that compared with the classical linear and non-linear dimensionality reduction methods, the performance index of SMA is more in line with practical application requirements.

Dimensionality reduction and unsupervised clustering for EELS-SI

A novel combination of machine learning algorithms is proposed for the differentiation of distinct spectra in a large electron energy loss spectroscopy spectrum image (EELS-SI) dataset. For clustering of the EEL spectra including similar fine structures in an efficient space, linear and nonlinear dimensionality reduction methods are used to project the EEL spectra onto a low-dimensional space. Then, a density-based clustering algorithm is applied to distinguish the meaningful data clusters. By applying this strategy to various experimental EELS-SI datasets, differentiation of several groups of EEL spectra representing specific fine structures was achieved. It is possible to investigate particular fine structures by averaging all of the spectra in each cluster. Also, the spatial distributions of each cluster in the scanning regions can be observed, which enables investigation of the locations of different fine structures in materials. This method does not require any prior knowledge, i.e., it is a data-driven analysis; therefore, it can be applied to any hyperspectral image.

BIOT: Explaining multidimensional nonlinear MDS embeddings using the Best Interpretable Orthogonal Transformation

Dimensionality reduction (DR) is a popular approach to data exploration in which instances in a given dataset are mapped to a lower-dimensional representation or “embedding.” For nonlinear dimensionality reduction (NLDR), the dimensions of the embedding may be difficult to understand. In such cases, it may be useful to learn how the different dimensions relate to a set of external features (i.e., relevant features that were not used for the DR). A variety of methods (e.g., PROFIT and BIR) use external features to explain embeddings generated by NLDR methods with rotation-invariant objective functions, such as multidimensional scaling (MDS). However, these methods are restricted to two-dimensional embeddings. In this paper, we propose BIOT, which makes it possible to explain an MDS embedding with any number of dimensions without requiring visualization.

Soft adaptive loss based Laplacian eigenmaps

The Laplacian eigenmaps (LE) is one of the most commonly used nonlinear dimensionality reduction methods and aims to find a low-dimensional representation to preserve the topological relationship between sample points in the original data. However, the l2-norm based loss function makes LE unable to preserve the relationship in many cases. Additionally, the topological relationship does not represent the real intrinsic structure of data. For example, the overemphasis of the topological relationship by LE easily breaks the manifold structure into multiple local areas in the embedding space, which makes the spectral clustering analysis of multi-manifold data more difficult to carry out. To solve this problem, we propose the soft adaptive loss based LE (SALE). With the soft adaptive loss, SALE can adaptively emphasize the topological relationship between sample points and the clustering structure of data. The model is tested and validated on UCI, face and gene expression data sets, and compared with some state-of-the-art models. The experimental results show that the method is robust to noise.

Supervised discriminant Isomap with maximum margin graph regularization for dimensionality reduction

As one of the most popular nonlinear dimensionality reduction methods, Isomap has been widely used in pattern recognition and machine learning. However, Isomap has the following problems: (1) Isomap is an unsupervised dimensionality reduction method, it cannot use class label information to obtain discriminative low dimensional embedding for classification; (2) The embedding performance of Isomap is sensitive to neighborhood size parameter; (3) Isomap cannot deal with outside new data by direct embedding. In this paper, a novel dimensionality reduction method called supervised discriminant Isomap is proposed to solve the first two problems mentioned above. Specifically, first, raw data points are partitioned into different manifolds by using their class label information. Then, supervised discriminant Isomap aims at seeking an optimal nonlinear subspace to preserve the geometrical structure of each manifold according to the Isomap criterion, and to enhance the discriminating capability by maximizing the distances between data points of different classes and the maximum margin graph regularization term. Finally, the corresponding optimization problems are solved by using eigen-decomposition algorithm. Further, we extend supervised discriminant Isomap to a linear dimensionality reduction method called supervised discriminant Isomap projection for handling the above three problems. Moreover, our approaches have three important characteristics: (1) Proposed methods adaptively estimate the local neighborhood surrounding each sample based on data density and similarity; (2) The objective functions of proposed methods can maximize margins between the each classes in the dimension-reduced feature space; (3) The objective functions of proposed methods have closed-form solutions. Furthermore, our methods can capture more discriminative information from raw data than other Isomap based methods. Extensive experiments on nine data sets demonstrate that the proposed methods are superior to the related state-of-the-art methods.

Autoencoder networks extract latent variables and encode these variables in their connectomes

Advances in electron microscopy and data processing techniques are leading to increasingly large and complete microscale connectomes. At the same time, advances in artificial neural networks have produced model systems that perform comparably rich computations with perfectly specified connectivity. This raises an exciting scientific opportunity for the study of both biological and artificial neural networks: to infer the underlying circuit function from the structure of its connectivity. A potential roadblock, however, is that – even with well constrained neural dynamics – there are in principle many different connectomes that could support a given computation. Here, we define a tractable setting in which the problem of inferring circuit function from circuit connectivity can be analyzed in detail: the function of input compression and reconstruction, in an autoencoder network with a single hidden layer. Here, in general there is substantial ambiguity in the weights that can produce the same circuit function, because largely arbitrary changes to input weights can be undone by applying the inverse modifications to the output weights. However, we use mathematical arguments and simulations to show that adding simple, biologically motivated regularization of connectivity resolves this ambiguity in an interesting way: weights are constrained such that the latent variable structure underlying the inputs can be extracted from the weights by using nonlinear dimensionality reduction methods.

Forward Stepwise Deep Autoencoder-Based Monotone Nonlinear Dimensionality Reduction Methods

Dimensionality reduction is an unsupervised learning task aimed at creating a low-dimensional summary and/or extracting the most salient features of a dataset. Principal component analysis is a linear dimensionality reduction method in the sense that each principal component is a linear combination of the input variables. To allow features that are nonlinear functions of the input variables, many nonlinear dimensionality reduction (NLDR) methods have been proposed. In this article, we propose novel NLDR methods based on bottleneck deep autoencoders. Our contributions are 2-fold: (1) We introduce a monotonicity constraint into bottleneck deep autoencoders for estimating a single nonlinear component and propose two methods for fitting the model. (2) We propose a new, forward stepwise deep learning architecture for estimating multiple nonlinear components. The former helps extract interpretable, monotone components when the assumption of monotonicity holds, and the latter helps evaluate reconstruction errors in the original data space for a range of components. We conduct numerical studies to compare different model fitting methods and use two real data examples from the studies of human immune responses to HIV to illustrate the proposed methods. Supplementary materials for this article are available online.

An Orthogonal Locality and Globality Dimensionality Reduction Method Based on Twin Eigen Decomposition

Dimensionality reduction is a hot research topic in pattern recognition. Traditional dimensionality reduction methods can be separated into linear dimensionality reduction methods and nonlinear dimensionality reduction methods. Linear dimensionality reduction methods usually utilize Euclidean distances to explore global geometric structures, and nonlinear dimensionality reduction methods can preserve local manifold structures in learned low-dimensional subspaces. However, redundant information and noises of the raw high-dimensional data restrict the classification performance of these methods. To solve the problem, we propose a novel orthogonal dimensionality reduction method based on twin eigen decomposition called orthogonal locality and globality preserving projections (OLGPP). Orthogonality, as a commonly used criterion in pattern recognition, is insensitive to the redundant information and the noises. OLGPP not only combines the advantages of global Euclidean structures and local manifold structures, but also is insensitive to the redundant information and noises of the raw high-dimensional data by embedding the orthogonality criterion. Additionally, the objective function based on twin eigen decomposition can be solved sequentially to obtain analytical solutions. On four well-known datasets, we design extensive experiments to evaluate the performance of OLGPP. From the experimental results, OLGPP has the optimal experimental results, and its average recognition rates are about 10% higher than the classic local dimensionality reduction method (i.e., locality preserving projection), which proves OLGPP is an effective dimensionality reduction method.

Optimization of a Comprehensive Sequence Forecasting Framework Based on DAE-LSTM Algorithm

Sales forecasting is a multivariate time series forecasting problem. The main challenges of the forecasting task are the high-dimensional influence variables with noise and the complex time series relationships. In this paper, we propose a DAE-LSTM algorithm combined with denoising autoencoder(DAE) and long short-term memory(LSTM) to deal with this problem. The DAE is a non-linear dimensionality reduction method that enhances data robustness. The LSTM is a deep learning algorithm suitable for dealing with multivariate time series problem. Combining the advantages of the above two algorithms, DAE-LSTM is a new multivariate time series prediction algorithm, which transforms unsupervised DAE into supervised data feature extraction method, and adds feature selection function to LSTM algorithm. In addition, in order to build a comprehensive forecasting framework, the hyperparameters of the model are determined by Bayesian optimization method(BOM) and the parameters of the model are solved by adaptive moment estimation(ADAM) optimization algorithm. In the training process, due to the adaptability of the BOM and ADAM, the forecasting model has few parameters that need to be set artificially, only including dimensional scale and time steps determined by data. Finally, the proposed forecasting framework is applied to forecast the sales of online retailers on JD.COM. In comparison with other machine learning algorithms, including multiple linear regression(MLR), support vector regression(SVR), artificial neural network(ANN) and LSTM, the proposed DAE-LSTM achieves the best prediction accuracy.

Assessing the Interplay of Shape and Physical Parameters by Unsupervised Nonlinear Dimensionality Reduction Methods

The article presents an exploratory study on the application to ship hydrodynamics of unsupervised nonlinear design-space dimensionality reduction methods, assessing the interaction of shape and physical parameters. Nonlinear extensions of the principal component analysis (PCA) are applied, namely local PCA (LPCA) and kernel PCA (KPCA). An artificial neural network approach, specifically a deep autoencoder (DAE) method, is also applied and compared with PCA-based approaches. The data set under investigation is formed by the results of 9000 potential flow simulations coming from an extensive exploration of a 27-dimensional design space, associated with a shape optimization problem of the DTMB 5415 model in calm water at 18 kn (Froude number, Fr = 25). Data include three heterogeneous distributed and suitably discretized parameters (shape modification vector, pressure distribution on the hull, and wave elevation pattern) and one lumped parameter (wave resistance coefficient), for a total of 9000 x 5101 elements. The reduced-dimensionality representation of shape and physical parameters is set to provide a normalized mean squared error smaller than 5%. The standard PCA meets the requirement using 19 principal components/parameters. LPCA and KPCA provide the most promising compression capability with 14 parameters required by the reduced-dimensionality parametrizations, indicating significant nonlinear interactions in the data structure of shape and physical parameters. The DAE achieves the same error with 17 components. Although the focus of the current work is on design-space dimensionality reduction, the formulation goes beyond shape optimization and can be applied to large sets of heterogeneous physical data from simulations, experiments, and real operation measurements.

D-EE: Distributed software for visualizing intrinsic structure of large-scale single-cell data.

BackgroundDimensionality reduction and visualization play vital roles in single-cell RNA sequencing (scRNA-seq) data analysis. While they have been extensively studied, state-of-the-art dimensionality reduction algorithms are often unable to preserve the global structures underlying data. Elastic embedding (EE), a nonlinear dimensionality reduction method, has shown promise in revealing low-dimensional intrinsic local and global data structure. However, the current implementation of the EE algorithm lacks scalability to large-scale scRNA-seq data.ResultsWe present a distributed optimization implementation of the EE algorithm, termed distributed elastic embedding (D-EE). D-EE reveals the low-dimensional intrinsic structures of data with accuracy equal to that of elastic embedding, and it is scalable to large-scale scRNA-seq data. It leverages distributed storage and distributed computation, achieving memory efficiency and high-performance computing simultaneously. In addition, an extended version of D-EE, termed distributed optimization implementation of time-series elastic embedding (D-TSEE), enables the user to visualize large-scale time-series scRNA-seq data by incorporating experimentally temporal information. Results with large-scale scRNA-seq data indicate that D-TSEE can uncover oscillatory gene expression patterns by using experimentally temporal information.ConclusionsD-EE is a distributed dimensionality reduction and visualization tool. Its distributed storage and distributed computation technique allow us to efficiently analyze large-scale single-cell data at the cost of constant time speedup. The source code for D-EE algorithm based on C and MPI tailored to a high-performance computing cluster is available at https://github.com/ShaokunAn/D-EE.

Multi-level adaptive neuro-fuzzy inference system-based reconstruction of 1D ISOMAP representations

This paper addresses the inverse problem of isometric feature mapping (ISOMAP) via multi-level adaptive neuro-fuzzy inference system (ML-ANFIS). ISOMAP is a conventional nonlinear dimensionality reduction (NLDR) method, which prospects for low dimensional interior structure embedded in high dimensional data space. The inverse problem of ISOMAP reconstructs the original high dimensional data from the related low dimensional ISOMAP representations and holds promising applications in data representations, generation, compression and visualization. Because the reconstruction of 1D ISOMAP representations is ill-posed and undetermined, ML-ANFIS is wielded to augment the recovery quality of general ISOMAP reconstruction algorithm. By linearly combing inputs with nonlinear weights as output, ML-ANFIS can efficiently achieve the latent nonlinear relationship between the low-performance result of general ISOMAP reconstruction algorithm and its original data. The membership functions and fuzzy rules of ML-ANFIS can be automatically learned by gradient descent method as deep learning. It is demonstrated by the experimental results that, in the situation of 1D representations, the proposed method is superior to the state-of-the-art methods, such as nearest neighbor (NN), discrete cosine transformation (DCT), sparse representations (SR) and classical ANFIS algorithms, in the reconstruction performance of video data.

Application of locally linear embedding algorithm on hotel data text classification

As a non-linear dimension reduction method, manifold learning algorithm projects high-dimensional input to a low-dimensional space by maintaining the local structure of the data, and discovers the inherent geometric structure hidden in the data. In this paper, we attempt to apply the manifold learning algorithm to the field of Chinese text classification, and use the locally linear embedding algorithm to reduce the dimension of the ctrip hotel review data set. Then, we utilize extreme gradient boosting (XGBoost) and logistic regression to classify the text. Experimental results show that it is effective and feasible to use manifold learning algorithm for text classification. Moreover, the classification effect of logistic regression is better than XGBoost in the text classification of hotel reviews.