Low-dimensional Manifold Structure Research Articles

Euclidean mirrors and dynamics in network time series

– Analyzing changes in network evolution is central to statistical network inference. We consider a dynamic network model in which each node has an associated time-varying low-dimensional latent vector of feature data, and connection probabilities are functions of these vectors. Under mild assumptions, the evolution of latent vectors exhibits low-dimensional manifold structure under a suitable distance. This distance can be approximated by a measure of separation between the observed networks themselves, and there exist Euclidean representations for underlying network structure, as characterized by this distance. These Euclidean representations, called Euclidean mirrors, permit the visualization of network dynamics and lead to methods for change point and anomaly detection in networks. We illustrate our methodology with real and synthetic data, and identify change points corresponding to massive shifts in pandemic policies in a communication network of a large organization.

On Manifold Learning in Plato's Cave: Remarks on Manifold Learning and Physical Phenomena.

Many techniques in machine learning attempt explicitly or implicitly to infer a low-dimensional manifold structure of an underlying physical phenomenon from measurements without an explicit model of the phenomenon or the measurement apparatus. This paper presents a cautionary tale regarding the discrepancy between the geometry of measurements and the geometry of the underlying phenomenon in a benign setting. The deformation in the metric illustrated in this paper is mathematically straightforward and unavoidable in the general case, and it is only one of several similar effects. While this is not always problematic, we provide an example of an arguably standard and harmless data processing procedure where this effect leads to an incorrect answer to a seemingly simple question. Although we focus on manifold learning, these issues apply broadly to dimensionality reduction and unsupervised learning.

An autoencoder-based model for learning regularizations in unmixing problems

Signal unmixing is a class of complex, ill-posed inverse problems, which includes blind source separation or underdetermined signal separation to cite only two. Retrieving signals from their mixtures generally relies on adapted representations allowing to disentangle them. When dealing with real-world scientific data, the main challenge is to further build meaningful signal representations, which generally means capturing the underlying low-dimensional manifold structure of the signals to be recovered. Since the latter is generally unknown, this calls for a learning-based approach, which is a challenging task, especially when available training samples are scarce. The objective of the paper is to investigate a new learning model to build low-dimensional signal representations from few training samples. Based on an encoder-decoder architecture, the proposed approach aims to learn a non-linear interpolating scheme from examples. Extensive numerical experiments have been carried out to evaluate the performances of the proposed approach. We further illustrate how the learned representations can be conveniently deployed to tackle challenging semi-blind unmixing problems in the field of γ-ray spectroscopy.

Manifold Learning via Linear Tangent Space Alignment (LTSA) for Accelerated Dynamic MRI With Sparse Sampling.

The spatial resolution and temporal frame-rate of dynamic magnetic resonance imaging (MRI) can be improved by reconstructing images from sparsely sampled k -space data with mathematical modeling of the underlying spatiotemporal signals. These models include sparsity models, linear subspace models, and non-linear manifold models. This work presents a novel linear tangent space alignment (LTSA) model-based framework that exploits the intrinsic low-dimensional manifold structure of dynamic images for accelerated dynamic MRI. The performance of the proposed method was evaluated and compared to state-of-the-art methods using numerical simulation studies as well as 2D and 3D in vivo cardiac imaging experiments. The proposed method achieved the best performance in image reconstruction among all the compared methods. The proposed method could prove useful for accelerating many MRI applications, including dynamic MRI, multi-parametric MRI, and MR spectroscopic imaging.

Intrinsic Low-Dimensional Nonlinear Manifold Structure of Radio Frequency Signals

Nonlinear dynamic features are crucial in specific emitter identification (SEI), which works based on the nonlinearity of the emitter. Nevertheless, its underlying principles and stability have not been systematically studied, so these methods are close to a black box, making the recognition performance of the system unstable in applications. In this study, we provide an in-depth analysis of its mechanism and theoretical model, extracting, for the first time, the intrinsic low-dimensional nonlinear manifold structure (IMS) of radio frequency signals based on manifold learning and analyzing its robustness and distinctiveness for SEI. Hardware type and intentional modulation are found to determine the overall shape of IMS, and the device-specific information for SEI is subtle and distributed locally on IMS. Notably, sampling-related random disturbances cause the distortion, zooming, or rotation of IMS, which lowers the identification accuracy by a range of 1.74%–52%, depending on the feature, device type, signal type, and other factors.

Model-Centric Data Manifold: The Data Through the Eyes of the Model

We discover that deep ReLU neural network classifiers can see a low-dimensional Riemannian manifold structure on data. Such structure comes via the local data matrix, a variation of the Fisher information matrix, where the role of the model parameters is taken by the data variables. We obtain a foliation of the data domain and we show that the dataset on which the model is trained lies on a leaf, the data leaf, whose dimension is bounded by the number of classification labels. We validate our results with some experiments with the MNIST dataset: paths on the data leaf connect valid images, while other leaves cover noisy images.

Multi-manifold discriminant local spline embedding

Manifold learning reveals the intrinsic low-dimensional manifold structure of high-dimensional data and has achieved great success in a wide spectrum of applications. However, traditional manifold learning methods assume that all the data lie on a common manifold, hence fail to capture the complicated geometry structure of the real-world data lying on multiple manifolds. This paper proposes a novel Multi-manifold Discriminant Local Spline Embedding (MDLSE) algorithm for high-dimensional classification, which considers a more realistic scenario where data of the same class lies on the same manifold. On the basis of this assumption, MDLSE seeks to reconstruct multiple manifolds for different classes of data in the embedding and separate them as apart as possible. In order to preserve the geometry structure of all the manifolds, MDLSE employs thin plate splines to align the local patches within each manifold compatibly in the global embedding. Meanwhile, to separate the different manifolds, MDLSE utilizes discriminative information to ensure the neighboring data from different manifolds to be mapped far from each other. Extensive experiments on both synthetic and real-world datasets demonstrate the superiority of MDLSE over the other representative manifold learning algorithms. The advantage of MDLSE is often more obvious on smaller size of training data and in lower embedding dimensions.

A novel AdaBoost ensemble model based on the reconstruction of local tangent space alignment and its application to multiple faults recognition

In order to recognize the coupling faults of complex industrial processes effectively, this paper proposed an AdaBoost ensemble (AdBE) model based on the reconstruction of local tangent space alignment (RLTSA). First, to obtain the low-dimensional manifold structure embedded in original data space, RLTSA algorithm is designed by constructing the tangent space in the neighborhood of each data point to represent the local geometry and then aligning them to obtain the embedding coordinates. Secondly, to solve the loss of global feature information, an affine matrix is used to inversely map the low-dimensional coordinates to restore the global structure information. Thirdly, based on the above reconstruction of local tangent space alignment, an AdaBoost ensemble (AdBE) classifier is constructed for multiple faults recognition in which the AdaBoost algorithm is used to improve the performance of Decision Tree (DT), and One vs. Rest (OvR) ensemble strategy is introduced to establish the RLTSA-AdBE model. Case studies are conducted using a three-dimensional S_curve data set and the Tennessee Eastman process (TEP) to respectively verify the performance of the RLTSA algorithm and the proposed RLTSA-AdBE model. The simulation results indicate that the proposed method guarantees high diagnosis accuracy and macro_F1 Score of coupling faults recognition.

Minimum Eigenvector Collaborative Representation Discriminant Projection for Feature Extraction

High-dimensional signals, such as image signals and audio signals, usually have a sparse or low-dimensional manifold structure, which can be projected into a low-dimensional subspace to improve the efficiency and effectiveness of data processing. In this paper, we propose a linear dimensionality reduction method—minimum eigenvector collaborative representation discriminant projection—to address high-dimensional feature extraction problems. On the one hand, unlike the existing collaborative representation method, we use the eigenvector corresponding to the smallest non-zero eigenvalue of the sample covariance matrix to reduce the error of collaborative representation. On the other hand, we maintain the collaborative representation relationship of samples in the projection subspace to enhance the discriminability of the extracted features. Also, the between-class scatter of the reconstructed samples is used to improve the robustness of the projection space. The experimental results on the COIL-20 image object database, ORL, and FERET face databases, as well as Isolet database demonstrate the effectiveness of the proposed method, especially in low dimensions and small training sample size.

Harnessing the Manifold Structure of Cardiomechanical Signals for Physiological Monitoring During Hemorrhage.

Local oscillation of the chest wall in response to events during the cardiac cycle may be captured using a sensing modality called seismocardiography (SCG), which is commonly used to infer cardiac time intervals (CTIs) such as the pre-ejection period (PEP). An important factor impeding the ubiquitous application of SCG for cardiac monitoring is that morphological variability of the signals makes consistent inference of CTIs a difficult task in the time-domain. The goal of this work is therefore to enable SCG-based physiological monitoring during trauma-induced hemorrhage using signal dynamics rather than morphological features. We introduce and explore the observation that SCG signals follow a consistent low-dimensional manifold structure during periods of changing PEP induced in a porcine model of trauma injury. Furthermore, we show that the distance traveled along this manifold correlates strongly to changes in PEP ( ∆PEP). ∆PEP estimation during hemorrhage was achieved with a median R2 of 92.5% using a rapid manifold approximation method, comparable to an ISOMAP reference standard, which achieved an R2 of 95.3%. Rapidly approximating the manifold structure of SCG signals allows for physiological inference abstracted from the time-domain, laying the groundwork for robust, morphology-independent processing methods. Ultimately, this work represents an important advancement in SCG processing, enabling future clinical tools for trauma injury management.

Calibration Transfer for Near-Infrared (NIR) Spectroscopy Based on Neighborhood Preserving Embedding

Calibration transfer is a subtle issue in the practical application of near-infrared (NIR) spectroscopy technique. In this paper, a novel method to calibration transfer based on neighborhood preserving embedding (CTNPE) for correcting spectral differences was proposed. As a manifold learning method, neighborhood preserving embedding (NPE) can not only capture the nonlinear manifold structure, but also retain the linearity and show good generalization ability. Since this approach can reveal low dimensional manifold structure in high dimensional spectroscopic data, it is beneficial to construct the transform relationship between source and target spectra. The performance of CTNPE was assessed and compared to that of piecewise direct standardization (PDS) and other four dimensionality reduction-based methods, including transfer based on target factor analysis (TTFA), spectral space transformation (SST), calibration transfer based on canonical correlation analysis (CTCCA) and based on independent component analysis (CTICA), in two real cases. The results indicated that CTNPE was able to successfully transfer spectra between instruments and samples in different physical states. Furthermore, CTNPE provided lower prediction errors than PDS, TTFA, CTCCA, SST, CTICA and direct prediction without a transfer function. Therefore, the comprehensive investigation carried out in the presented work demonstrates that CTNPE is a promising calibration transfer method for NIR, especially for correcting the variations for samples in different physical states.

Representing Closed Transformation Paths in Encoded Network Latent Space

Deep generative networks have been widely used for learning mappings from a low-dimensional latent space to a high-dimensional data space. In many cases, data transformations are defined by linear paths in this latent space. However, the Euclidean structure of the latent space may be a poor match for the underlying latent structure in the data. In this work, we incorporate a generative manifold model into the latent space of an autoencoder in order to learn the low-dimensional manifold structure from the data and adapt the latent space to accommodate this structure. In particular, we focus on applications in which the data has closed transformation paths which extend from a starting point and return to nearly the same point. Through experiments on data with natural closed transformation paths, we show that this model introduces the ability to learn the latent dynamics of complex systems, generate transformation paths, and classify samples that belong on the same transformation path.

Maize haploid recognition study based on nuclear magnetic resonance spectrum and manifold learning

Haploid breeding is a significant technology of maize breeding. Nondestructively, rapidly and accurately haploid kernel identification method is the basis of developing haploid breeding technology. The commonly adopted maize haploid recognition methods at present are mainly near-infrared spectroscopy (NIRS), machine vision and nuclear magnetic resonance (NMR) oil measurement. NMR spectrum analysis method based on pattern recognition was used in this paper for haploid recognition, which on the one hand could improve recognition efficiency, and on the other hand could overcome the limitation of NMR oil measurement method namely it could not be applied to maize kernels produced by conventional inducer. NMR spectrum as a kind of high-dimensional data, manifold learning could effectively maintain the nonlinear structural properties of data while reducing dimensionality and extract easily identifiable features from these structures. Most manifold learning algorithms used at present map data of different categories onto the same low-dimensional embedded manifold. In order to better reserve essential structures of different categories of data, a new multi-manifold recognition framework was proposed in this paper for haploid recognition. The new framework uses the manifold learning algorithm to conduct feature extraction of NMR spectra of haploid and diploid respectively, and two low-dimensional manifold expressions are established; new samples are discriminated using the distance measurement method after being respectively mapped to two low-dimensional manifolds. For the difficulty existing in the calculation of point-to-manifold distance, the low-dimensional manifold structure is expressed in way of manifold coverage, and then point-to-manifold distance is expressed by calculating the distance from the sample point to the covered geometry. Maize kernels generated by high-oil induction system and conventional induction system were experimented in this paper. First of all, the feasibility of NMR spectrum analysis method based on pattern recognition for haploid identification was analyzed, the experiment was carried out using single-manifold and multi-manifold identification frameworks respectively, and stability of the multi-manifold identification framework was discussed finally. Experimental results indicate that the recognition rate of maize kernels induced by high-oil inducer can reach as high as 98.33% and the recognition rate of maize kernels induced by conventional inducer can reach as high as 90%, it proved that NMR spectrum combining manifold learning algorithm is feasible for haploid recognition; in the meantime, the multi-manifold recognition framework proposed in this paper has achieved better result than single-manifold recognition framework with the recognition rate elevated by 5% or so.

Nonlinear Feature Normalization for Hyperspectral Domain Adaptation and Mitigation of Nonlinear Effects

Domain adaptation in remote sensing aims at the automatic knowledge transfer between a set of multitemporal and multisource images. This process is often impaired by nonlinear effects in the data, e.g., varying illumination conditions, different viewing angles, and geometry-dependent reflection. In this paper, we introduce the Nonlinear Feature Normalization (NFN), a fast and robust way to align the spectral characteristics of multiple hyperspectral data sets. NFN employs labeled training spectra for the different classes in an image to describe the corresponding underlying low-dimensional manifold structure. A linear basis for data representation is defined by arbitrary class reference vectors, and the image is aligned to the new basis in the same space. This results in samples of the same class being pulled closer together and samples of different classes pushed apart. NFN transforms the data in its original domain, preserving physical interpretability. We use the continuous invertibility of NFN to derive the NFN Alignment (NFNalign) transformation, which can be used for domain adaptation, by transforming one data set to the domain of a chosen reference. The evaluation is performed on multiple hyperspectral data sets as well as our new benchmark for multitemporal hyperspectral data. In a first step, we show that the NFN transformation successfully mitigates nonlinear effects by comparing classification of the linear Spectral Angle Mapper on original and transformed data. Finally, we demonstrate successful domain adaptation with NFNalign by applying it to the task of hyperspectral data preprocessing. The evaluation shows that our approach for alignment of multitemporal data produces high-spectral similarity and successfully allows knowledge transfer, e.g., of classifier models and training data.

Manifold Based Low-Rank Regularization for Image Restoration and Semi-Supervised Learning

Low-rank structures play important roles in recent advances of many problems in image science and data science. As a natural extension of low-rank structures for data with nonlinear structures, the concept of the low-dimensional manifold structure has been considered in many data processing problems. Inspired by this concept, we consider a manifold based low-rank regularization as a linear approximation of manifold dimension. This regularization is less restricted than the global low-rank regularization, and thus enjoy more flexibility to handle data with nonlinear structures. As applications, we demonstrate the proposed regularization to classical inverse problems in image sciences and data sciences including image inpainting, image super-resolution, X-ray computer tomography image reconstruction and semi-supervised learning. We conduct intensive numerical experiments in several image restoration problems and a semi-supervised learning problem of classifying handwritten digits using the MINST data. Our numerical tests demonstrate the effectiveness of the proposed methods and illustrate that the new regularization methods produce outstanding results by comparing with many existing methods.

SP-SVM: Large Margin Classifier for Data on Multiple Manifolds

As one of the most important state-of-the-art classification techniques, Support Vector Machine (SVM) has been widely adopted in many real-world applications, such as object detection, face recognition, text categorization, etc., due to its competitive practical performance and elegant theoretical interpretation. However, it treats all samples independently, and ignores the fact that, in many real situations especially when data are in high dimensional space, samples typically lie on low dimensional manifolds of the feature space and thus a sample can be related to its neighbors by being represented as a linear combination of other samples on the same manifold. This linear representation, which is usually sparse, reflects the structure of underlying manifolds. It has been extensively explored in the recent literature and proven to be critical for the performance of classification. To benefit from both the underlying low dimensional manifold structure and the large margin classifier, this paper proposes a novel method called Sparsity Preserving Support Vector Machine(SP-SVM), which explicitly considers the sparse representation of samples while maximizing the margin between different classes. Consequently, SP-SVM inherits both the discriminative power of support vector machine and the merits of sparsity. A set of experiments on real-world benchmark data sets show that SP-SVM achieves significantly higher precision on recognition task than various competitive baselines including the traditional SVM, the sparse representation based method and the classical nearest neighbor classifier.

Convex nonnegative matrix factorization with manifold regularization

Nonnegative Matrix Factorization (NMF) has been extensively applied in many areas, including computer vision, pattern recognition, text mining, and signal processing. However, nonnegative entries are usually required for the data matrix in NMF, which limits its application. Besides, while the basis and encoding vectors obtained by NMF can represent the original data in low dimension, the representations do not always reflect the intrinsic geometric structure embedded in the data. Motivated by manifold learning and Convex NMF (CNMF), we propose a novel matrix factorization method called Graph Regularized and Convex Nonnegative Matrix Factorization (GCNMF) by introducing a graph regularized term into CNMF. The proposed matrix factorization technique not only inherits the intrinsic low-dimensional manifold structure, but also allows the processing of mixed-sign data matrix. Clustering experiments on nonnegative and mixed-sign real-world data sets are conducted to demonstrate the effectiveness of the proposed method.

Stepped Frequency Radar Target Recognition Using Locality Preserving Projections

In this paper, the idea of manifold learning is introduced into Stepped Frequency Radar (SFR) target recognition, a new method based on Locality Preserving Projections (LPP) algorithm and k-nearest neighbour classification for Stepped Frequency Radar target recognition is proposed. LPP is a subspace analytical method based on manifold learning, which is used to reduce the dimension of the High Resolution Range Profile (HRRP) and extract features from HRRP. The feature extraction method by LPP not only preserves the global topology structure, but also captures the local information of the different targets. Then three kinds of target are classified by k-nearest neighbour classification after the LPP. Experimental results on the three different targets suggest that the proposed method has the capability of finding the low-dimensional manifold structure embedded in the high-dimensional HRRP space and can provide a higher recognition rate in Stepped Frequency Radar target recognition.

Semi-supervised learning via sparse model

Graph-based Semi-Supervised Learning (SSL) methods are the widely used SSL methods due to their high accuracy. They can well meet the manifold assumption with high computational cost, but don't meet the cluster assumption. In this paper, we propose a Semi-supervised learning via SPArse (SSPA) model. Since SSPA uses sparse matrix multiplication to depict the adjacency relations among samples, SSPA can approximate low dimensional manifold structure of samples with lower computational complexity than these graph-based SSL methods. Each column of this sparse matrix corresponds to one sparse representation of a sample. The rational is that the inner product of sparse representations can also be sparse under certain constraint. Since the dictionary in the SSPA model can depict the distribution of the entire samples, the sparse representation of a sample encodes its spatial location information. Therefore, in the SSPA model the manifold structure of samples is computed via their locations in the intrinsic geometry of the distribution instead of their feature vectors. In order to meet the cluster assumption, we propose an structured dictionary learning algorithm to explicitly reveal the cluster structure of the dictionary. We develop the SSPA algorithms with the structured dictionary besides non-structured one, and experiments show that our methods are efficient and outperform state-of-the-art graph-based SSL methods.

Fault diagnosis method based on incremental enhanced supervised locally linear embedding and adaptive nearest neighbor classifier

A novel fault diagnosis method based on incremental enhanced supervised locally linear embedding (I-ESLLE) and adaptive nearest neighbor classifier (ANNC) is proposed to improve the accuracy of machinery fault diagnosis. Firstly, I-ESLLE is proposed for the non-linear dimensionality reduction of high-dimensional fault samples obtained from vibration signals. I-ESLLE can not only acquire the low-dimensional intrinsic manifold structure embedded in the high-dimensional input space, but also can deal with new fault samples in an iterative and batch model. Then, the low-dimensional fault samples are fed into the proposed ANNC for fault type identification. ANNC exploits “representation-based distance” to select the nearest training samples of new fault sample and identifies fault type in a weighting strategy. Moreover, the number of nearest training samples of each new fault sample is adaptively determined according to the density of the local distribution of the new fault sample. To verify the validity of the proposed fault diagnosis method, a fault diagnosis experiment of gearbox is performed, and the results indicate that the proposed fault diagnosis method outperforms the traditional methods and achieves higher diagnostic accuracy.