Nonlinear Manifold Learning Research Articles

A Study of the Methane Oxidation Mechanism and Reaction Pathways Using Reactive Molecular Simulation and Nonlinear Manifold Learning

A Study of the Methane Oxidation Mechanism and Reaction Pathways Using Reactive Molecular Simulation and Nonlinear Manifold Learning

Learning the latent dynamics of fluid flows from high-fidelity numerical simulations using parsimonious diffusion maps

We use parsimonious diffusion maps (PDMs) to discover the latent dynamics of high-fidelity Navier–Stokes simulations with a focus on the two-dimensional (2D) fluidic pinball problem. By varying the Reynolds number Re, different flow regimes emerge, ranging from steady symmetric flows to quasi-periodic asymmetric and chaos. The proposed non-linear manifold learning scheme identifies in a crisp manner the expected intrinsic dimension of the underlying emerging dynamics over the parameter space. In particular, PDMs estimate that the emergent dynamics in the oscillatory regime can be captured by just two variables, while in the chaotic regime, the dominant modes are three as anticipated by the normal form theory. On the other hand, proper orthogonal decomposition/principal component analysis (POD/PCA), most commonly used for dimensionality reduction in fluid mechanics, does not provide such a crisp separation between the dominant modes. To validate the performance of PDMs, we also compute the reconstruction error, by constructing a decoder using geometric harmonics (GHs). We show that the proposed scheme outperforms the POD/PCA over the whole Re number range. Thus, we believe that the proposed scheme will allow for the development of more accurate reduced order models for high-fidelity fluid dynamics simulators, relaxing the curse of dimensionality in numerical analysis tasks such as bifurcation analysis, optimization, and control.

Unsupervised Anomaly Detection via Nonlinear Manifold Learning

Abstract Anomalies are samples that significantly deviate from the rest of the data and their detection plays a major role in building machine learning models that can be reliably used in applications such as data-driven design and novelty detection. The majority of existing anomaly detection methods either are exclusively developed for (semi) supervised settings, or provide poor performance in unsupervised applications where there are no training data with labeled anomalous samples. To bridge this research gap, we introduce a robust, efficient, and interpretable methodology based on nonlinear manifold learning to detect anomalies in unsupervised settings. The essence of our approach is to learn a low-dimensional and interpretable latent representation (aka manifold) for all the data points such that normal samples are automatically clustered together and hence can be easily and robustly identified. We learn this low-dimensional manifold by designing a learning algorithm that leverages either a latent map Gaussian process (LMGP) or a deep autoencoder (AE). Our LMGP-based approach, in particular, provides a probabilistic perspective on the learning task and is ideal for high-dimensional applications with scarce data. We demonstrate the superior performance of our approach over existing technologies via multiple analytic examples and real-world datasets.

GP+: A Python library for kernel-based learning via Gaussian processes

In this paper we introduce GP+, an open-source library for kernel-based learning via Gaussian processes (GPs) which are powerful statistical models that are completely characterized by their parametric covariance and mean functions. GP+ is built on PyTorch and provides a user-friendly and object-oriented tool for probabilistic learning and inference. As we demonstrate with a host of examples, GP+ has a few unique advantages over other GP modeling libraries. We achieve these advantages primarily by integrating nonlinear manifold learning techniques with GPs’ covariance and mean functions. As part of introducing GP+, in this paper we also make methodological contributions that (1) enable probabilistic data fusion and inverse parameter estimation, and (2) equip GPs with parsimonious parametric mean functions which span mixed feature spaces that have both categorical and quantitative variables. We demonstrate the impact of these contributions in the context of Bayesian optimization, multi-fidelity modeling, sensitivity analysis, and calibration of computer models.

Analysis of Manifold and its Application

Manifold learning is a field of study in machine learning and statistics that is closely associated with dimensionality reduction algorithmic techniques is gaining popularity these days. There are two types of manifold learning approaches: linear and nonlinear. Principal component analysis (PCA) and multidimensional scaling (MDS) are two examples of linear techniques that have long been staples in the statistician's arsenal for evaluating multivariate data. Nonlinear manifold learning, which encompasses diffusion maps, Laplacian Eigenmaps, Hessian Eigenmaps, Isomap, and local linear embedding, has seen a surge in research effort recently. A few of these methods are nonlinear extensions of linear approaches. A nearest search, the definition of distances or affinities between points (a crucial component of these methods' effectiveness), and an Eigen problem for embedding high-dimensional points into a lower dimensional space make up the algorithmic process of the majority of these techniques. The strengths and weaknesses of the new method are briefly reviewed in this article. In the field of computer graphics, we utilize a particular manifold learning method was first presented in statistics and machine learning to create a global, Spectral-based shape descriptor.

Robust locally nonlinear embedding (RLNE) for dimensionality reduction of high-dimensional data with noise

Local Linear Embedding (LLE) is a nonlinear manifold learning method for dimensionality reduction in high-dimensional data. However, when the data is distorted by noise, efficiency of LLE significantly diminishes. This paper proposes a robust locally nonlinear embedding (RLNE) method to alleviate the impact of noise. This is achieved by constructing nonlinear functions between data neighbors in high-dimensional space, and then mapping the relationships to low manifolds. The constrained least squares method is used to obtain more uniform weights to ensure that the neighborhood is approximately located on the local nonlinear patches of the manifold. Theoretical analysis is conducted on the reasons underlying RLNE's robustness to noise. Experimental results on synthetic and real-world data highlight RLNE's ability to preserve the intrinsic structure of data, showcasing robustness across various types data with various levels of noise, as well as with a larger number of nearest neighbors.

Plasma Etching Endpoint Detection in the Presence of Chamber Variations Through Nonlinear Manifold Learning and Density-Based Clustering

Plasma Etching Endpoint Detection in the Presence of Chamber Variations Through Nonlinear Manifold Learning and Density-Based Clustering

Damage Identification Using Nonlinear Manifold Learning Method under Changing Environments

Damage Identification Using Nonlinear Manifold Learning Method under Changing Environments

Mapping multi-decadal wetland loss: Comparative analysis of linear and nonlinear spatiotemporal characterization

Wetlands provide critical habitat for birds and endangered species, and can be influenced by shifts in water availability and land use. The Landsat image archive can help both document historical wetland change and monitor current dynamics, but multi-decadal retrospective studies face two important challenges: 1) many important ecohydrological processes occur at scales finer than the field of view of a 30 m Landsat pixel, and 2) ecologically meaningful wetland response units frequently exhibit spatiotemporal complexity in both vegetation and surface water which may not be well-represented by a simple study area spatial mean. Here we explore a novel method for contending with these challenges with application to the rapidly changing Andrade Mesa wetlands along the US-Mexico border. Using a 26-year (1995–2021) Landsat image time series, we track changes in surface water and vegetation using green vegetation (V) and dark (D) fractions estimated from spectral mixture analysis (SMA). Spatiotemporal dynamics in both V and D fractions were characterized using two techniques: linear Principal Component Analysis (PCA) and nonlinear manifold learning (via Uniform Manifold Approximation and Projection; UMAP). Bounding temporal endmembers were identified from PCA and used to construct a temporal mixture model which quantified total area of surface water and vegetation loss, including an uncertainty estimate. UMAP clearly distinguished areas with differing onset of vegetation and water loss which were not identifiable using PCA. Change maps were then associated with different mechanisms and outcomes, including possible mechanical clearing of vegetation and unexpected greening of woody vegetation. The overall timing of surface water and vegetation decline follows the lining of the All-American Canal, suggesting a potential cross-border hydrologic association between loss of a wetland in Mexico and water efficiency measures implemented in the United States.

Data-driven prediction of αIIbβ3 integrin activation paths using manifold learning and deep generative modeling

The integrin heterodimer is a transmembrane protein critical for driving cellular process and is a therapeutic target in the treatment of multiple diseases linked to its malfunction. Activation of integrin involves conformational transitions between bent and extended states. Some of the conformations that are intermediate between bent and extended states of the heterodimer have been experimentally characterized, but the full activation pathways remain unresolved both experimentally due to their transient nature and computationally due to the challenges in simulating rare barrier crossing events in these large molecular systems. An understanding of the activation pathways can provide new fundamental understanding of the biophysical processes associated with the dynamic interconversions between bent and extended states and can unveil new putative therapeutic targets. In this work, we apply nonlinear manifold learning to coarse-grained molecular dynamics simulations of bent, extended, and two intermediate states of αIIbβ3 integrin to learn a low-dimensional embedding of the configurational phase space. We then train deep generative models to learn an inverse mapping between the low-dimensional embedding and high-dimensional molecular space and use these models to interpolate the molecular configurations constituting the activation pathways between the experimentally characterized states. This work furnishes plausible predictions of integrin activation pathways and reports a generic and transferable multiscale technique to predict transition pathways for biomolecular systems.

Functional diffusion maps

Nowadays many real-world datasets can be considered as functional, in the sense that the processes which generate them are continuous. A fundamental property of this type of data is that in theory they belong to an infinite-dimensional space. Although in practice we usually receive finite observations, they are still high-dimensional and hence dimensionality reduction methods are crucial. In this vein, the main state-of-the-art method for functional data analysis is Functional PCA. Nevertheless, this classic technique assumes that the data lie in a linear manifold, and hence it could have problems when this hypothesis is not fulfilled. In this research, attention has been placed on a non-linear manifold learning method: Diffusion Maps. The article explains how to extend this multivariate method to functional data and compares its behavior against Functional PCA over different simulated and real examples.

Probabilistic neural data fusion for learning from an arbitrary number of multi-fidelity data sets

In many applications in engineering and sciences analysts have simultaneous access to multiple data sources. In such cases, the overall cost of acquiring information can be reduced via data fusion or multi-fidelity (MF) modeling where one leverages inexpensive low-fidelity (LF) sources to reduce the reliance on expensive high-fidelity (HF) data. In this paper, we employ neural networks (NNs) for data fusion in scenarios where data is very scarce and obtained from an arbitrary number of sources with varying levels of fidelity and cost. We introduce a unique NN architecture that converts MF modeling into a nonlinear manifold learning problem. Our NN architecture inversely learns non-trivial (e.g., non-additive and non-hierarchical) biases of the LF sources in an interpretable and visualizable manifold where each data source is encoded via a low-dimensional distribution. This probabilistic manifold quantifies model form uncertainties such that LF sources with small bias are encoded close to the HF source. Additionally, we endow the output of our NN with a parametric distribution not only to quantify aleatoric uncertainties, but also to reformulate the network’s loss function based on strictly proper scoring rules which improve robustness and accuracy on unseen HF data. Through a set of analytic and engineering examples, we demonstrate that our approach provides a high predictive power while quantifying various sources of uncertainty. Our codes and examples can be accessed via GitLab.

Kernel-based Nonlinear Manifold Learning for EEG-based Functional Connectivity Analysis and Channel Selection with Application to Alzheimer’s Disease

Dynamical, causal, and cross-frequency coupling analysis using the electroencephalogram (EEG) has gained significant attention for diagnosing and characterizing neurological disorders. Selecting important EEG channels is crucial for reducing computational complexity in implementing these methods and improving classification accuracy. In neuroscience, measures of (dis) similarity between EEG channels are often used as functional connectivity (FC) features, and important channels are selected via feature selection. Developing a generic measure of (dis) similarity is important for FC analysis and channel selection. In this study, learning of (dis) similarity information within the EEG is achieved using kernel-based nonlinear manifold learning. The focus is on FC changes and, thereby, EEG channel selection. Isomap and Gaussian Process Latent Variable Model (Isomap-GPLVM) are employed for this purpose. The resulting kernel (dis) similarity matrix is used as a novel measure of linear and nonlinear FC between EEG channels. The analysis of EEG from healthy controls (HC) and patients with mild to moderate Alzheimer’s disease (AD) are presented as a case study. Classification results are compared with other commonly used FC measures. Our analysis shows significant differences in FC between bipolar channels of the occipital region and other regions (i.e. parietal, centro-parietal, and fronto-central) between AD and HC groups. Furthermore, our results indicate that FC changes between channels along the fronto-parietal region and the rest of the EEG are important in diagnosing AD. Our results and its relation to functional networks are consistent with those obtained from previous studies using fMRI, resting-state fMRI and EEG.

Histogram Refined Local Ternary Pattern-based Bilateral LPP for Vision Sensor-based Robot Navigation Guidance under Challenging Environments

The present paper proposes a new variant of bilateral locality preserving projection (BLPP), a nonlinear manifold learning technique, for vision sensor-based robot navigation guidance in challenging environments, suffering from both variations in ambient illumination levels and vision sensor noises and uncertainties in image acquisition. In its basic form, the adjacency graph in BLPP is formulated both in feature subspace and spatial subspace, simultaneously. The present work proposes to compute the feature weights using local ternary pattern (LTP), a lesser noise-sensitive feature descriptor, which is also robust against illumination variations. In addition, histogram refinement has been performed on the feature descriptor to encode the local neighborhood structure of the data, proposing a further augmented version of BLPP named histogram refined local ternary pattern-based BLPP (HRLTP-BLPP). Extensive real-life experiments with challenging photometric conditions and sensor noises have been performed to show the usefulness of HRLTP-BLPP over other competing, state-of-the-art approaches, utilized for robot navigation guidance.

Calibration transfer for near-infrared (NIR) spectroscopy based on local preserving projection

Calibration transfer is a mathematical or statistical technique to solve the universality of near-infrared (NIR) spectroscopy, which is particularly important in practical applications. Therefore, a calibration transfer method based on local preserving projection (CTLPP) to correct spectral differences was proposed in this paper. Local preserving projection (LPP) is a linear feature extraction algorithm, which is a linear approximation of Laplacian Eigenmap manifold learning. It can not only overcome the shortcomings of linear feature extraction algorithms which are difficult to maintain the nonlinear manifold of the original data, but also solve the problem that nonlinear manifold learning methods are difficult to directly map new samples, which is conducive to building the conversion relationship between the source spectra and the target spectra. The transfer performance of the proposed method was evaluated in three datasets, and compared with five approaches, namely, piecewise direct standardization (PDS), calibration transfer based on independent component analysis (CTICA), spectral space transformation (SST), principal components canonical correlation analysis (PC-CCA) and calibration transfer based on neighborhood preserving embedding (CTNPE). Experimental results indicated that this proposed method can successfully correct the detection spectra among different apparatus. In addition, it can obtain relatively low prediction root mean square error (RMSEP). Therefore, the comprehensive research conducted in this work demonstrated that the proposed method is a promising calibration transfer method in NIR applications, which can provide a reference when sharing the NIR spectral models with more and more instruments.

Altered Brain Dynamics Across Bipolar Disorder and Schizophrenia During Rest and Task Switching Revealed by Overlapping Brain States

BackgroundIndividuals with bipolar disorder (BD) and schizophrenia (SCZ) show aberrant brain dynamics (i.e., altered recruitment or traversal through different brain states over time). Existing investigations of brain dynamics typically assume that one dominant brain state characterizes each time point. However, as multiple brain states likely are engaged at any given moment, this approach can obscure alterations in less prominent but critical brain states. Here, we examined brain dynamics in BD and SCZ by implementing a novel framework that simultaneously assessed the engagement of multiple brain states. MethodsFour recurring brain states were identified by applying nonlinear manifold learning and k-means clustering to the Human Connectome Project task-based functional magnetic resonance imaging data. We then assessed moment-to-moment state engagement in 2 independent samples of healthy control participants and patients with BD or SCZ using resting-state (N = 336) or task-based (N = 217) functional magnetic resonance imaging data. Relative state engagement and state engagement variability were extracted and compared across groups using multivariate analysis of covariance, controlling for site, medication, age, and sex. ResultsOur framework identified dynamic alterations in BD and SCZ, while a state discretization approach revealed no significant group differences. Participants with BD or SCZ showed reduced state engagement variability, but not relative state engagement, across multiple brain states during resting-state and task-based functional magnetic resonance imaging. We found decreased state engagement variability in older participants and preliminary evidence suggesting an association with avolition. ConclusionsAssessing multiple brain states simultaneously can reflect the complexity of aberrant brain dynamics in BD and SCZ, providing a more comprehensive understanding of the neural mechanisms underpinning these conditions.

Multi-view manifold learning of human brain-state trajectories.

The complexity of the human brain gives the illusion that brain activity is intrinsically high-dimensional. Nonlinear dimensionality-reduction methods such as uniform manifold approximation and t-distributed stochastic neighbor embedding have been used for high-throughput biomedical data. However, they have not been used extensively for brain activity data such as those from functional magnetic resonance imaging (fMRI), primarily due to their inability to maintain dynamic structure. Here we introduce a nonlinear manifold learning method for time-series data-including those from fMRI-called temporal potential of heat-diffusion for affinity-based transition embedding (T-PHATE). In addition to recovering a low-dimensional intrinsic manifold geometry from time-series data, T-PHATE exploits the data's autocorrelative structure to faithfully denoise and unveil dynamic trajectories. We empirically validate T-PHATE on three fMRI datasets, showing that it greatly improves data visualization, classification, and segmentation of the data relative to several other state-of-the-art dimensionality-reduction benchmarks. These improvements suggest many potential applications of T-PHATE to other high-dimensional datasets of temporally diffuse processes.

Comparison of manifold representations of sonar data

There are many challenges in active sonar target recognition due to the dynamic nature of the environment, unknown target geometries, and various clutter present within the ocean. These factors combine and entangle the target features within the received response. This research assumes the informative and discriminatory acoustic target features lie on a low dimensional manifold and can be extracted using statistical and machine learning techniques. Linear techniques, such as principal component analysis and linear discriminant analysis, are useful for dimension reduction but will typically not accurately capture the underlying non-linearity within a dataset. Non-linear manifold learning techniques, such as T-distributed stochastic neighbor embedding and uniform manifold approximation and projection, are relatively recent techniques that create low-dimensional embeddings in an unsupervised fashion and can capture the non-linearity. Previously, persistent braid features have been reported in real sonar data1. Our objective here is to do comparison between these different manifold representations for both simulated data with a known ground truth and experimental active sonar data. [1] A. Sen Gupta, B. Kubicek, A. Christensen, and I. Kirsteins, “Geometric feature representation in active sonar signal processing,” OCEANS 2022—Chennai, 2022, pp. 1–5. [Work supported by NDSEG 2021 and ONR Grant No. N00014-21-1-2420].

Data-driven control of agent-based models: An Equation/Variable-free machine learning approach

We present an Equation/Variable free machine learning (EVFML) framework for the control of the collective dynamics of complex/multiscale systems modeled via microscopic/agent-based simulators. The approach obviates the need for construction of surrogate, reduced-order models. The proposed implementation consists of three steps: (A) from high-dimensional agent-based simulations, machine learning, in particular, non-linear manifold learning (Diffusion Maps (DMs)) identifies a set of coarse-grained variables that parametrize the low-dimensional manifold on which the emergent/collective dynamics evolve. The out-of-sample extension and pre-image problems, i.e. the construction of non-linear mappings from the high-dimensional input space to the low-dimensional manifold and back, are solved by coupling DMs with the Nyström extension and Geometric Harmonics, respectively; (B) having identified the manifold and its coordinates, we exploit the Equation-free approach to perform numerical bifurcation analysis of the emergent dynamics; then (C) based on the previous steps, we design data-driven embedded wash-out controllers that drive the agent-based simulators to their intrinsic, imprecisely known, emergent open-loop unstable steady-states, thus demonstrating that the scheme is robust against numerical approximation errors and modeling uncertainty. The efficiency of the framework is illustrated by controlling emergent unstable (i) traveling waves of a deterministic agent-based model of traffic dynamics, and (ii) equilibria of a stochastic financial market model with mimesis.

From snapshots to manifolds – a tale of shear flows

We propose a novel nonlinear manifold learning from snapshot data and demonstrate its superiority over proper orthogonal decomposition (POD) for shedding-dominated shear flows. Key enablers are isometric feature mapping, Isomap, as encoder and,$K$-nearest neighbours ($K$NN) algorithm as decoder. The proposed technique is applied to numerical and experimental datasets including the fluidic pinball, a swirling jet and the wake behind a couple of tandem cylinders. Analysing the fluidic pinball, the manifold is able to describe the pitchfork bifurcation and the chaotic regime with only three feature coordinates. These coordinates are linked to the vortex-shedding phases and the force coefficients. The manifold coordinates of the swirling jet are comparable to the POD mode amplitudes, yet allow for a more distinct and less noise-sensitive manifold identification. A similar observation is made for the wake of two tandem cylinders. The tandem cylinders are aligned and located at a streamwise distance which corresponds to the transition between the single bluff body and the reattachment regimes of vortex shedding. Isomap unveils these two shedding regimes while the Lissajous plot of the first two POD mode amplitudes features a single circle. The reconstruction error of the manifold model is small compared with the fluctuation level, indicating that the low embedding dimensions contain the coherent structure dynamics. The proposed Isomap–$K$NN manifold learner is expected to be of great importance in estimation, dynamic modelling and control for a large range of configurations with dominant coherent structures.