High-dimensional Nonlinear Dynamical Systems Research Articles

Large-scale-aware data augmentation for reduced-order models of high-dimensional flows

Convolutional autoencoders have proven to be an adequate tool to perform reduced-order modeling for high-dimensional nonlinear dynamical systems. Their goal is to reduce dimensionality strongly while preserving the most characteristic features of the system. Here, we show that these models rely sensitively on the completeness of the provided data. This is particularly challenging for fully turbulent flows with their coherent structures ranging from large-scale superstructures to dissipative eddies over orders of magnitude in time and space. As a result, an unrealistically large number of data snapshots would be required to properly cover all the essential dynamics, whereas the features on small time and length scales require only a small number of snapshots of the respective flow, especially the long lasting large-scale structures that are difficult to characterize either numerically or experimentally. We demonstrate for three types of flows that a missing representation of large-scale turbulent structures leads to failures in the training process. We suggest a method to mitigate this shortcoming. This includes the transformation of data samples to new large-scale structures, which enhance the data. Furthermore, we skip augmentations that are more detrimental to the model performance. We evaluate our method for three datasets, two from numerical simulations of turbulent Rayleigh–Bénard convection flows and one from laboratory experiment for the flow past an array of cylinders. We show that the method can substantially improve model utility for high-dimensional data. In this way, we avoid an intensive grid search through possible augmentation combinations without further knowledge about the underlying system.

Probabilistic response determination of high-dimensional nonlinear dynamical systems enforced by parametric multiple Poisson white noises

Probabilistic response determination of high-dimensional nonlinear dynamical systems enforced by parametric multiple Poisson white noises

A novel method for response probability density of nonlinear stochastic dynamic systems

This paper presents a novel method for analyzing high-dimensional nonlinear stochastic dynamic systems. In particular, we attempt to obtain the solution of the Fokker–Planck–Kolmogorov (FPK) equation governing the response probability density of the system without using the FPK equation directly. The method consists of several important components including the radial basis function neural networks (RBFNN), Feynman–Kac formula and the short-time Gaussian property of the response process. In the area of solving partial differential equations (PDEs) using neural networks, known as physics-informed neural network (PINN), the proposed method presents an effective alternative for obtaining solutions of PDEs without directly dealing with the equation, thus avoids evaluating the derivatives of the equation. This approach has a potential to make the neural network-based solution more efficient and accurate. Several highly challenging examples of nonlinear stochastic systems are presented in the paper to illustrate the effectiveness of the proposed method in comparison to the equation-based RBFNN approach.

A hybrid proper orthogonal decomposition and next generation reservoir computing approach for high-dimensional chaotic prediction: Application to flow-induced vibration of tube bundles.

Chaotic time series prediction is a central science problem in diverse areas, ranging from engineering, economy to nature. Classical chaotic prediction techniques are limited to short-term prediction of low- or moderate-dimensional systems. Chaotic prediction of high-dimensional engineering problems is notoriously challenging. Here, we report a hybrid approach by combining proper orthogonal decomposition (POD) with the recently developed next generation reservoir computing (NGRC) for the chaotic forecasting of high-dimensional systems. The hybrid approach integrates the synergistic features of the POD for model reduction and the high efficiency of NGRC for temporal data analysis, resulting in a new paradigm on data-driven chaotic prediction. We perform the first chaotic prediction of the nonlinear flow-induced vibration (FIV) of loosely supported tube bundles in crossflow. Reducing the FIV of a continuous beam into a 3-degree-of-freedom system using POD modes and training the three time coefficients via a NGRC network with three layers, the hybrid approach can predict time series of a weakly chaotic system with root mean square prediction error less than 1% to 19.3 Lyapunov time, while a three Lyapunov time prediction is still achieved for a highly chaotic system. A comparative study demonstrates that the POD-NGRC outperforms the other existing methods in terms of either predictability or efficiency. The efforts open a new avenue for the chaotic prediction of high-dimensional nonlinear dynamic systems.

A decoupled approach for determination of the joint probability density function of a high-dimensional nonlinear stochastic dynamical system via the probability density evolution method

The joint probability density function (PDF) of multiple response processes of a system is a crucial topic in the fields of science and engineering. It is adopted to describe the dependent uncertainty propagation in complex stochastic dynamical systems, including the displacement and velocity of one degree of freedom (DOF), and the nonlinear probabilistic dependencies between multiple variables. However, the partial differential equations (PDEs) governing the time evolution of joint PDFs for multiple stochastic processes are often high-dimensional and coupled, making them analytically or numerically intractable for systems with high dimensionality. The probability density evolution method (PDEM) has provided a new perspective from the standpoint of random events description of the principle of preservation of probability. For an n-dimensional system, if merely the joint PDF of m processes is needed, where m is not larger than n, then it is sufficient to solve the generalized density evolution equation (GDEE), i.e., the Li–Chen equation, in m dimensions. This approach is particularly convenient for cases where n is large while m is small. However, for cases where m is not less than two, it is still challenging to avoid solving high-dimensional PDEs. This paper proposes a new method for the determination of joint PDF of multiple responses based on the principle of preservation of probability. To capture the joint PDF of m processes, the m-dimensional PDE can be transformed into m one-dimensional PDEs, significantly reducing the computational complexity of numerical implementation. The proposed method gives an enhanced version to the PDEM, enabling it to not only determine the individual probabilistic response, but also efficiently determine the dependent probabilistic distribution of multiple responses of high-dimensional stochastic dynamical systems without solving the coupling PDE. It reduces the computational cost from exponential growth of m powers to linear growth. Some numerical examples are presented to verify the effectiveness of the proposed method. Finally, problems for future investigations are discussed.

Dimension of Activity in Random Neural Networks.

Neural networks are high-dimensional nonlinear dynamical systems that process information through the coordinated activity of many connected units. Understanding how biological and machine-learning networks function and learn requires knowledge of the structure of this coordinated activity, information contained, for example, in cross covariances between units. Self-consistent dynamical mean field theory (DMFT) has elucidated several features of random neural networks-in particular, that they can generate chaotic activity-however, a calculation of cross covariances using this approach has not been provided. Here, we calculate cross covariances self-consistently via a two-site cavity DMFT. We use this theory to probe spatiotemporal features of activity coordination in a classic random-network model with independent and identically distributed (i.i.d.) couplings, showing an extensive but fractionally low effective dimension of activity and a long population-level timescale. Our formulas apply to a wide range of single-unit dynamics and generalize to non-i.i.d. couplings. As an example of the latter, we analyze the case of partially symmetric couplings.

Anti-circulant dynamic mode decomposition with sparsity-promoting for highway traffic dynamics analysis

Highway traffic state data collected from a network of sensors can be considered as a high-dimensional nonlinear dynamical system. In this paper, we develop a novel data-driven method–anti-circulant dynamic mode decomposition with sparsity-promoting (circDMDsp)–to study the dynamics of highway traffic speed data. Particularly, circDMDsp addresses several issues that hinder the application of existing DMD models: limited spatial dimension, presence of both recurrent and non-recurrent patterns, high level of noise, and known mode stability. The proposed circDMDsp framework allows us to numerically extract spatial–temporal coherent structures with physical meanings/interpretations: the dynamic modes reflect coherent spatial bases, and the corresponding temporal patterns capture the temporal oscillation/evolution of these dynamic modes. Our result based on Seattle highway loop detector data showcases that traffic speed data is governed by a set of periodic components, e.g., mean pattern, daily pattern, and weekly pattern, and each of them has a unique spatial structure. The spatiotemporal patterns can also be used to recover/denoise observed data and predict future values at any timestamp by extrapolating the temporal Vandermonde matrix. Our experiments also demonstrate that the proposed circDMDsp framework is more accurate and robust in data reconstruction and prediction than other DMD-based models. The code for the algorithm and experiment is available at https://github.com/mcgill-smart-transport/circDMDsp.

GLaSDI: Parametric physics-informed greedy latent space dynamics identification

A parametric adaptive physics-informed greedy Latent Space Dynamics Identification (gLaSDI) method is proposed for accurate, efficient, and robust data-driven reduced-order modeling of high-dimensional nonlinear dynamical systems. In the proposed gLaSDI framework, an autoencoder discovers intrinsic nonlinear latent representations of high-dimensional data, while dynamics identification (DI) models capture local latent-space dynamics. An interactive training algorithm is adopted for the autoencoder and local DI models, which enables identification of simple latent-space dynamics and enhances accuracy and efficiency of data-driven reduced-order modeling. To maximize and accelerate the exploration of the parameter space for the optimal model performance, an adaptive greedy sampling algorithm integrated with a physics-informed residual-based error indicator and random-subset evaluation is introduced to search for the optimal training samples on the fly. Further, to exploit local latent-space dynamics captured by the local DI models for an improved modeling accuracy with a minimum number of local DI models in the parameter space, a k-nearest neighbor convex interpolation scheme is employed. The effectiveness of the proposed framework is demonstrated by modeling various nonlinear dynamical problems, including Burgers equations, nonlinear heat conduction, and radial advection. The proposed adaptive greedy sampling outperforms the conventional predefined uniform sampling in terms of accuracy. Compared with the high-fidelity models, gLaSDI achieves 17 to 2,658× speed-up with 1 to 5% relative errors.

DETONATE: Nonlinear Dynamic Evolution Modeling of Time-dependent 3-dimensional Point Cloud Profiles

Modeling the evolution of a 3D profile over time as a function of heterogeneous input data and the previous time steps’ 3D shape is a challenging, yet fundamental problem in many applications. We introduce a novel methodology for the nonlinear modeling of dynamically evolving 3D shape profiles. Our model integrates heterogeneous, multimodal inputs that may affect the evolvement of the 3D shape profiles. We leverage the forward and backward temporal dynamics to preserve the underlying temporal physical structures. Our approach is based on the Koopman operator theory for high-dimensional nonlinear dynamical systems. We leverage the theoretical Koopman framework to develop a deep learning-based framework for nonlinear, dynamic 3D modeling with consistent temporal dynamics. We evaluate our method on multiple high-dimensional and short-term dependent problems, and it achieves accurate estimates, while also being robust to noise.

Learning a Low-Dimensional Representation of a Safe Region for Safe Reinforcement Learning on Dynamical Systems.

For the safe application of reinforcement learning algorithms to high-dimensional nonlinear dynamical systems, a simplified system model is used to formulate a safe reinforcement learning (SRL) framework. Based on the simplified system model, a low-dimensional representation of the safe region is identified and used to provide safety estimates for learning algorithms. However, finding a satisfying simplified system model for complex dynamical systems usually requires a considerable amount of effort. To overcome this limitation, we propose a general data-driven approach that is able to efficiently learn a low-dimensional representation of the safe region. By employing an online adaptation method, the low-dimensional representation is updated using the feedback data to obtain more accurate safety estimates. The performance of the proposed approach for identifying the low-dimensional representation of the safe region is illustrated using the example of a quadcopter. The results demonstrate a more reliable and representative low-dimensional representation of the safe region compared with previous works, which extends the applicability of the SRL framework.

Deep learning-enhanced ensemble-based data assimilation for high-dimensional nonlinear dynamical systems

Data assimilation (DA) is a key component of many forecasting models in science and engineering. DA allows one to estimate better initial conditions using an imperfect dynamical model of the system and noisy/sparse observations available from the system. Ensemble Kalman filter (EnKF) is a DA algorithm that is widely used in applications involving high-dimensional nonlinear dynamical systems. However, EnKF requires evolving large ensembles of forecasts using the dynamical model of the system. This often becomes computationally intractable, especially when the number of states of the system is very large, e.g., for weather prediction. With small ensembles, the estimated background error covariance matrix in the EnKF algorithm suffers from sampling error, leading to an erroneous estimate of the analysis state (initial condition for the next forecast cycle). In this work, we propose hybrid ensemble Kalman filter (H-EnKF), which is applied to a two-layer quasi-geostrophic turbulent flow as a test case. This framework utilizes a pre-trained deep learning-based data-driven surrogate that inexpensively generates and evolves a large data-driven ensemble of the states to accurately compute the background error covariance matrix with smaller sampling errors. The H-EnKF framework outperforms EnKF with only dynamical model or only the data-driven surrogate, and estimates a better initial condition without the need for any ad-hoc localization strategies. H-EnKF can be extended to any ensemble-based DA algorithm, e.g., particle filters, which are currently too expensive to use for high-dimensional systems.

First-passage probability estimation of high-dimensional nonlinear stochastic dynamic systems by a fractional moments-based mixture distribution approach

First-passage probability estimation of high-dimensional nonlinear stochastic dynamic systems is a significant task to be solved in many science and engineering fields, but remains still an open challenge. The present paper develops a novel approach, termed ‘fractional moments-based mixture distribution’, to address such challenge. This approach is implemented by capturing the extreme value distribution (EVD) of the system response with the concepts of fractional moment and mixture distribution. In our context, the fractional moment itself is by definition a high-dimensional integral with a complicated integrand. To efficiently compute the fractional moments, a parallel adaptive sampling scheme that allows for sample size extension is developed using the refined Latinized stratified sampling (RLSS). In this manner, both variance reduction and parallel computing are possible for evaluating the fractional moments. From the knowledge of low-order fractional moments, the EVD of interest is then expected to be reconstructed. Based on introducing an extended inverse Gaussian distribution and a log extended skew-normal distribution, one flexible mixture distribution model is proposed, where its fractional moments are derived in analytic form. By fitting a set of fractional moments, the EVD can be recovered via the proposed mixture model. Accordingly, the first-passage probabilities under different thresholds can be obtained from the recovered EVD straightforwardly. The performance of the proposed method is verified by three examples consisting of two test examples and one engineering problem.

Globally-evolving-based generalized density evolution equation for nonlinear systems involving randomness from both system parameters and excitations

It has long been one of the main challenges in science and engineering to capture the probabilistic response of high-dimensional nonlinear stochastic dynamic systems involving double randomness, i.e. randomness in both system parameters and excitations. For this purpose, a globally-evolving-based generalized density evolution equation (GE-GDEE) is established. Generally, for a multi-dimensional nonlinear system involving double randomness, if one single physical quantity as a response of the system is of interest, a GE-GDEE, as a two-dimensional partial differential equation (PDE) governing the probability density function (PDF), can be derived. The effective drift coefficients, which represent the physically driving force function in the GE-GDEE, can be determined based on the data from some representative deterministic dynamic analyses of the underlying physical system. A new estimator for effective drift coefficients is developed based on the vine copulas. Once the effective drift coefficients are determined, the GE-GDEE can be solved to capture the probability distributions of the quantities of interest. Several numerical examples, including linear and nonlinear multi-degree-of-freedom (MDOF) systems subjected to white noise or non-stationary earthquake ground motions, are presented to verify the effectiveness of the proposed method. Finally, problems for future investigations are discussed.

Recurrent neural networks enable design of multifunctional synthetic human gut microbiome dynamics.

Predicting the dynamics and functions of microbiomes constructed from the bottom-up is a key challenge in exploiting them to our benefit. Current models based on ecological theory fail to capture complex community behaviors due to higher order interactions, do not scale well with increasing complexity and in considering multiple functions. We develop and apply a long short-term memory (LSTM) framework to advance our understanding of community assembly and health-relevant metabolite production using a synthetic human gut community. A mainstay of recurrent neural networks, the LSTM learns a high dimensional data-driven non-linear dynamical system model. We show that the LSTM model can outperform the widely used generalized Lotka-Volterra model based on ecological theory. We build methods to decipher microbe-microbe and microbe-metabolite interactions from an otherwise black-box model. These methods highlight that Actinobacteria, Firmicutes and Proteobacteria are significant drivers of metabolite production whereas Bacteroides shape community dynamics. We use the LSTM model to navigate a large multidimensional functional landscape to design communities with unique health-relevant metabolite profiles and temporal behaviors. In sum, the accuracy of the LSTM model can be exploited for experimental planning and to guide the design of synthetic microbiomes with target dynamic functions.

Ensemble Riemannian data assimilation: towards large-scale dynamical systems

Abstract. This paper presents the results of the ensemble Riemannian data assimilation for relatively high-dimensional nonlinear dynamical systems, focusing on the chaotic Lorenz-96 model and a two-layer quasi-geostrophic (QG) model of atmospheric circulation. The analysis state in this approach is inferred from a joint distribution that optimally couples the background probability distribution and the likelihood function, enabling formal treatment of systematic biases without any Gaussian assumptions. Despite the risk of the curse of dimensionality in the computation of the coupling distribution, comparisons with the classic implementation of the particle filter and the stochastic ensemble Kalman filter demonstrate that, with the same ensemble size, the presented methodology could improve the predictability of dynamical systems. In particular, under systematic errors, the root mean squared error of the analysis state can be reduced by 20 % (30 %) in the Lorenz-96 (QG) model.

Deep Learning-Enhanced Ensemble-Based Data Assimilation for High-Dimensional Nonlinear Dynamical Systems

Deep Learning-Enhanced Ensemble-Based Data Assimilation for High-Dimensional Nonlinear Dynamical Systems

Neural Network Optimal Feedback Control With Guaranteed Local Stability

Recent research shows that supervised learning can be an effective tool for designing near-optimal feedback controllers for high-dimensional nonlinear dynamic systems. But the behavior of neural network controllers is still not well understood. In particular, some neural networks with high test accuracy can fail to even locally stabilize the dynamic system. To address this challenge we propose several novel neural network architectures, which we show guarantee local asymptotic stability while retaining the approximation capacity to learn the optimal feedback policy semi-globally. The proposed architectures are compared against standard neural network feedback controllers through numerical simulations of two high-dimensional nonlinear optimal control problems: stabilization of an unstable Burgers-type partial differential equation, and altitude and course tracking for an unmanned aerial vehicle. The simulations demonstrate that standard neural networks can fail to stabilize the dynamics even when trained well, while the proposed architectures are always at least locally stabilizing and can achieve near-optimal performance.

Glasdi: Parametric Physics-Informed Greedy Latent Space Dynamics Identification

A parametric adaptive physics-informed greedy Latent Space Dynamics Identification (gLaSDI) method is proposed for accurate, efficient, and robust data-driven reduced-order modeling of high-dimensional nonlinear dynamical systems. In the proposed gLaSDI framework, an autoencoder discovers intrinsic nonlinear latent representations of high-dimensional data, while dynamics identification (DI) models capture local latent-space dynamics. An interactive training algorithm is adopted for the autoencoder and local DI models, which enables identification of simple latent-space dynamics and enhances accuracy and efficiency of data-driven reduced-order modeling. To maximize and accelerate the exploration of the parameter space for the optimal model performance, an adaptive greedy sampling algorithm integrated with a physics-informed residual-based error indicator and random-subset evaluation is introduced to search for the optimal training samples on-the-fly. Further, to exploit local latent-space dynamics captured by the local DI models for an improved modeling accuracy with a minimum number of local DI models in the parameter space, an efficient k-nearest neighbor convex interpolation scheme is employed. The effectiveness of the proposed framework is demonstrated by modeling various nonlinear dynamical problems, including Burgers equations, nonlinear heat conduction, and radial advection. The proposed adaptive greedy sampling outperforms the conventional predefined uniform sampling in terms of accuracy. Compared with the high-fidelity models, gLaSDI achieves 66 to 4,417x speed-up with 1 to 5% relative errors.

Threefold way to the dimension reduction of dynamics on networks: An application to synchronization

Several complex systems can be modeled as large networks in which the state of the nodes continuously evolves through interactions among neighboring nodes, forming a high-dimensional nonlinear dynamical system. One of the main challenges of Network Science consists in predicting the impact of network topology and dynamics on the evolution of the states and, especially, on the emergence of collective phenomena, such as synchronization. We address this problem by proposing a Dynamics Approximate Reduction Technique (DART) that maps high-dimensional (complete) dynamics unto low-dimensional (reduced) dynamics while preserving the most salient features, both topological and dynamical, of the original system. DART generalizes recent approaches for dimension reduction by allowing the treatment of complex-valued dynamical variables, heterogeneities in the intrinsic properties of the nodes as well as modular networks with strongly interacting communities. Most importantly, we identify three major reduction procedures whose relative accuracy depends on whether the evolution of the states is mainly determined by the intrinsic dynamics, the degree sequence, or the adjacency matrix. We use phase synchronization of oscillator networks as a benchmark for our threefold method. We successfully predict the synchronization curves for three phase dynamics (Winfree, Kuramoto, theta) on the stochastic block model. Moreover, we obtain the bifurcations of the Kuramoto-Sakaguchi model on the mean stochastic block model with asymmetric blocks and we show numerically the existence of periphery chimera state on the two-star graph. This allows us to highlight the critical role played by the asymmetry of community sizes on the existence of chimera states. Finally, we systematically recover well-known analytical results on explosive synchronization by using DART for the Kuramoto-Sakaguchi model on the star graph.

Addressing the curse of dimensionality in stochastic dynamics: a Wiener path integral variational formulation with free boundaries.

A Wiener path integral variational formulation with free boundaries is developed for determining the stochastic response of high-dimensional nonlinear dynamical systems in a computationally efficient manner. Specifically, a Wiener path integral representation of a marginal or lower-dimensional joint response probability density function is derived. Due to this a priori marginalization, the associated computational cost of the technique becomes independent of the degrees of freedom (d.f.) or stochastic dimensions of the system, and thus, the 'curse of dimensionality' in stochastic dynamics is circumvented. Two indicative numerical examples are considered for highlighting the capabilities of the technique. The first relates to marine engineering and pertains to a structure exposed to nonlinear flow-induced forces and subjected to non-white stochastic excitation. The second relates to nano-engineering and pertains to a 100-d.f. stochastically excited nonlinear dynamical system modelling the behaviour of large arrays of coupled nano-mechanical oscillators. Comparisons with pertinent Monte Carlo simulation data demonstrate the computational efficiency and accuracy of the developed technique.