High-dimensional Dynamical Systems Research Articles

Waddington landscape for prototype learning in generalized Hopfield networks

Networks in machine learning offer examples of complex high-dimensional dynamical systems inspired by and reminiscent of biological systems. Here, we study the learning dynamics of generalized Hopfield networks, which permit visualization of internal memories. These networks have been shown to proceed through a “feature-to-prototype” transition, as the strength of network nonlinearity is increased, wherein the learned, or terminal, states of internal memories transition from mixed to pure states. Focusing on the prototype learning dynamics of the internal memories, we observe stereotypical dynamics of memories wherein similar subgroups of memories sequentially split at well-defined saddles. The splitting order is interpretable and reproducible from one simulation to the other. The dynamics prior to splits are robust to variations in many features of the system. To develop a more rigorous understanding of these global dynamics, we study smaller subsystems that exhibit similar properties to the full system. Within these smaller systems, we combine analytical calculations with numerical simulations to study the dynamics of the feature-to-prototype transition, and the emergence of saddle points in the learning landscape. We exhibit regimes where saddles appear and disappear through saddle-node bifurcations, qualitatively changing the distribution of learned memories as the strength of the nonlinearity is varied—allowing us to systematically investigate the mechanisms that underlie the emergence of the learning dynamics. Several features of the learning dynamics are reminiscent of the Waddington's caricature of cellular differentiation, and we attempt to make this analogy more precise. Memories can thus differentiate in a predictive and controlled way, revealing bridges between experimental biology, dynamical systems theory, and machine learning. Published by the American Physical Society 2024

The fluctuational transition mechanism of non-hyperbolic chaotic invariant sets

In order to reveal the general escape mechanism of non-hyperbolic chaotic invariant sets under both weak noise limit and finite noise intensity, the simplest example of Hénon map, which represents the stretching and folding of the phase space, is taken to study the escape mechanism under two kinds of global bifurcation: the fractal boundary crisis and the attractor contact crisis. In this paper, we revealed the general exit mechanism by analyzing the escape paths derived from the shooting method and Monte Carlo simulation. Finally, to further demonstrate universality, an example of high-dimensional differential dynamical system, namely the transient chaos Duffing oscillator, is examined, which underscores the main idea of this paper analyzing specific deterministic structures not only enhances the comprehensibility of the escape process, but also allows for predictions of general escape behavior under weak noise intensity.

Fixed Attracting Closed Surfaces in Three and Higher Dimensional Dynamical Systems

Fixed Attracting Closed Surfaces in Three and Higher Dimensional Dynamical Systems

Probabilistic response determination of high-dimensional nonlinear dynamical systems enforced by parametric multiple Poisson white noises

Probabilistic response determination of high-dimensional nonlinear dynamical systems enforced by parametric multiple Poisson white noises

A novel method for response probability density of nonlinear stochastic dynamic systems

AbstractThis paper presents a novel method for analyzing high-dimensional nonlinear stochastic dynamic systems. In particular, we attempt to obtain the solution of the Fokker–Planck–Kolmogorov (FPK) equation governing the response probability density of the system without using the FPK equation directly. The method consists of several important components including the radial basis function neural networks (RBFNN), Feynman–Kac formula and the short-time Gaussian property of the response process. In the area of solving partial differential equations (PDEs) using neural networks, known as physics-informed neural network (PINN), the proposed method presents an effective alternative for obtaining solutions of PDEs without directly dealing with the equation, thus avoids evaluating the derivatives of the equation. This approach has a potential to make the neural network-based solution more efficient and accurate. Several highly challenging examples of nonlinear stochastic systems are presented in the paper to illustrate the effectiveness of the proposed method in comparison to the equation-based RBFNN approach.

Boosting reservoir computer performance with multiple delays.

Time delays play a significant role in dynamical systems, as they affect their transient behavior and the dimensionality of their attractors. The number, values, and spacing of these time delays influences the eigenvalues of a nonlinear delay-differential system at its fixed point. Here we explore a multidelay system as the core computational element of a reservoir computer making predictions on its input in the usual regime close to fixed point instability. Variations in the number and separation of time delays are first examined to determine the effect of such parameters of the delay distribution on the effectiveness of time-delay reservoirs for nonlinear time series prediction. We demonstrate computationally that an optoelectronic device with multiple different delays can improve the mapping of scalar input into higher-dimensional dynamics, and thus its memory and prediction capabilities for input time series generated by low- and high-dimensional dynamical systems. In particular, this enhances the suitability of such reservoir computers for predicting input data with temporal correlations. Additionally, we highlight the pronounced harmful resonance condition for reservoir computing when using an electro-optic oscillator model with multiple delays. We illustrate that the resonance point may shift depending on the task at hand, such as cross prediction or multistep ahead prediction, in both single delay and multiple delay cases.

Hermite kernel surrogates for the value function of high-dimensional nonlinear optimal control problems

Numerical methods for the optimal feedback control of high-dimensional dynamical systems typically suffer from the curse of dimensionality. In the current presentation, we devise a mesh-free data-based approximation method for the value function of optimal control problems, which partially mitigates the dimensionality problem. The method is based on a greedy Hermite kernel interpolation scheme and incorporates context knowledge by its structure. Especially, the value function surrogate is elegantly enforced to be 0 in the target state, non-negative and constructed as a correction of a linearized model. The algorithm allows formulation in a matrix-free way which ensures efficient offline and online evaluation of the surrogate, circumventing the large-matrix problem for multivariate Hermite interpolation. Additionally, an incremental Cholesky factorization is utilized in the offline generation of the surrogate. For finite time horizons, both convergence of the surrogate to the value function and for the surrogate vs. the optimal controlled dynamical system are proven. Experiments support the effectiveness of the scheme, using among others a new academic model with an explicitly given value function. It may also be useful for the community to validate other optimal control approaches.

A novel approach for dimensionality reduction of high-dimensional stochastic dynamical systems using symbolic regression

This paper introduces an innovative data-driven approach for dimensionality reduction in stochastic dynamical systems. The method combines stochastic averaging theory with symbolic regression. First, this approach generates simulation data from the high-dimensional stochastic dynamical system that needs to be reduced. Then, a loss function is designed based on stochastic averaging theory, and symbolic regression is used to obtain a parameterized one-dimensional equation from the simulation data. Finally, the effectiveness of the approach proposed in this paper is demonstrated through validation on Hamiltonian systems with 5, 10, and 20 degrees of freedom (DOF) subjected to noisy excitation. The primary advantage of this method is that it ultimately provides a closed form rather than a black-box process. Furthermore, it overcomes the numerical integration challenges that stochastic averaging methods often face when dealing with high-dimensional systems, therefore it performs effectively with high-dimensional stochastic dynamical systems.

Learning Reduced Fluid Dynamics

Predicting the state evolution of ultra high-dimensional, time-reversible fluid dynamic systems is a crucial but computationally expensive task. Existing physics-informed neural networks either incur high inference cost or cannot preserve the time-reversible nature of the underlying dynamics system. We propose a model-based approach to identify low-dimensional, time reversible, nonlinear fluid dynamic systems. Our method utilizes the symplectic structure of reduced Eulerian fluid and use stochastic Riemann optimization to obtain a low-dimensional bases that minimize the expected trajectory-wise dimension-reduction error over a given distribution of initial conditions. We show that such minimization is well-defined since the reduced trajectories are differentiable with respect to the subspace bases over the entire Grassmannian manifold, under proper choices of timestep sizes and numerical integrators. Finally, we propose a loss function measuring the trajectory-wise discrepancy between the original and reduced models. By tensor precomputation, we show that gradient information of such loss function can be evaluated efficiently over a long trajectory without time-integrating the high-dimensional dynamic system. Through evaluations on a row of simulation benchmarks, we show that our method reduces the discrepancy by 50-90 percent over conventional reduced models and we outperform PINNs by exactly preserving the time reversibility.

Predicting multiple observations in complex systems through low-dimensional embeddings

Forecasting all components in complex systems is an open and challenging task, possibly due to high dimensionality and undesirable predictors. We bridge this gap by proposing a data-driven and model-free framework, namely, feature-and-reconstructed manifold mapping (FRMM), which is a combination of feature embedding and delay embedding. For a high-dimensional dynamical system, FRMM finds its topologically equivalent manifolds with low dimensions from feature embedding and delay embedding and then sets the low-dimensional feature manifold as a generalized predictor to achieve predictions of all components. The substantial potential of FRMM is shown for both representative models and real-world data involving Indian monsoon, electroencephalogram (EEG) signals, foreign exchange market, and traffic speed in Los Angeles Country. FRMM overcomes the curse of dimensionality and finds a generalized predictor, and thus has potential for applications in many other real-world systems.

A hybrid proper orthogonal decomposition and next generation reservoir computing approach for high-dimensional chaotic prediction: Application to flow-induced vibration of tube bundles.

Chaotic time series prediction is a central science problem in diverse areas, ranging from engineering, economy to nature. Classical chaotic prediction techniques are limited to short-term prediction of low- or moderate-dimensional systems. Chaotic prediction of high-dimensional engineering problems is notoriously challenging. Here, we report a hybrid approach by combining proper orthogonal decomposition (POD) with the recently developed next generation reservoir computing (NGRC) for the chaotic forecasting of high-dimensional systems. The hybrid approach integrates the synergistic features of the POD for model reduction and the high efficiency of NGRC for temporal data analysis, resulting in a new paradigm on data-driven chaotic prediction. We perform the first chaotic prediction of the nonlinear flow-induced vibration (FIV) of loosely supported tube bundles in crossflow. Reducing the FIV of a continuous beam into a 3-degree-of-freedom system using POD modes and training the three time coefficients via a NGRC network with three layers, the hybrid approach can predict time series of a weakly chaotic system with root mean square prediction error less than 1% to 19.3 Lyapunov time, while a three Lyapunov time prediction is still achieved for a highly chaotic system. A comparative study demonstrates that the POD-NGRC outperforms the other existing methods in terms of either predictability or efficiency. The efforts open a new avenue for the chaotic prediction of high-dimensional nonlinear dynamic systems.

The causality measure of partial mutual information from mixed embedding (PMIME) revisited.

The measure of partial mutual information from mixed embedding (PMIME) is an information theory-based measure to accurately identify the direct and directional coupling, termed Granger causality or simply causality, between the observed variables or subsystems of a high-dimensional dynamical and complex system, without any a priori assumptions about the nature of the coupling relationship. In its core, it is a forward selection procedure that aims to iteratively identify the lag-dependence structure of a given observed variable (response) to all the other observed variables (candidate drivers). This model-free approach is capable of detecting nonlinear interactions, abundantly present in real-world complex systems, and it was shown to perform well on multivariate time series of moderately high dimension. However, the PMIME presents some inefficiencies in its performance mainly when applied on strongly stochastic (linear or nonlinear) systems as it may falsely detect non-existent relationships. Moreover, and by construction, the measure cannot extract purely synergetic relationships present in a system. In the current work, the issue of false detections is addressed by introducing an improved resampling significance test and a procedure of rechecking the identified drivers (backward revision). Regarding the inability to detect synergetic relationships, the PMIME is further enhanced by checking pairs as candidate drivers for the response variable after having considered all drivers individually. The effects of these modifications are investigated in a systematic simulation study on properly designed systems involving strong stochasticity, regressor terms with synergetic effects, and a system dimension ranging from 3 to 30. The overall results of the simulations indicate that these modifications indeed improve the performance of PMIME and alleviate to a significant degree the issues of the original algorithm. Guidelines for balancing between accuracy and computational efficiency are also given, particularly relevant for real-world applications. Finally, the measure performance is investigated in the study of futures of various government bonds and stock market indices in the period around COVID-19 pandemic.

Koopman-based surrogate models for multi-objective optimization of agent-based systems

Agent-based models (ABMs) provide an intuitive and powerful framework for studying social dynamics by modeling the interactions of individuals from the perspective of each individual. In addition to simulating and forecasting the dynamics of ABMs, the demand to solve optimization problems to support, for example, decision-making processes naturally arises. Most ABMs, however, are non-deterministic, high-dimensional dynamical systems, so objectives defined in terms of their behavior are computationally expensive. In particular, if the number of agents is large, evaluating the objective functions often becomes prohibitively time-consuming. We consider data-driven reduced models based on the Koopman generator to enable the efficient solution of multi-objective optimization problems involving ABMs. In a first step, we show how to obtain data-driven reduced models of non-deterministic dynamical systems (such as ABMs) that depend potentially nonlinearly on control inputs. We then use them in the second step as surrogate models to solve multi-objective optimal control problems. We first illustrate our approach using the example of a voter model, where we compute optimal controls to steer the agents to a predetermined majority, and then using the example of an epidemic ABM, where we compute optimal containment strategies in a prototypical situation. We demonstrate that the surrogate models effectively approximate the Pareto-optimal points of the ABM dynamics by comparing the surrogate-based results with test points, where the objectives are evaluated using the ABM. Our results show that when objectives are defined by the dynamic behavior of ABMs, data-driven surrogate models support or even enable the solution of multi-objective optimization problems.

Application of zeroed neural networks to stability analysis of continuous dynamic systems

Abstract Modern production processes frequently require steady-state analysis of continuous dynamic systems. Traditional numerical approaches, however, fall short in efficiency when tasked with addressing large-scale or dynamic problems. To tackle the inverse problem inherent in stability analysis, this study presents an innovative approach by integrating a combined excitation function into the foundational zeroing neural network (ZNN) model. This integration constrains the ZNN model, evolving it into an enhanced EZNN model specifically designed for solving the inverse of dynamic complex matrices. Additionally, this paper conducts a rigorous theoretical analysis of the robust performance of the EZNN model when excited by the combined function, both in the presence and absence of noise interference. The model solution process is promoted by using a class of high-dimensional continuous dynamic systems as an example, and numerical simulation experiments are used for validation. Considering the dynamic system satisfying { A ( t ) = ( 4 + sin ( 2 t ) 4 - cos ( 2 t ) 5 + sin ( 2 t ) ) C ( t ) = ( cos ( t ) sin ( t ) - cos ( t ) - sin ( t ) cos ( t ) sin ( t ) ) b ( t ) = 4 + cos ( 4 t ) , d ( t ) = ( cos ( 2 t ) , cos ( 2 t ) ) T \left\{ \matrix{ A(t) = \left( {4 + \sin (2t)\quad 4 - \cos (2t)\quad 5 + \sin (2t)} \right) \hfill \cr C(t) = \left( {\matrix{{\cos (t)} & {\sin (t)} & { - \cos (t)} \cr { - \sin (t)} & {\cos (t)} & {\sin (t)} \cr } } \right) \hfill \cr b(t) = 4 + \cos (4t),d(t) = {(\cos (2t),\cos (2t))^T} \hfill \cr} \right. , the error E 1(x(t),t) obtained by the EZNN model with combinatorial function excitation always remains negative or tends rapidly to 0. The x (t) obtained by the model converges rapidly to an exact solution of the system. Through the discussion of parametric conditions, it is also found that increasing the value of parameter γ increases the rate of convergence of the ZNN model.

On the Phenomenon of Low-Frequency, Large-Amplitude Oscillations in a High-Dimensional Linear Dynamical System

This article provides an explanation of the observed phenomenon on the basis of a complete analytical solution of the system. The solution is constructed in a standard way by reducing the dynamic problem to the problem of eigenvalues and eigenvectors for the system matrix. When solving, we use methods from the theory of orthogonal polynomials. In addition, we discuss two physical interpretations of the system. The connection between these interpretations and the system is established through the Schrödinger variables.

Ensemble forecasts in reproducing kernel Hilbert space family

A methodological framework for ensemble-based estimation and simulation of high dimensional dynamical systems such as the oceanic or atmospheric flows is proposed. To that end, the dynamical system is embedded in a family of reproducing kernel Hilbert spaces (RKHS) with kernel functions driven by the dynamics. In the RKHS family, the Koopman and Perron–Frobenius operators are unitary and uniformly continuous. This property warrants they can be expressed in exponential series of diagonalizable bounded evolution operators defined from their infinitesimal generators. Access to Lyapunov exponents and to exact ensemble based expressions of the tangent linear dynamics are directly available as well. The RKHS family enables us the devise of strikingly simple ensemble data assimilation methods for trajectory reconstructions in terms of constant-in-time linear combinations of trajectory samples. Such an embarrassingly simple strategy is made possible through a fully justified superposition principle ensuing from several fundamental theorems.

Reservoir computing with higher-order interactive coupled pendulums.

The reservoir computing approach utilizes a time series of measurements as input to a high-dimensional dynamical system known as a reservoir. However, the approach relies on sampling a random matrix to define its underlying reservoir layer, which leads to numerous hyperparameters that need to be optimized. Here, we propose a nonlocally coupled pendulum model with higher-order interactions as a novel reservoir, which requires no random underlying matrices and fewer hyperparameters. We use Bayesian optimization to explore the hyperparameter space within a minimal number of iterations and train the coupled pendulums model to reproduce the chaotic attractors, which simplifies complicated hyperparameter optimization. We illustrate the effectiveness of our technique with the Lorenz system and the Hindmarsh-Rose neuronal model, and we calculate the Pearson correlation coefficients between time series and the Hausdorff metrics in the phase space. We demonstrate the contribution of higher-order interactions by analyzing the interaction between different reservoir configurations and prediction performance, as well as computations of the largest Lyapunov exponents. The chimera state is found as the most effective dynamical regime for prediction. The findings, where we present a new reservoir structure, offer potential applications in the design of high-performance modeling of dynamics in physical systems.

Locating saddle points using gradient extremals on manifolds adaptively revealed as point clouds.

Steady states are invaluable in the study of dynamical systems. High-dimensional dynamical systems, due to separation of time scales, often evolve toward a lower dimensional manifold M. We introduce an approach to locate saddle points (and other fixed points) that utilizes gradient extremals on such a priori unknown (Riemannian) manifolds, defined by adaptively sampled point clouds, with local coordinates discovered on-the-fly through manifold learning. The technique, which efficiently biases the dynamical system along a curve (as opposed to exhaustively exploring the state space), requires knowledge of a single minimum and the ability to sample around an arbitrary point. We demonstrate the effectiveness of the technique on the Müller-Brown potential mapped onto an unknown surface (namely, a sphere). Previous work employed a similar algorithmic framework to find saddle points using Newton trajectories and gentlest ascent dynamics; we, therefore, also offer a brief comparison with these methods.

Investigation on the 1:2:3 internal resonance of high dimensional nonlinear dynamic system for FGGP reinforced rotating pre-twisted blade under the aerodynamic forces

The study of internal resonances is one of the main research areas in nonlinear dynamics. The energy transfer behaviors between multiple vibrational modes caused by internal resonances lead to complex nonlinear vibration behaviors. In this paper, the rotating blade is modeled as a functionally graded graphene platelet (FGGP) reinforced rotating pre-twisted plate. The nonlinear vibration of a high dimensional nonlinear dynamic system for FGGP reinforced rotating pre-twisted blade under the aerodynamic forces with 1:2:3 internal resonance is investigated. The axial excitations on the blade are primarily caused by axial aerodynamic force in the tip clearance and contact forces at the top of the blade. The subsonic airflow is treated as a transverse excitation and is calculated using the vortex lattice method. Rayleigh-Ritz method is used to obtain the mode shapes of the rotating composite pre-twisted blade. The three-degree-of-freedom ordinary differential equations of the rotating FGGP reinforced pre-twisted plate are derived by using Lagrange equation. Based on the multiscale method, the averaged equations of the rotating FGGP reinforced pre-twisted plate in the case of primary resonance and 1:2:3 internal resonance under axial and transverse excitations are acquired. We present a discussion on the amplitude-frequency response, force-amplitude response, bifurcations, and chaotic motions of a rotating FGGP reinforced pre-twisted cantilever plate under both axial and transverse excitations. A crescent-shaped independent resonance region is found in the amplitude-frequency response curves. The system exhibits complex nonlinear vibration behavior as axial or transverse excitation amplitude changes.

Nonlinear vibration of porous truncated conical shell under unified boundary condition and mechanical load

The aim of this article is to research nonlinear vibration responses of a metal porous truncated conical shell undergoing mechanical harmonic loads in thickness direction. The artificial spring technique is employed to unify boundary condition which can arrive at arbitrary classical boundary condition in real-word by changing the stiffness of springs. Utilizing Hamilton's principle, theoretical formulations of the conical shell are developed in view of FSDT and von-Kármán geometric relation. Afterwards, by employing generalized differential quadrature method (GDQM), the governing equation of nonlinear vibration in the form of partial differential equation is truncated and discretized into an ordinary differential equation which is a high-dimensional nonlinear dynamic system described in physical space rather than the mode space. On this basis, the analysis of effect of exciting frequency on the amplitude is carried out for three different distributing types of porous truncated conical shells by numerical integration. The frequency-amplitude responses of the system are depicted utilizing the sweeping frequency. The bifurcation diagrams, time series and phase diagrams are utilized to analyze the nonlinear dynamics of the porous truncated conical shell under unified boundary condition. It is found that the multiple solutions high-incidence area is near the 1st primary resonance, and vibration amplitude is dominated by the 1st order primary resonance and the 3rd one. The hardening-spring nature decreases for Type3, Type2 and Type1 successively.