Expressive architectures enhance interpretability of dynamics-based neural population models.

Embedding nonlinear dynamical systems into artificial neural networks is a powerful new formalism for machine learning. By parameterizing ordinary differential equations (ODEs) as neural network layers, these Neural ODEs are memory-efficient to train, process time series naturally, and incorporate knowledge of physical systems into deep learning (DL) models. However, the practical applications of Neural ODEs are limited due to long inference times because the outputs of the embedded ODE layers are computed numerically with differential equation solvers that can be computationally demanding. Here, we show that mathematical model order reduction (MOR) methods can be used for compressing and accelerating Neural ODEs by accurately simulating the continuous nonlinear dynamics in low-dimensional subspaces. We implement our novel compression method by developing Neural ODEs that integrate the necessary subspace-projection and interpolation operations as layers of the neural network. We validate our approach by comparing it to neuron pruning and singular value decomposition (SVD)-based weight truncation methods from the literature in image and time-series classification tasks. The methods are evaluated by acceleration versus accuracy when adjusting the level of compression. On this spectrum, we achieve a favorable balance over existing methods by using MOR when compressing a convolutional Neural ODE. In compressing a recurrent Neural ODE, SVD-based weight truncation yields good performance. Based on our results, our integration of MOR with Neural ODEs can facilitate efficient, dynamical system-driven DL in resource-constrained applications.

We present and review an algorithmic and theoretical framework for improving neural network architecture design via momentum. As case studies, we consider how momentum can improve the architecture design for recurrent neural networks (RNNs), neural ordinary differential equations (ODEs), and transformers. We show that integrating momentum into neural network architectures has several remarkable theoretical and empirical benefits, including (1) integrating momentum into RNNs and neural ODEs can overcome the vanishing gradient issues in training RNNs and neural ODEs, resulting in effective learning long-term dependencies; (2) momentum in neural ODEs can reduce the stiffness of the ODE dynamics, which significantly enhances the computational efficiency in training and testing; (3) momentum can improve the efficiency and accuracy of transformers.

This work proposes an extension of neural ordinary differential equations (NODEs) by introducing an additional set of ODE input parameters to NODEs. This extension allows NODEs to learn multiple dynamics specified by the input parameter instances. Our extension is inspired by the concept of parameterized ODEs, which are widely investigated in computational science and engineering contexts, where characteristics of the governing equations vary over the input parameters. We apply the proposed parameterized NODEs (PNODEs) for learning latent dynamics of complex dynamical processes that arise in computational physics, which is an essential component for enabling rapid numerical simulations for time-critical physics applications. For this, we propose an encoder–decoder-type framework, which models latent dynamics as PNODEs. We demonstrate the effectiveness of PNODEs on benchmark problems from computational physics.

This work proposes an extension of neural ordinary differential equations (NODEs) by introducing an additional set of ODE input parameters to NODEs. This extension allows NODEs to learn multiple dynamics specified by the input parameter instances. Our extension is inspired by the concept of parameterized ordinary differential equations, which are widely investigated in computational science and engineering contexts, where characteristics of the governing equations vary over the input parameters. We apply the proposed parameterized NODEs (PNODEs) for learning latent dynamics of complex dynamical processes that arise in computational physics, which is an essential component for enabling rapid numerical simulations for time-critical physics applications. For this, we propose an encoder-decoder-type framework, which models latent dynamics as PNODEs. We demonstrate the effectiveness of PNODEs with important benchmark problems from computational physics.

This work advances the application of neural ordinary differential equations (ODEs) to circuit modeling. Prior works primarily utilized the recurrent neural network (RNN), which is a specific type of neural ODE. In this work, the capability of neural ODEs to represent different types of circuits is studied. Stability conditions are presented, both for neural ODEs in a standalone configuration and for neural ODEs with feedback connections, and practical techniques to impose the stability constraints during training are demonstrated. Based on the theoretical and experimental results, this work provides guidance as to when and how an accurate and stable neural ODE circuit model can be generated.

Flash floods pose serious risks to human life and infrastructure, leading to significant economic losses. While traditional conceptual models have long been used for runoff estimation, recent advancements in artificial intelligence have introduced machine learning and deep learning models for more accurate predictions. This study presents a deep learning framework that integrates Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), and Neural Ordinary Differential Equations (Neural ODEs) to enhance precipitation-induced runoff forecasting. A six-year dataset (2016–2022) from Al Ain, United Arab Emirates (UAE), was employed for model training, with validation conducted using data from a severe April 2024 flash flood. The proposed framework was compared against standalone CNN, RNN, and Neural ODE models to evaluate its predictive performance. Results show that the combination of the CNN’s feature extraction, the RNN’s temporal analysis, and the Neural ODE’s continuous-time modeling achieves superior accuracy, with an R2 value of 0.98, RMSE = 2.87 × 106, MAE = 1.13 × 106, and PBIAS of −8.38. These findings highlight the model’s ability to effectively capture complex hydrological dynamics. The framework provides a valuable tool for improving flash-flood forecasting and water resource management, especially in arid regions like the UAE. Future work may explore its application in different climates and integration with real-time monitoring systems.

Enhancing continuous time series modelling with a latent ODE-LSTM approach

Optical satellite sensors cannot see the earth’s surface through clouds. Despite the periodic revisit cycle, image sequences acquired by earth observation satellites are, therefore, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">irregularly</i> sampled in time. State-of-the-art methods for crop classification (and other time-series analysis tasks) rely on techniques that implicitly assume regular temporal spacing between observations, such as recurrent neural networks (RNNs). We propose to use neural ordinary differential equations (NODEs) in combination with RNNs to classify crop types in irregularly spaced image sequences. The resulting ODE-RNN models consist of two steps: an update step, where a recurrent unit assimilates new input data into the model’s hidden state, and a prediction step, in which NODE propagates the hidden state until the next observation arrives. The prediction step is based on a continuous representation of the latent dynamics, which has several advantages. At the conceptual level, it is a more natural way to describe the mechanisms that govern the phenological cycle. From a practical point of view, it makes it possible to sample the system state at arbitrary points in time such that one can integrate observations whenever they are available and extrapolate beyond the last observation. Our experiments show that ODE-RNN, indeed, improves classification accuracy over common baselines, such as LSTM, GRU, temporal convolutional network, and transformer. The gains are most prominent in the challenging scenario where only few observations are available (i.e., frequent cloud cover). Moreover, we show that the ability to extrapolate translates to better classification performance early in the season, which is important for forecasting.

In the context of high penetration of renewables, the need to build dynamic models of power system components based on accessible measurement data has become urgent. To address this challenge, firstly, a neural ordinary differential equations (ODE) module and a neural differential-algebraic equations (DAE) module are proposed to form a data-driven modeling framework that accurately captures components' dynamic characteristics and flexibly adapts to various interface settings. Secondly, analytical models and data-driven models learned by the neural ODE and DAE modules are integrated together and simulated simultaneously using unified transient stability simulation methods. Finally, the neural ODE and DAE modules are implemented with Python and made public on GitHub. Using the portal measurements, three simple but representative cases of excitation controller modeling, photovoltaic power plant modeling, and equivalent load modeling of a regional power network are carried out in the IEEE-39 system and 2383wp system. Neural dynamic model-integrated simulations are compared with the original model-based ones to verify the feasibility and potentiality of the proposed neural ODE and DAE modules.

This article presents an in-depth analysis and evaluation of artificial neural networks (ANNs) when applied to replicate trajectories in molecular dynamics (MD) simulations or other particle methods. This study focuses on several architectures—feedforward neural networks (FNNs), convolutional neural networks (CNNs), recurrent neural networks (RNNs), time convolutions (TCs), self-attention (SA), graph neural networks (GNNs), neural ordinary differential equation (ODENets), and an example of physics-informed machine learning (PIML) model—assessing their effectiveness and limitations in understanding and replicating the underlying physics of particle systems. Through this analysis, this paper introduces a comprehensive set of criteria designed to evaluate the capability of ANNs in this context. These criteria include the minimization of losses, the permutability of particle indices, the ability to predict trajectories recursively, the conservation of particles, the model’s handling of boundary conditions, and its scalability. Each network type is systematically examined to determine its strengths and weaknesses in adhering to these criteria. While, predictably, none of the networks fully meets all criteria, this study extends beyond the simple conclusion that only by integrating physics-based models into ANNs is it possible to fully replicate complex particle trajectories. Instead, it probes and delineates the extent to which various neural networks can “understand” and interpret aspects of the underlying physics, with each criterion targeting a distinct aspect of this understanding.

Accurate path travel time prediction is often hindered by sparse and heterogeneous traffic data. This paper proposes FusionODE-TT, a novel model designed to address these challenges by modeling traffic as a continuous-time process. The model features a Recurrent Neural Network encoder that processes multi-source time-series data to initialize a latent state vector, which then evolves over the prediction horizon using a Neural Ordinary Differential Equation (NODE). The core innovation is a guided fusion mechanism that leverages sparse but high-fidelity Automatic Vehicle Identification (AVI) data to apply strong, event-based corrections to the model’s continuous latent state, mitigating error accumulation in the prediction process. Experiments were conducted on a real-world dataset comprising AVI, GPS, and point sensor data from a major urban expressway. The experimental results demonstrate that the proposed model achieves superior accuracy, outperforming a suite of baseline models in terms of prediction accuracy and robustness. Furthermore, a comprehensive ablation study was performed to validate the efficacy of our design. The study quantitatively confirms that both the continuous-time dynamics modeled by the NODE and the guided fusion mechanism are essential components, each providing a significant and independent contribution to the model’s overall performance.

Traffic flow prediction is the core problem of traffic planning management and the basic basis of traffic control. However, the existing methods often do not fully consider the differences of different road nodes when modeling spatial dependence. In this paper, we propose a traffic flow prediction framework called graph neural ordinary differential equation recurrent neural network (GODE-RNN). Our method fuses graph convolution into neural ordinary differential equation to capture the spatial correlation, while the temporal correlation is extracted by sequence-to-sequence model. Specifically, we use neural ordinary differential equation (ODE) to expand the hidden state of graph convolution network, and different weights are used to weight and sum the hidden states of ODE for different road nodes, so that the network can learn the difference of spatial correlation of different road nodes independently. On the real-world data set, our method is significantly better than the baseline.

Neural controlled differential equations (NCDEs), which are continuous analogues to recurrent neural networks (RNNs), are a specialized model in (irregular) time-series processing. In comparison with similar models, e.g., neural ordinary differential equations (NODEs), the key distinctive characteristics of NCDEs are i) the adoption of the continuous path created by an interpolation algorithm from each raw discrete time-series sample and ii) the adoption of the Riemann--Stieltjes integral. It is the continuous path which makes NCDEs be analogues to continuous RNNs. However, NCDEs use existing interpolation algorithms to create the path, which is unclear whether they can create an optimal path. To this end, we present a method to generate another latent path (rather than relying on existing interpolation algorithms), which is identical to learning an appropriate interpolation method. We design an encoder-decoder module based on NCDEs and NODEs, and a special training method for it. Our method shows the best performance in both time-series classification and forecasting.

Fatigue damage characterization for composite laminates using deep learning and laser ultrasonic

This article introduces a deep state‐space modeling (DSSM) framework tailored for monitoring complex and dynamic deteriorating systems operating under varying operating conditions and sensor‐based condition monitoring. By integrating stochastic recurrent neural networks (RNNs) with a generative model in a variational autoencoder (VAE) form, the proposed framework effectively approximates the complex behaviors inherent in degrading systems with latent states and captures long‐term dependencies and latent dynamics without relying on unrealistic distributional and parametric assumptions. The framework leverages RNNs to model temporal dependencies and VAE to model robust probabilistic inference, enabling accurate latent state estimation and time to event (TE) prediction. Moreover, its flexibility extends to accommodating both continuous and discrete latent states, enriching the representation of underlying data dynamics. By performing joint inference and learning, utilizing VAE for system dynamics modeling offers significant advantages over traditional state‐space models (SSMs), which require high computational resources and tuning. We have tested this framework using both simulation and real‐world datasets. Also, a case study on a wind turbine dataset demonstrates the effectiveness of the proposed framework in early fault detection.