Neural Ordinary Differential Equations Research Articles

Attention Mechanism and Neural Ordinary Differential Equations for the Incomplete Trajectory Information Prediction of Unmanned Aerial Vehicles Using Airborne Radar

Attention Mechanism and Neural Ordinary Differential Equations for the Incomplete Trajectory Information Prediction of Unmanned Aerial Vehicles Using Airborne Radar

Adaptive Decision Spatio-temporal neural ODE for traffic flow forecasting with Multi-Kernel Temporal Dynamic Dilation Convolution

Traffic flow prediction is crucial for efficient traffic management. It involves predicting vehicle movement patterns to reduce congestion and enhance traffic flow. However, the highly non-linear and complex patterns commonly observed in traffic flow pose significant challenges for this task. Current Graph Neural Network (GNN) models often construct shallow networks, which limits their ability to extract deeper spatio-temporal representations. Neural ordinary differential equations for traffic prediction address over-smoothing but require significant computational resources, leading to inefficiencies, and sometimes deeper networks may lead to poorer predictions for complex traffic information. In this study, we propose an Adaptive Decision spatio-temporal Neural Ordinary Differential Network, which can adaptively determine the number of layers of ODE according to the complexity of traffic information. It can solve the over-smoothing problem better, improving overall efficiency and prediction accuracy. In addition, traditional temporal convolution methods make it difficult to deal with complex and variable traffic time information with a large time span. Therefore, we introduce a multi-kernel temporal dynamic expansive convolution to handle the traffic time information. Multi-kernel temporal dynamic expansive convolution employs a dynamic dilation strategy, dynamically adjusting the network’s receptive field across levels, effectively capturing temporal dependencies, and can better adapt to the changing time data of traffic information. Additionally, multi-kernel temporal dynamic expansive convolution integrates multi-scale convolution kernels, enabling the model to learn features across diverse temporal scales. We evaluated our proposed method on several real-world traffic datasets. Experimental results show that our method outperformed state-of-the-art benchmarks.

Graph neural ordinary differential equations for epidemic forecasting

Graph neural ordinary differential equations for epidemic forecasting

A grey-box model with neural ordinary differential equations for the slow voltage dynamics of lithium-ion batteries: Application to single-cell experiments

Lithium-ion batteries exhibit a complex, nonlinear and dynamic voltage behaviour. Modelling their slow dynamics is a challenge due to the multiple potential causes involved. We present here a neural equivalent circuit model for lithium-ion batteries including slow voltage dynamics. The model uses an equivalent circuit with voltage source, series resistor, and diffusion element. The series resistance is parameterized using neural networks. The diffusion element is based on a discretized form of Fickian diffusion, parameterized using a neural network and learnable parameters. It is flexible to represent not only Warburg behaviour, but also resistor-capacitor-type dynamics. Mathematically, the resulting model is given by a differential–algebraic equation system combining ordinary and neural differential equations. Therefore, the model combines concepts of both physical theory (white-box model) and artificial intelligence (black-box model) to a combined framework (grey-box model). We apply this approach to a lithium iron phosphate based lithium-ion battery cell. The experimental voltage behaviour during constant-current cycles as well as the dynamics during pulse tests are well reproduced by the model. Only at very high and very low states of charge the simulation significantly deviates from experiments, which might result from insufficient training data in these regions.

Forecasting dynamics by an incomplete equation of motion and an auto-encoder Koopman operator

The governing equation of mass–spring–damper dynamics usually consists of a known structure and undetermined nonlinear components. This work proposes a physics-informed method to learn the unknown component by modeling its evolution with an auto-encoder Koopman operator. The Koopman operator is well known for its global linearization of dynamics and sensitivity to spectral modes. The undetermined component in an oscillatory equation of motion (EOM) is first reconstructed by a time-delay neural network whose inputs stack past sequences of observables and the control. Then, the evolution of the undetermined component is globally linearized and propagated by the Koopman operator in a latent space created by an auto-encoder neural network. Ultimately, observables at the future timestep are forecasted by timestepping the original EOM. Uniting an incomplete EOM and the Koopman operator, the proposed physics-informed Koopman neural ordinary differential equation (PiKNODE) method shows competitive edges compared with several black- and grey-box benchmarks in four synthetic and experimental case studies. Additionally, an anomaly-detection technique based on the Jensen–Shannon distance is proposed to boost the practicality of the PiKNODE for structural health monitoring.

Neural Ordinary Differential Equation Networks for Fintech Applications Using Internet of Things

Neural Ordinary Differential Equation Networks for Fintech Applications Using Internet of Things

Optimal potential shaping on SE(3) via neural ordinary differential equations on Lie groups

This work presents a novel approach for the optimization of dynamic systems on finite-dimensional Lie groups. We rephrase dynamic systems as so-called neural ordinary differential equations (neural ODEs), and formulate the optimization problem on Lie groups. A gradient descent optimization algorithm is presented to tackle the optimization numerically. Our algorithm is scalable, and applicable to any finite-dimensional Lie group, including matrix Lie groups. By representing the system at the Lie algebra level, we reduce the computational cost of the gradient computation. In an extensive example, optimal potential energy shaping for control of a rigid body is treated. The optimal control problem is phrased as an optimization of a neural ODE on the Lie group SE(3), and the controller is iteratively optimized. The final controller is validated on a state-regulation task.

Deep JKO: Time-implicit particle methods for general nonlinear gradient flows

We develop novel neural network-based implicit particle methods to compute high-dimensional Wasserstein-type gradient flows with linear and nonlinear mobility functions. The main idea is to use the Lagrangian formulation in the Jordan–Kinderlehrer–Otto (JKO) framework, where the velocity field is approximated using a neural network. We leverage the formulations from the neural ordinary differential equation (neural ODE) in the context of continuous normalizing flow for efficient density computation. Additionally, we make use of an explicit recurrence relation for computing derivatives, which greatly streamlines the backpropagation process. Our methodology demonstrates versatility in handling a wide range of gradient flows, accommodating various potential functions and nonlinear mobility scenarios. Extensive experiments demonstrate the efficacy of our approach, including an illustrative example from Bayesian inverse problems. This underscores that our scheme provides a viable alternative solver for the Kalman-Wasserstein gradient flow.

Stable predictive control of continuous stirred-tank reactors using deep learning

In this work, a Lyapunov-based predictive control method utilizing deep learning techniques is proposed for driving continuous-time nonlinear processes towards the desired equilibrium point. Initially, a neural ordinary differential equation model is developed to learn the continuous-time dynamics from discrete-time sampling data. Subsequently, the theoretical conditions that the control Lyapunov function (CLF) developed on the learned continuous-time system is feasible to act on the actual system through a zero-order-holder are analyzed. Building upon the theoretical analysis and the constructed continuous-time dynamics, a deep neural network control Lyapunov function (DNN-CLF) controller is developed. Importantly, the DNN-CLF is independent of the affine system formulation, effectively addressing the challenge of designing the CLF. Benefiting from the stability guarantee provided by the established DNN-CLF, it is integrated into the model predictive control (MPC) framework to develop Lyapunov-based MPC to further improve the control performance. To ensure safety, considering error propagation and associated risks with enlarging the feasible region of the CLF, tubes are constructed and integrated into the MPC scheme. Finally, a numerical simulation of a chemical process is conducted to validate the efficiency of the proposed method.

Mutli-modal straight flow matching for accelerated MR imaging

Diffusion models have garnered great interest lately in Magnetic Resonance (MR) image reconstruction. A key component of generating high-quality samples from noise is iterative denoising for thousands of steps. However, the complexity of inference steps has limited its applications. To solve the challenge in obtaining high-quality reconstructed images with fewer inference steps and computational complexity, we introduce a novel straight flow matching, based on a neural ordinary differential equation (ODE) generative model. Our model creates a linear path between undersampled images and reconstructed images, which can be accurately simulated with a few Euler steps. Furthermore, we propose a multi-modal straight flow matching model, which uses relatively easily available modalities as supplementary information to guide the reconstruction of target modalities. We introduce the low frequency fusion layer and the high frequency fusion layer into our multi-modal model, which has been proved to produce promising results in fusion tasks. The proposed multi-modal straight flow matching (MMSflow) achieves state-of-the-art performances in task of reconstruction in fastMRI and Brats-2020 and improves the sampling rate by an order of magnitude than other methods based on stochastic differential equations (SDE).

DDE: Deep Dynamic Epidemiological Modeling for Infectious Illness Development Forecasting in Multi-level Geographic Entities.

Understanding and addressing the dynamics of infectious diseases, such as coronavirus disease 2019, are essential for effectively managing the current situation and developing intervention strategies. Epidemiologists commonly use mathematical models, known as epidemiological equations (EE), to simulate disease spread. However, accurately estimating the parameters of these models can be challenging due to factors like variations in social distancing policies and intervention strategies. In this study, we propose a novel method called deep dynamic epidemiological modeling (DDE) to address these challenges. The DDE method combines the strengths of EE with the capabilities of deep neural networks to improve the accuracy of fitting real-world data. In DDE, we apply neural ordinary differential equations to solve variant-specific equations, ensuring a more precise fit for disease progression in different geographic regions. In the experiment, we tested the performance of the DDE method and other state-of-the-art methods using real-world data from five diverse geographic entities: the USA, Colombia, South Africa, Wuhan in China, and Piedmont in Italy. Compared to the state-of-the-art method, DDE significantly improved accuracy, with an average fitting Pearson coefficient exceeding 0.97 across the five geographic entities. In summary, the DDE method enhances the accuracy of parameter fitting in epidemiological models and provides a foundation for constructing simpler models adaptable to different geographic areas.

Continuous-discrete extended Kalman filtering based on the neural ordinary differential equations method

In bearings-only tracking (BOT), the accuracy of state estimation significantly diminishes when confronted with an unknown target motion model. Therefore, this study proposed a novel continuous-discrete filtering algorithm, the neural ordinary differential equations-based continuous-discrete extended Kalman filtering (NODE-CD-EKF). The approach utilizes the neural ordinary differential equation (NODE) to build the target's system model, which is then used in the time update to precisely predict the target state at the next time point within the continuous-discrete Kalman framework. The NODE concisely and accurately describes the motion model in the form of ordinary differential equations in a data-driven way. This approach addresses the limitations of traditional Kalman filters, which suffer from degraded performance due to a lack of a precise mathematical model of target states. Our method is evaluated and compared to existing multiple-model and other data-driven algorithms in simulations of BOT and another scenario using the Accumulated Root Mean Square Error (ARMSE) of position. The ARMSE of our method is only 16.30 % of the multiple-model methods at least. Overall, this approach outperforms other multiple-model and data-driven algorithms and possesses the potential for solving other nonlinear tasks in real-world applications.

Biologically informed NeuralODEs for genome-wide regulatory dynamics

BackgroundGene regulatory network (GRN) models that are formulated as ordinary differential equations (ODEs) can accurately explain temporal gene expression patterns and promise to yield new insights into important cellular processes, disease progression, and intervention design. Learning such gene regulatory ODEs is challenging, since we want to predict the evolution of gene expression in a way that accurately encodes the underlying GRN governing the dynamics and the nonlinear functional relationships between genes. Most widely used ODE estimation methods either impose too many parametric restrictions or are not guided by meaningful biological insights, both of which impede either scalability, explainability, or both.ResultsWe developed PHOENIX, a modeling framework based on neural ordinary differential equations (NeuralODEs) and Hill-Langmuir kinetics, that overcomes limitations of other methods by flexibly incorporating prior domain knowledge and biological constraints to promote sparse, biologically interpretable representations of GRN ODEs. We tested the accuracy of PHOENIX in a series of in silico experiments, benchmarking it against several currently used tools. We demonstrated PHOENIX’s flexibility by modeling regulation of oscillating expression profiles obtained from synchronized yeast cells. We also assessed the scalability of PHOENIX by modeling genome-scale GRNs for breast cancer samples ordered in pseudotime and for B cells treated with Rituximab.ConclusionsPHOENIX uses a combination of user-defined prior knowledge and functional forms from systems biology to encode biological “first principles” as soft constraints on the GRN allowing us to predict subsequent gene expression patterns in a biologically explainable manner.

Applying Neural ODEs to Derive a Mechanism-Based Model for Characterizing Maturation-Related Serum Creatinine Dynamics in Preterm Newborns.

Serum creatinine in neonates follows complex dynamics due to maturation processes, most pronounced in the first few weeks of life. The development of a mechanism-based model describing complex dynamics requires high expertise in pharmacometric (PMX) modeling and substantial model development time. A recently published machine learning (ML) approach of low-dimensional neural ordinary differential equations (NODEs) is capable of modeling such data from newborns automatically. However, this efficient data-driven approach in itself does not result in a clinically interpretable model. In this work, an approach to deriving an interpretable model with reasonable PMX-type functions is presented. This "translation" was applied to derive a PMX model for serum creatinine in neonates considering maturation processes and covariates. The developed model was compared to a previously published mechanism-based PMX model whereas both models had similar mechanistic structures. The developed model was then utilized to simulate serum creatinine concentrations in the first few weeks of life considering different covariate values for gestational age and birth weight. The reference serum creatinine values derived from these simulations are consistent with observed serum creatinine values and previously published reference values. Thus, the presented NODE-based ML approach to model complex serum creatinine dynamics in newborns and derive interpretable, mathematical-statistical components similar to those in a conventional PMX model demonstrates a novel, viable approach to facilitate the modeling of complex dynamics in clinical settings and pediatric drug development.

An embedding model for temporal knowledge graphs with long and irregular intervals

As the basis of various downstream applications such as automated question answering and knowledge reasoning, Temporal Knowledge Graph (TKG) embedding learning has attracted many interests. Recent works have explored several methods to learn dynamic embedding for TKGs and led to success in many applications, however, they are difficult to be optimized. In order to accurately catch semantic information propagation of entity-to-entity and entity-to-relation, as well as track dynamics of entities and relations in continuous time, we proposed a novel TKG embedding model ODETKGE. Instead of using recurrent neural network to model dynamics of semantic representations in continuous time, we leverage neural ordinary differential equations to model the dynamic evolution, which enable us to capture long-range dependencies between embeddings at different timestamps, track the dynamics of TKGs with irregular intervals and remain computationally efficient. Furthermore, to effectively model semantics of knowledge graph structure, we combine entity-relation collaboratively updated graph convolutional networks and neural ordinary differential equations. Additionally, to acquire important semantics such as creation and disappearance of relations, we utilize a graph transition layer. Comparing with RNN based models, the proposed ODETKGE can use fewer parameters to achieve deeper layers, and make the transformation of representations between two timestamps smoother. Extensive experiments have been executed on real-world datasets and the results verify that ODETKGE is superior than the state-of-the-art models in link forecasting, long-interval link forecasting, irregular-interval link forecasting, ablation study and training time.

Continuous patient state attention model for addressing irregularity in electronic health records

BackgroundIrregular time series (ITS) are common in healthcare as patient data is recorded in an electronic health record (EHR) system as per clinical guidelines/requirements but not for research and depends on a patient’s health status. Due to irregularity, it is challenging to develop machine learning techniques to uncover vast intelligence hidden in EHR big data, without losing performance on downstream patient outcome prediction tasks.MethodsIn this paper, we propose Perceiver, a cross-attention-based transformer variant that is computationally efficient and can handle long sequences of time series in healthcare. We further develop continuous patient state attention models, using Perceiver and transformer to deal with ITS in EHR. The continuous patient state models utilise neural ordinary differential equations to learn patient health dynamics, i.e., patient health trajectory from observed irregular time steps, which enables them to sample patient state at any time.ResultsThe proposed models’ performance on in-hospital mortality prediction task on PhysioNet-2012 challenge and MIMIC-III datasets is examined. Perceiver model either outperforms or performs at par with baselines, and reduces computations by about nine times when compared to the transformer model, with no significant loss of performance. Experiments to examine irregularity in healthcare reveal that continuous patient state models outperform baselines. Moreover, the predictive uncertainty of the model is used to refer extremely uncertain cases to clinicians, which enhances the model’s performance. Code is publicly available and verified at https://codeocean.com/capsule/4587224.ConclusionsPerceiver presents a computationally efficient potential alternative for processing long sequences of time series in healthcare, and the continuous patient state attention models outperform the traditional and advanced techniques to handle irregularity in the time series. Moreover, the predictive uncertainty of the model helps in the development of transparent and trustworthy systems, which can be utilised as per the availability of clinicians.

Using neural ordinary differential equations to predict complex ecological dynamics from population density data.

Simple models have been used to describe ecological processes for over a century. However, the complexity of ecological systems makes simple models subject to modelling bias due to simplifying assumptions or unaccounted factors, limiting their predictive power. Neural ordinary differential equations (NODEs) have surged as a machine-learning algorithm that preserves the dynamic nature of the data (Chen et al. 2018 Adv. Neural Inf. Process. Syst.). Although preserving the dynamics in the data is an advantage, the question of how NODEs perform as a forecasting tool of ecological communities is unanswered. Here, we explore this question using simulated time series of competing species in a time-varying environment. We find that NODEs provide more precise forecasts than autoregressive integrated moving average (ARIMA) models. We also find that untuned NODEs have a similar forecasting accuracy to untuned long-short term memory neural networks and both are outperformed in accuracy and precision by empirical dynamical modelling . However, we also find NODEs generally outperform all other methods when evaluating with the interval score, which evaluates precision and accuracy in terms of prediction intervals rather than pointwise accuracy. We also discuss ways to improve the forecasting performance of NODEs. The power of a forecasting tool such as NODEs is that it can provide insights into population dynamics and should thus broaden the approaches to studying time series of ecological communities.

LFT: Neural Ordinary Differential Equations With Learnable Final-Time.

Since the last decade, deep neural networks have shown remarkable capability in learning representations. The recently proposed neural ordinary differential equations (NODEs) can be viewed as the continuous-time equivalence of residual neural networks. It has been shown that NODEs have a tremendous advantage over the conventional counterparts in terms of spatial complexity for modeling continuous-time processes. However, existing NODEs methods entail their final time to be specified in advance, precluding the models from choosing a desirable final time and limiting their expressive capabilities. In this article, we propose learnable final-time (LFT) NODEs to overcome this limitation. LFT rebuilds the NODEs learning process as a final-time-free optimal control problem and employs the calculus of variations to derive the learning algorithm of NODEs. In contrast to existing NODEs methods, the new approach empowers the NODEs models to choose their suitable final time, thus being more flexible in adjusting the model depth for given tasks. Additionally, we analyze the gradient estimation errors caused by numerical ordinary differential equations (ODEs) solvers and employ checkpoint-based methods to obtain accurate gradients. We demonstrate the effectiveness of the proposed method with experimental results on continuous normalizing flows (CNFs) and feedforward models.

Towards complex dynamic physics system simulation with graph neural ordinary equations

The great learning ability of deep learning facilitates us to comprehend the real physical world, making learning to simulate complicated particle systems a promising endeavour both in academia and industry. However, the complex laws of the physical world pose significant challenges to the learning based simulations, such as the varying spatial dependencies between interacting particles and varying temporal dependencies between particle system states in different time stamps, which dominate particles’ interacting behavior and the physical systems’ evolution patterns. Existing learning based methods fail to fully account for the complexities, making them unable to yield satisfactory simulations. To better comprehend the complex physical laws, we propose a novel model – Graph Networks with Spatial–Temporal neural Ordinary Differential Equations (GNSTODE) – that characterizes the varying spatial and temporal dependencies in particle systems using a united end-to-end framework. Through training with real-world particle–particle interaction observations, GNSTODE can simulate any possible particle systems with high precisions. We empirically evaluate GNSTODE’s simulation performance on two real-world particle systems, Gravity and Coulomb, with varying levels of spatial and temporal dependencies. The results show that GNSTODE yields better simulations than state-of-the-art methods, showing that GNSTODE can serve as an effective tool for particle simulation in real-world applications. Our code is made available at https://github.com/Guangsi-Shi/AI-for-physics-GNSTODE.

ASV station keeping under wind disturbances using neural network simulation error minimization model predictive control

AbstractStation keeping is an essential maneuver for autonomous surface vehicles (ASVs), mainly when used in confined spaces, to carry out surveys that require the ASV to keep its position or in collaboration with other vehicles where the relative position has an impact over the mission. However, this maneuver can become challenging for classic feedback controllers due to the need for an accurate model of the ASV dynamics and the environmental disturbances. This work proposes a model predictive controller using neural network simulation error minimization (NNSEM–MPC) to accurately predict the dynamics of the ASV under wind disturbances. The performance of the proposed scheme under wind disturbances is tested and compared against other controllers in simulation, using the robotics operating system and the multipurpose simulation environment Gazebo. A set of six tests was conducted by combining two varying wind speeds that are modeled as the Harris spectrum and three wind directions (, , and ). The simulation results clearly show the advantage of the NNSEM–MPC over the following methods: backstepping controller, sliding mode controller, simplified dynamics MPC (SD‐MPC), neural ordinary differential equation MPC (NODE‐MPC), and knowledge‐based NODE MPC. The proposed NNSEM–MPC approach performs better than the rest in five out of the six test conditions, and it is the second best in the remaining test case, reducing the mean position and heading error by at least % ( m) and % (), respectively, across all the test cases. In terms of execution speed, the proposed NNSEM–MPC is at least 36% faster than the rest of the MPC controllers. The field experiments on two different ASV platforms showed that ASVs can effectively keep the station utilizing the proposed method, with a position error as low as m and a heading error as low as within time windows of at least s. This would increase the potential applications of ASVs for launch, recovery, and replenishment in long‐term surveys in collaboration with other autonomous systems.