Parametric Ordinary Differential Equations Research Articles

Generalized Tikhonov regularization in estimation of ordinary differential equations models

We consider estimation of parameters in models defined by systems of ordinary differential equations (ODEs). This problem is important because many processes in different fields of science are modelled by systems of ODEs. Various estimation methods based on smoothing have been suggested to bypass numerical integration of the ODE system. In this paper, we do not propose another method based on smoothing but show how some of the existing ones can be brought together under one unifying framework. The framework is based on generalized Tikhonov regularization and extremum estimation. We define an approximation of the ODE solution by viewing the system of ODEs as an operator equation and exploiting the connection with regularization theory. Combining the introduced regularized solution with an extremum criterion function provides a general framework for estimating parameters in ODEs, which can handle partially observed systems. If the extremum criterion function is the negative log‐likelihood, then suitable regularized solutions yield estimators that are consistent and asymptotically efficient. The well‐known generalized profiling procedure fits into the proposed framework. Copyright © 2016 John Wiley & Sons, Ltd.

Higher-order Lie symmetries in identifiability and predictability analysis of dynamic models.

Parameter estimation in ordinary differential equations (ODEs) has manifold applications not only in physics but also in the life sciences. When estimating the ODE parameters from experimentally observed data, the modeler is frequently concerned with the question of parameter identifiability. The source of parameter nonidentifiability is tightly related to Lie group symmetries. In the present work, we establish a direct search algorithm for the determination of admitted Lie group symmetries. We clarify the relationship between admitted symmetries and parameter nonidentifiability. The proposed algorithm is applied to illustrative toy models as well as a data-based ODE model of the NFκB signaling pathway. We find that besides translations and scaling transformations also higher-order transformations play a role. Enabled by the knowledge about the explicit underlying symmetry transformations, we show how models with nonidentifiable parameters can still be employed to make reliable predictions.

Statistical diagnosis for HIV dynamics based on mean shift outlier model

Ordinary differential equation (ODE) are widely used for quantifying HIV viral dynamics. It is interesting but challenging to estimate ODE parameters from noisy data, especially when the data have some outliers. In this study, the authors use the Mean Shift Outlier Model (MSOM) to detect outliers in HIV model based on the two-step estimation of ODE. Approximate formula for shift parameter is derived. Furthermore, a score test statistic is constructed and its approximating distribution is established. The simulation results show that: 1) The boundary points have more impact on the parameter estimation relative to interior points. 2) The proposed procedure can detect the outliers effectively. The authors illustrate the proposed approach using an application example from an HIV clinical trial and find similar pattern to the simulation studies.

Regularized Semiparametric Estimation for Ordinary Differential Equations

Ordinary differential equations (ODEs) are widely used in modeling dynamic systems and have ample applications in the fields of physics, engineering, economics, and biological sciences. The ODE parameters often possess physiological meanings and can help scientists gain better understanding of the system. One key interest is thus to well estimate these parameters. Ideally, constant parameters are preferred due to their easy interpretation. In reality, however, constant parameters can be too restrictive such that even after incorporating error terms, there could still be unknown sources of disturbance that lead to poor agreement between observed data and the estimated ODE system. In this article, we address this issue and accommodate short-term interferences by allowing parameters to vary with time. We propose a new regularized estimation procedure on the time-varying parameters of an ODE system so that these parameters could change with time during transitions but remain constants within stable stages. We found, through simulation studies, that the proposed method performs well and tends to have less variation in comparison to the nonregularized approach. On the theoretical front, we derive finite-sample estimation error bounds for the proposed method. Applications of the proposed method to modeling the hare–lynx relationship and the measles incidence dynamic in Ontario, Canada lead to satisfactory and meaningful results. Supplementary materials for this article are available online.

Higher-order Discrete Adjoint ODE Solver in C++ for Dynamic Optimization

Abstract Parametric ordinary differential equations (ODE) arise in many engineering applications. We consider ODE solutions to be embedded in an overall objective function which is to be minimized, e.g. for parameter estimation. For derivative-based optimization algorithms adjoint methods should be used. In this article, we present a discrete adjoint ODE integration framework written in C + + (NIXE 2.0) combined with Algorithmic Differentiation by overloading (dco/c + +). All required derivatives, i.e. Jacobians for the integration as well as gradients and Hessians for the optimization, are generated automatically. With this framework, derivatives of arbitrary order can be implemented with minimal programming effort. The practicability of this approach is demonstrated in a dynamic parameter estimation case study for a batch fermentation process using sequential method of dynamic optimization. Ipopt is used as the optimizer which requires second derivatives.

Stable Set-Valued Integration of Nonlinear Dynamic Systems using Affine Set-Parameterizations

Many set-valued integration algorithms for parametric ordinary differential equations (ODEs) implement a combination of Taylor series expansion with either interval arithmetic or Taylor model arithmetic. Due to the wrapping effect, the diameter of the solution-set enclosures computed with these algorithms typically diverges to infinity on finite integration horizons, even though the ODE trajectories themselves may be asymptotically stable. This paper starts by describing a new discretized set-valued integration algorithm that uses a predictor-validation approach to propagate generic affine set-parameterizations, whose images are guaranteed to enclose the ODE solution set. Sufficient conditions are then derived for this algorithm to be locally asymptotically stable, in the sense that the computed enclosures are guaranteed to remain stable on infinite time horizons when applied to a dynamic system in the neighborhood of a locally asymptotically stable periodic orbit (or equilibrium point). The key requirement here is quadratic Hausdorff convergence of function extensions in the chosen affine set-parameterization, which is proved to be the case, for instance, for Taylor models with ellipsoidal remainders. These stability properties are illustrated with the case study of a cubic oscillator system.

Inferring reaction systems from ordinary differential equations

In Mathematical Biology, many dynamical models of biochemical reaction systems are presented with Ordinary Differential Equations (ODE). Once kinetic parameter values are fixed, this simple mathematical formalism completely defines the dynamical behavior of a system of biochemical reactions and provides powerful tools for deterministic simulations, parameter sensitivity analysis, bifurcation analysis, etc. However, without requiring any information on the reaction kinetics and parameter values, various qualitative analyses can be performed using the structure of the reactions, provided the reactants, products and modifiers of each reaction are precisely defined. In order to apply these structural methods to parametric ODE models, we study a mathematical condition for expressing the consistency between the structure and the kinetics of a reaction, without restricting to Mass Action law kinetics. This condition, satisfied in particular by standard kinetic laws, entails a remarkable property of independence of the influence graph from the kinetics of the reactions. We derive from this study a heuristic algorithm which, given a system of ODEs as input, computes a system of reactions with the same ODE semantics, by inferring well-formed reactions whenever possible. We show how this strategy is capable of automatically curating the writing of ODE models in SBML, and present some statistics obtained on the model repository biomodels.net.

Time-course window estimator for ordinary differential equations linear in the parameters

In many applications obtaining ordinary differential equation descriptions of dynamic processes is scientifically important. In both, Bayesian and likelihood approaches for estimating parameters of ordinary differential equations, the speed and the convergence of the estimation procedure may crucially depend on the choice of initial values of the parameters. Extending previous work, we show in this paper how using window smoothing yields a fast estimator for systems that are linear in the parameters. Using weak assumptions on the measurement error, we prove that the proposed estimator is $$\sqrt{n}$$ -consistent. The estimator does not require an initial guess for the parameters and is computationally fast and, therefore, it can serve as a good initial estimate for more efficient estimators. In simulation studies and on real data we illustrate the performance of the proposed estimator.

Estimation of nonlinear differential equation model for glucose-insulin dynamics in type I diabetic patients using generalized smoothing.

In this work, we develop an ordinary differential equations (ODE) model of physiological regulation of glycemia in type 1 diabetes mellitus (T1DM) patients in response to meals and intravenous insulin infusion. Unlike for majority of existing mathematical models of glucose-insulin dynamics, parameters in our model are estimable from a relatively small number of noisy observations of plasma glucose and insulin concentrations. For estimation, we adopt the generalized smoothing estimation of nonlinear dynamic systems of Ramsay et al. (2007). In this framework, the ODE solution is approximated with a penalized spline, where the ODE model is incorporated in the penalty. We propose to optimize the generalized smoothing by using penalty weights that minimize the covariance penalties criterion (Efron, 2004). The covariance penalties criterion provides an estimate of the prediction error for nonlinear estimation rules resulting from nonlinear and/or non-homogeneous ODE models, such as our model of glucose-insulin dynamics. We also propose to select the optimal number and location of knots for B-spline bases used to represent the ODE solution. The results of the small simulation study demonstrate advantages of optimized generalized smoothing in terms of smaller estimation errors for ODE parameters and smaller prediction errors for solutions of differential equations. Using the proposed approach to analyze the glucose and insulin concentration data in T1DM patients we obtained good approximation of global glucose-insulin dynamics and physiologically meaningful parameter estimates.

Generalized Derivatives for Solutions of Parametric Ordinary Differential Equations with Non-differentiable Right-Hand Sides

Sensitivity analysis provides useful information for equation-solving, optimization, and post-optimality analysis. However, obtaining useful sensitivity information for systems with nonsmooth dynamic systems embedded is a challenging task. In this article, for any locally Lipschitz continuous mapping between finite-dimensional Euclidean spaces, Nesterov's lexicographic derivatives are shown to be elements of the plenary hull of the (Clarke) generalized Jacobian whenever they exist. It is argued that in applications, and in several established results in nonsmooth analysis, elements of the plenary hull of the generalized Jacobian of a locally Lipschitz continuous function are no less useful than elements of the generalized Jacobian itself. Directional derivatives and lexicographic derivatives of solutions of parametric ordinary differential equation (ODE) systems are expressed as the unique solutions of corresponding ODE systems, under Caratheodory-style assumptions. Hence, the scope of numerical methods for nonsmooth equation-solving and local optimization is extended to systems with nonsmooth parametric ODEs embedded.

Stochastic differential equations as a tool to regularize the parameter estimation problem for continuous time dynamical systems given discrete time measurements

In this paper we consider the problem of estimating parameters in ordinary differential equations given discrete time experimental data. The impact of going from an ordinary to a stochastic differential equation setting is investigated as a tool to overcome the problem of local minima in the objective function. Using two different models, it is demonstrated that by allowing noise in the underlying model itself, the objective functions to be minimized in the parameter estimation procedures are regularized in the sense that the number of local minima is reduced and better convergence is achieved. The advantage of using stochastic differential equations is that the actual states in the model are predicted from data and this will allow the prediction to stay close to data even when the parameters in the model is incorrect. The extended Kalman filter is used as a state estimator and sensitivity equations are provided to give an accurate calculation of the gradient of the objective function. The method is illustrated using in silico data from the FitzHugh–Nagumo model for excitable media and the Lotka–Volterra predator–prey system. The proposed method performs well on the models considered, and is able to regularize the objective function in both models. This leads to parameter estimation problems with fewer local minima which can be solved by efficient gradient-based methods.

Parametric Estimation of Ordinary Differential Equations With Orthogonality Conditions

Differential equations are commonly used to model dynamical deterministic systems in applications. When statistical parameter estimation is required to calibrate theoretical models to data, classical statistical estimators are often confronted to complex and potentially ill-posed optimization problem. As a consequence, alternative estimators to classical parametric estimators are needed for obtaining reliable estimates. We propose a gradient matching approach for the estimation of parametric Ordinary Differential Equations (ODE) observed with noise. Starting from a nonparametric proxy of a true solution of the ODE, we build a parametric estimator based on a variational characterization of the solution. As a Generalized Moment Estimator, our estimator must satisfy a set of orthogonal conditions that are solved in the least squares sense. Despite the use of a nonparametric estimator, we prove the - consistency and asymptotic normality of the Orthogonal Conditions estimator. We can derive confidence sets thanks to a closed-form expression for the asymptotic variance. Finally, the OC estimator is compared to classical estimators in several (simulated and real) experiments and ODE models to show its versatility and relevance with respect to classical Gradient Matching and Nonlinear Least Squares estimators. In particular, we show on a real dataset of influenza infection that the approach gives reliable estimates. Moreover, we show that our approach can deal directly with more elaborated models such as Delay Differential Equation (DDE). Supplementary materials for this article are available online.

Using gene expression programming to infer gene regulatory networks from time-series data

Gene regulatory networks inference is currently a topic under heavy research in the systems biology field. In this paper, gene regulatory networks are inferred via evolutionary model based on time-series microarray data. A non-linear differential equation model is adopted. Gene expression programming (GEP) is applied to identify the structure of the model and least mean square (LMS) is used to optimize the parameters in ordinary differential equations (ODEs). The proposed work has been first verified by synthetic data with noise-free and noisy time-series data, respectively, and then its effectiveness is confirmed by three real time-series expression datasets. Finally, a gene regulatory network was constructed with 12 Yeast genes. Experimental results demonstrate that our model can improve the prediction accuracy of microarray time-series data effectively.

Bayesian P-spline estimation in hierarchical models specified by systems of affine differential equations

Ordinary differential equations (ODEs) are widely used to model physical, chemical and biological processes. Current methods for parameter estimation can be computationally intensive and/or not suitable for inference and prediction. Frequentist approaches based on ODE-penalized smoothing techniques have recently solved part of these drawbacks. A full Bayesian approach based on ODE-penalized B-splines is proposed to jointly estimate ODE parameters and state functions from affine systems of differential equations. Simulations inspired by pharmacokinetic studies show that the proposed method provides comparable results to methods based on explicit solution of the ODEs and outperforms the frequentist ODE-penalized smoothing approach. The basic model is extended to a hierarchical one in order to study cases where several subjects are involved. This Bayesian hierarchical approach is illustrated on real data for the study of perfusion ratio after a femoral artery occlusion. Model selection is feasible through the analysis of the posterior distributions of the ODE adhesion parameters and is illustrated on a real pharmacokinetic dataset.

Improved relaxations for the parametric solutions of ODEs using differential inequalities

A new method is described for computing nonlinear convex and concave relaxations of the solutions of parametric ordinary differential equations (ODEs). Such relaxations enable deterministic global optimization algorithms to be applied to problems with ODEs embedded, which arise in a wide variety of engineering applications. The proposed method computes relaxations as the solutions of an auxiliary system of ODEs, and a method for automatically constructing and numerically solving appropriate auxiliary ODEs is presented. This approach is similar to two existing methods, which are analyzed and shown to have undesirable properties that are avoided by the new method. Two numerical examples demonstrate that these improvements lead to significantly tighter relaxations than previous methods.

Nonlinear convex and concave relaxations for the solutions of parametric ODEs

SUMMARYConvex and concave relaxations for the parametric solutions of ordinary differential equations (ODEs) are central to deterministic global optimization methods for nonconvex dynamic optimization and open‐loop optimal control problems with control parametrization. Given a general system of ODEs with parameter dependence in the initial conditions and right‐hand sides, this work derives sufficient conditions under which an auxiliary system of ODEs describes convex and concave relaxations of the parametric solutions, pointwise in the independent variable. Convergence results for these relaxations are also established. A fully automatable procedure for constructing an appropriate auxiliary system has been developed previously by the authors. Thus, the developments here lead to an efficient, automatic method for computing convex and concave relaxations for the parametric solutions of a very general class of nonlinear ODEs. The proposed method is presented in detail for a simple example problem. Copyright © 2012 John Wiley & Sons, Ltd.

Penalized Nonlinear Least Squares Estimation of Time-Varying Parameters in Ordinary Differential Equations

Ordinary differential equations (ODEs) are widely used in biomedical research and other scientific areas to model complex dynamic systems. It is an important statistical problem to estimate parameters in ODEs from noisy observations. In this article we propose a method for estimating the time-varying coefficients in an ODE. Our method is a variation of the nonlinear least squares where penalized splines are used to model the functional parameters and the ODE solutions are approximated also using splines. We resort to the implicit function theorem to deal with the nonlinear least squares objective function that is only defined implicitly. The proposed penalized nonlinear least squares method is applied to estimate a HIV dynamic model from a real dataset. Monte Carlo simulations show that the new method can provide much more accurate estimates of functional parameters than the existing two-step local polynomial method which relies on estimation of the derivatives of the state function. Supplemental materials for the article are available online.

Peer Two-Step Methods with Embedded Sensitivity Approximation for Parameter-Dependent ODEs

Peer two-step methods have been successfully applied to initial value problems for stiff and nonstiff ordinary differential equations (ODEs) both on parallel and sequential computers. Their essential property is the use of several stages per time step with the same accuracy. As a new application area these methods are now used for parameter-dependent ODEs where the peer stages approximate the solution also at different places in the parameter space. The main interest here is sensitivity data through an approximation of solution derivatives in different parameter directions. Basic stability and convergence properties are discussed and peer methods of order 2 and 3 in the time stepsize are constructed. The computed sensitivity matrix is used in approximate Newton and Gauss--Newton methods for shooting in boundary value problems, where initial values and/or ODE parameters are searched for, and in parameter identification from partial information on trajectories.

Parameter preserving model order reduction for MEMS applications

Model order reduction techniques are known to work reliably for finite element-type simulations of micro-electro-mechanical systems devices. These techniques can tremendously shorten computational times for transient and harmonic analyses. However, standard model reduction techniques cannot be applied if the equation system incorporates time-varying matrices or parameters that are to be preserved for the reduced model. However, design cycles often involve parameter modification, which should remain possible also in the reduced model. In this article we demonstrate a novel parameterization method to numerically construct highly accurate parametric ordinary differential equation systems based on a small number of systems with different parameter settings. This method is demonstrated to parameterize the geometry of a model of a micro-gyroscope, where the relative error introduced by the parameterization lies in the region of . We also present recent developments on semi-automatic order reduction methods that can preserve scalar parameters or functions during the reduction process. The first approach is based on a multivariate Padé-type expansion. The second approach is a coupling of the balanced truncation method for model order reduction of (deterministic) linear, time-invariant systems with interpolation. The approach is quite flexible in allowing the use of numerous interpolation techniques like polynomial, Hermite, rational, sinc and spline interpolation. As technical examples we investigate a micro anemometer as well as the gyroscope. Speed-up factors of 20–80 could be achieved, while retaining up to six parameters and keeping typical relative errors below 1%.

Robust Estimation for Ordinary Differential Equation Models

Applied scientists often like to use ordinary differential equations (ODEs) to model complex dynamic processes that arise in biology, engineering, medicine, and many other areas. It is interesting but challenging to estimate ODE parameters from noisy data, especially when the data have some outliers. We propose a robust method to address this problem. The dynamic process is represented with a nonparametric function, which is a linear combination of basis functions. The nonparametric function is estimated by a robust penalized smoothing method. The penalty term is defined with the parametric ODE model, which controls the roughness of the nonparametric function and maintains the fidelity of the nonparametric function to the ODE model. The basis coefficients and ODE parameters are estimated in two nested levels of optimization. The coefficient estimates are treated as an implicit function of ODE parameters, which enables one to derive the analytic gradients for optimization using the implicit function theorem. Simulation studies show that the robust method gives satisfactory estimates for the ODE parameters from noisy data with outliers. The robust method is demonstrated by estimating a predator-prey ODE model from real ecological data.