Automatic Differentiation Package Research Articles

An inverse mapping approach for process systems engineering using automatic differentiation and the implicit function theorem

AbstractThe objective in this work is to propose a novel approach for solving inverse problems from the output space to the input space using automatic differentiation coupled with the implicit function theorem and a path integration scheme. A common way of solving inverse problems in process systems engineering (PSE) and in science, technology, engineering and mathematics (STEM) in general is using nonlinear programming (NLP) tools, which may become computationally expensive when both the underlying process model complexity and dimensionality increase. The proposed approach takes advantage of recent advances in robust automatic differentiation packages to calculate the input space region by integration of governing differential equations of a given process. Such calculations are performed based on an initial starting point from the output space and are capable of maintaining accuracy and reducing computational time when compared to using NLP‐based approaches to obtain the inverse mapping. Two nonlinear case studies, namely a continuous stirred tank reactor (CSTR) and a membrane reactor for conversion of natural gas to value‐added chemicals are addressed using the proposed approach and compared against: (i) extensive (brute‐force) search for forward mapping and (ii) using NLP solvers for obtaining the inverse mapping. The obtained results show that the novel approach is in agreement with the typical approaches, while computational time and complexity are considerably reduced, indicating that a new direction for solving inverse problems is developed in this work.

Pychastic: Precise Brownian dynamics using Taylor-Itō integrators in Python

In the last decade, Python-powered physics simulations ecosystem has been growing steadily, allowing greater interoperability, and becoming an important tool in numerical exploration of physical phenomena, particularly in soft matter systems. Driven by the need for fast and precise numerical integration in colloidal dynamics, here we formulate the problem of Brownian Dynamics (BD) in a mathematically consistent formalism of the Itō calculus, and develop a Python package to assist numerical computations. We show that, thanks to the automatic differentiation packages, the classical truncated Taylor-Itō integrators can be implemented without the burden of computing the derivatives of the coefficient functions beforehand. Furthermore, we show how to circumvent the difficulties of BD simulations such as calculations of the divergence of the mobility tensor in the diffusion equation and discontinuous trajectories encountered when working with dynamics on S^2S2 and SO(3)SO(3). The resulting Python package, Pychastic, is capable of performing BD simulations including hydrodynamic interactions at speeds comparable to dedicated implementations in lower-level programming languages, but with a much simpler end-user interface.

Codebase release 0.2 for Pychastic

In the last decade, Python-powered physics simulations ecosystem has been growing steadily, allowing greater interoperability, and becoming an important tool in numerical exploration of physical phenomena, particularly in soft matter systems. Driven by the need for fast and precise numerical integration in colloidal dynamics, here we formulate the problem of Brownian Dynamics (BD) in a mathematically consistent formalism of the Itō calculus, and develop a Python package to assist numerical computations. We show that, thanks to the automatic differentiation packages, the classical truncated Taylor-Itō integrators can be implemented without the burden of computing the derivatives of the coefficient functions beforehand. Furthermore, we show how to circumvent the difficulties of BD simulations such as calculations of the divergence of the mobility tensor in the diffusion equation and discontinuous trajectories encountered when working with dynamics on S^2S2 and SO(3)SO(3). The resulting Python package, Pychastic, is capable of performing BD simulations including hydrodynamic interactions at speeds comparable to dedicated implementations in lower-level programming languages, but with a much simpler end-user interface.

Nonlinear programming reformulation of dynamic flux balance analysis models

Dynamic flux balance analysis (dFBA) models are very popular in systems biology because of the many possible applications. dFBA consists of dynamic mass balance equations that describe the concentration of external metabolites and an optimization model to compute the internal flux distribution of the cell. Here, we applied a nonlinear programming formulation for solving dFBA models. The ODE system is discretized using the orthogonal collocation technique and the optimization problem is replaced by its first-order optimality conditions. In addition, an adaptive mesh scheme is applied to improve performance by efficiently handling changes in the active set of constraints. This formulation ensures total differentiability of the model, which can be solved by large-scale solvers and take advantage of automatic differentiation packages. The new approach presents results equivalent to state-of-the-art methods and outperforms them in dynamic optimization problems by avoiding calling an external solver to handle the embedded optimization model.

High-order differentiable autoencoder for nonlinear model reduction

This paper provides a new avenue for exploiting deep neural networks to improve physics-based simulation. Specifically, we integrate the classic Lagrangian mechanics with a deep autoencoder to accelerate elastic simulation of deformable solids. Due to the inertia effect, the dynamic equilibrium cannot be established without evaluating the second-order derivatives of the deep autoencoder network. This is beyond the capability of off-the-shelf automatic differentiation packages and algorithms, which mainly focus on the gradient evaluation. Solving the nonlinear force equilibrium is even more challenging if the standard Newton's method is to be used. This is because we need to compute a third-order derivative of the network to obtain the variational Hessian. We attack those difficulties by exploiting complex-step finite difference, coupled with reverse automatic differentiation. This strategy allows us to enjoy the convenience and accuracy of complex-step finite difference and in the meantime, to deploy complex-value perturbations as collectively as possible to save excessive network passes. With a GPU-based implementation, we are able to wield deep autoencoders (e.g., $10+$ layers) with a relatively high-dimension latent space in real-time. Along this pipeline, we also design a sampling network and a weighting network to enable \emph{weight-varying} Cubature integration in order to incorporate nonlinearity in the model reduction. We believe this work will inspire and benefit future research efforts in nonlinearly reduced physical simulation problems.

High-order differentiable autoencoder for nonlinear model reduction

This paper provides a new avenue for exploiting deep neural networks to improve physics-based simulation. Specifically, we integrate the classic Lagrangian mechanics with a deep autoencoder to accelerate elastic simulation of deformable solids. Due to the inertia effect, the dynamic equilibrium cannot be established without evaluating the second-order derivatives of the deep autoencoder network. This is beyond the capability of off-the-shelf automatic differentiation packages and algorithms, which mainly focus on the gradient evaluation. Solving the nonlinear force equilibrium is even more challenging if the standard Newton's method is to be used. This is because we need to compute a third-order derivative of the network to obtain the variational Hessian. We attack those difficulties by exploiting complex-step finite difference, coupled with reverse automatic differentiation. This strategy allows us to enjoy the convenience and accuracy of complex-step finite difference and in the meantime, to deploy complex-value perturbations as collectively as possible to save excessive network passes. With a GPU-based implementation, we are able to wield deep autoencoders (e.g., 10+ layers) with a relatively high-dimension latent space in real-time. Along this pipeline, we also design a sampling network and a weighting network to enable weight-varying Cubature integration in order to incorporate nonlinearity in the model reduction. We believe this work will inspire and benefit future research efforts in nonlinearly reduced physical simulation problems.

Computing the sparsity pattern of Hessians using automatic differentiation

We compare two methods that calculate the sparsity pattern of Hessian matrices using the computational framework of automatic differentiation. The first method is a forward-mode algorithm by Andrea Walther in 2008 which has been implemented as the driver called hess_pat in the automatic differentiation package ADOL-C. The second is edge_push_sp, a new reverse mode algorithm descended from the edge_pushing algorithm for calculating Hessians by Gower and Mello in 2012. We present complexity analysis and perform numerical tests for both algorithms. The results show that the new reverse algorithm is very promising.

Computation of derivatives for structure preserving optimal control using automatic differentiation

AbstractIn this work we combine a recently developed method, Discrete Mechanics and Optimal Control (DMOC), with the well established Automatic Differentiation package ADOL‐C. DMOC is based on the discretization of the variational structure of the mechanical system which leads to structure (symplectic‐momentum) preserving time‐stepping equations. The discretized equations provide equality constraints for the resulting finite dimensional nonlinear optimization problem. For the solution of this problem standard nonlinear optimization techniques like sequential quadratic programming (SQP) are used. To ensure robustness of these techniques the computation of derivatives of the objective function and the constraints is of great importance. The concept of Automatic Differentiation (AD) is used to improve the performance of the SQP method. (© 2010 Wiley‐VCH Verlag GmbH & Co. KGaA, Weinheim)

TaylUR 3, a multivariate arbitrary-order automatic differentiation package for Fortran 95

This new version of TaylUR is based on a completely new core, which now is able to compute the numerical values of all of a complex-valued function's partial derivatives up to an arbitrary order, including mixed partial derivatives.

New version announcement for TaylUR, an arbitrary-order diagonal automatic differentiation package for Fortran 95

We present a new version of TaylUR, a Fortran 95 module to automatically compute the numerical values of a complex-valued function's derivatives with respect to several variables up to an arbitrary order in each variable, but excluding mixed derivatives. The new version fixes a potentially serious bug in the code for exponential-related functions that could corrupt the imaginary parts of derivatives, as well as being compatible with a wider range of compilers. Program summary Title of program: TaylUR Catalogue identifier: ADXR_v2_0 Program summary URL: http://cpc.cs.qub.ac.uk/summaries/ADXR_v2_0 Program obtainable from: CPC Program Library, Queen's University of Belfast, N. Ireland Licensing provisions: none Programming language used: Fortran 95 Computer: Any computer with a conforming Fortran 95 compiler Operating system: Any system with a conforming Fortran 95 compiler No. of lines in distributed program, including test data, etc.: 6548 No. of bytes in distributed program, including test data, etc.: 17 468 Catalogue identifier of previous version: ADXR_v1_0 Journal reference of previous version: Comput. Phys. Comm. 174 (2006) 569–576 Does the new version supersede the previous version?: yes Distribution format:tar.gz Nature of problem: Problems that require potentially high orders of derivatives with respect to some variables or derivatives of complex-valued functions, such as, e.g., expansions of Feynman diagrams in particle masses in perturbative Quantum Field Theory. Solution method: Arithmetic operators and Fortran intrinsics are overloaded to act correctly on objects of a defined type taylor, which encodes a function along with its first few derivatives with respect to the user-defined independent variables. Derivatives of products and composite functions are computed using Leibniz's rule and Fàa di Bruno's formula. Reasons for the new version: The previous version [G.M. von Hippel, TaylUR, an arbitrary-order diagonal automatic differentiation package for Fortran 95, Comput. Phys. Comm. 174 (2006) 569–576] contained a potentially serious bug in the functions overloading the exponential-related intrinsics ( EXP, LOG, SIN, COS, TAN, SINH, COSH, TANH), which could corrupt the imaginary parts of derivatives. It also contained some features which caused it to crash when compiled with certain compilers (notably the NAG and Lahey/Fujitsu compilers). Summary of revisions: The bug in the exponential-related intrinsics has been corrected. A number of additional changes have been made to the code to enable better compatibility with a greater range of compilers, including the NAG and Lahey/Fujitsu compilers. Users of some of these compilers may have to define useintrinsic as a preprocessor symbol when compiling TaylUR. Restrictions: Memory and CPU time constraints may restrict the number of variables and Taylor expansion order that can be achieved. Loss of numerical accuracy due to cancellation may become an issue at very high orders. Unusual features: No mixed higher-order derivatives are computed. The complex conjugation operation assumes all independent variables to be real. Running time: The running time of TaylUR operations depends linearly on the number of variables. Its dependence on the Taylor expansion order varies from linear (for linear operations) through quadratic (for multiplication) to exponential (for elementary function calls).

A software environment for simultaneous dynamic optimization

We describe a software environment for dynamic optimization using the simultaneous approach. Over the past few years, there has been significant development in the formulation of simultaneous dynamic optimization problems, using collocation on finite elements, and in the solution of the resulting large-scale nonlinear programming problem, using powerful barrier NLP solvers. Here we describe the background of the simultaneous approach as well as the components that make up the formulation and modeling environment, which we call DynoPC. DynoPC incorporates state-of-the-art NLP and DAE solvers along with a powerful automatic differentiation package. Moreover, it has a number of advanced features including model reduction and moving finite elements. Finally, we describe how DynoPC can be interfaced to other environments using Equation Set Objects (ESO) and the MINLP CAPE-OPEN object. All of these concepts are illustrated with three dynamic optimization examples including a large-scale industrial case study.

TaylUR, an arbitrary-order diagonal automatic differentiation package for Fortran 95

We present TaylUR, a Fortran 95 module to automatically compute the numerical values of a complex-valued function's derivatives with respect to several variables up to an arbitrary order in each variable, but excluding mixed derivatives. Arithmetic operators and Fortran intrinsics are overloaded to act correctly on objects of a defined type taylor, which encodes a function along with its first few derivatives with respect to the user-defined independent variables. Derivatives of products and composite functions are computed using Leibniz's rule and Faà di Bruno's formula. TaylUR makes heavy use of operator overloading and other Fortran 95 features such as elemental functions. Program summary Program title: TaylUR Catalogue identifier:ADXR_v1_0 Program summary URL: http://cpc.cs.qub.ac.uk/summaries/ADXR_v1_0 Program obtainable from: CPC Program Library, Queen's University of Belfast, N. Ireland Licensing provisions:none Programming language:Fortran 95 Computer:Any computer with a conforming Fortran 95 compiler Operating system:Any system with a conforming Fortran 95 compiler No. of lines in distributed program, including test data, etc.:6286 No. of bytes in distributed program, including test data, etc:14 994 Distribution format:tar.gz Nature of problem:Problems that require potentially high orders of derivatives with respect to some variables, such as e.g. expansions of Feynman diagrams in particle masses in perturbative Quantum Field Theory, and which cannot be treated using existing Fortran modules for automatic differentiation [C.W. Straka, ADF95: Tool for automatic differentiation of a FORTRAN code designed for large numbers of independent variables, Comput. Phys. Comm. 168 (2005) 123–139, arXiv:cs.MS/0503014; S. Stamatiadis, R. Prosmiti, S.C. Farantos, auto_deriv: Tool for automatic differentiation of a FORTRAN code, Comput. Phys. Comm. 127 (2000) 343–355]. Solution method:Arithmetic operators and Fortran intrinsics are overloaded to act correctly on objects of a defined type taylor, which encodes a function along with its first few derivatives with respect to the user-defined independent variables. Derivatives of products and composite functions are computed using Leibniz's rule and Faà di Bruno's formula. Restrictions:Memory and CPU time constraints may restrict the number of variables and Taylor expansion order that can be achieved. Loss of numerical accuracy due to cancellation may become an issue at very high orders. Unusual features:No mixed higher-order derivatives are computed. The complex conjugation operation assumes all independent variables to be real. Running time:The running time of TaylUR operations depends linearly on the number of variables. Its dependence on the Taylor expansion order varies from linear (for linear operations) through quadratic (for multiplication) to exponential (for elementary function calls).

Numerical Normal Forms for Codim 2 Bifurcations of Fixed Points with at Most Two Critical Eigenvalues

In this paper we derive explicit formulas for the normal form coefficients to verify the nondegeneracy of eight codimension two bifurcations of fixed points with one or two critical eigenvalues. These include all strong resonances, as well as the degenerate flip and Neimark--Sacker bifurcations. Applying our results to n-dimensional maps, one avoids the computation of the center manifold, but one can directly evaluate the critical normal form coefficients in the original basis. The formulas remain valid also for n=2 and allow one to avoid the transformation of the linear part of the map into Jordan form. The developed techniques are tested on two examples: (1) a three-dimensional map appearing in adaptive control; (2) a periodically forced epidemic model. We compute numerically the critical normal form coefficients for several codim 2 bifurcations occurring in these models. To compute the required derivatives of the Poincaré map for the epidemic model, the automatic differentiation package ADOL-C is used.

CUTEr and SifDec

The initial release of CUTE, a widely used testing environment for optimization software, was described by Bongartz, et al. [1995]. A new version, now known as CUTEr, is presented. Features include reorganisation of the environment to allow simultaneous multi-platform installation, new tools for, and interfaces to, optimization packages, and a considerably simplified and entirely automated installation procedure for unix systems. The environment is fully backward compatible with its predecessor, and offers support for Fortran 90/95 and a general C/C++ Application Programming Interface. The SIF decoder, formerly a part of CUTE, has become a separate tool, easily callable by various packages. It features simple extensions to the SIF test problem format and the generation of files suited to automatic differentiation packages.

System Inversion for Nonlinear Descriptor Systems

In control engineering, one problem of output tracking is the computation of an input signal for a prescribed output curve. We consider process models given by nonlinear semi-explicit descriptor systems. The authors derive a method to compute function values of the input signal by means of an univariate Taylor expansion of the systems' variables. To avoid computationally intensive symbolic calculations, we resort to automatic differentiation. Several automatic differentiation packages such as ADOL-C or TADIFF can be applied to compute the derivatives needed.

The role of automatic differentiation in power system analysis

In this paper, automatic differentiation of algorithms is utilized in power system analysis. Automatic differentiation is a computational tool to obtain the derivatives and the value of the function systematically without providing explicit expressions for the derivatives. The method is reliable and produces accurate derivatives instantaneously without the need for hand-coding the derivatives. The method is applied to compute the Jacobian and sensitivities in the continuation power flow. An automatic differentiation package (ADIFOR) that exploits sparsity techniques is used for the application.