Automatic Differentiation Tool Research Articles

Differentiable Cosmological Simulation with the Adjoint Method

Rapid advances in deep learning have brought not only a myriad of powerful neural networks, but also breakthroughs that benefit established scientific research. In particular, automatic differentiation (AD) tools and computational accelerators like GPUs have facilitated forward modeling of the Universe with differentiable simulations. Based on analytic or automatic backpropagation, current differentiable cosmological simulations are limited by memory, and thus are subject to a trade-off between time and space/mass resolution, usually sacrificing both. We present a new approach free of such constraints, using the adjoint method and reverse time integration. It enables larger and more accurate forward modeling at the field level, and will improve gradient-based optimization and inference. We implement it in an open-source particle-mesh (PM) N-body library pmwd (PM with derivatives). Based on the powerful AD system JAX, pmwd is fully differentiable, and is highly performant on GPUs.

Making Likelihood Calculations Fast: Automatic Differentiation Applied to RooFit

With the growing datasets of current and next-generation HighEnergy and Nuclear Physics (HEP/NP) experiments, statistical analysis has become more computationally demanding. These increasing demands elicit improvements and modernizations in existing statistical analysis software. One way to address these issues is to improve parameter estimation performance and numeric stability using Automatic Differentiation (AD). AD’s computational efficiency and accuracy are superior to the preexisting numerical differentiation techniques, and it offers significant performance gains when calculating the derivatives of functions with a large number of inputs, making it particularly appealing for statistical models with many parameters. For such models, many HEP/NP experiments use RooFit, a toolkit for statistical modeling and fitting that is part of ROOT. In this paper, we report on the effort to support the AD of RooFit likelihood functions. Our approach is to extend RooFit with a tool that generates overheadfree C++ code for a full likelihood function built from RooFit functional models. Gradients are then generated using Clad, a compiler-based source-codetransformation AD tool, using this C++ code. We present our results from applying AD to the entire minimization pipeline and profile likelihood calculations of several RooFit and HistFactory models at the LHC-experiment scale. We show significant reductions in calculation time and memory usage for the minimization of such likelihood functions. We also elaborate on this approach’s current limitations and explain our plans for the future.

Solving inverse problems in physics by optimizing a discrete loss: Fast and accurate learning without neural networks.

In recent years, advances in computing hardware and computational methods have prompted a wealth of activities for solving inverse problems in physics. These problems are often described by systems of partial differential equations (PDEs). The advent of machine learning has reinvigorated the interest in solving inverse problems using neural networks (NNs). In these efforts, the solution of the PDEs is expressed as NNs trained through the minimization of a loss function involving the PDE. Here, we show how to accelerate this approach by five orders of magnitude by deploying, instead of NNs, conventional PDE approximations. The framework of optimizing a discrete loss (ODIL) minimizes a cost function for discrete approximations of the PDEs using gradient-based and Newton's methods. The framework relies on grid-based discretizations of PDEs and inherits their accuracy, convergence, and conservation properties. The implementation of the method is facilitated by adopting machine-learning tools for automatic differentiation. We also propose a multigrid technique to accelerate the convergence of gradient-based optimizers. We present applications to PDE-constrained optimization, optical flow, system identification, and data assimilation. We compare ODIL with the popular method of physics-informed neural networks and show that it outperforms it by several orders of magnitude in computational speed while having better accuracy and convergence rates. We evaluate ODIL on inverse problems involving linear and nonlinear PDEs including the Navier-Stokes equations for flow reconstruction problems. ODIL bridges numerical methods and machine learning and presents a powerful tool for solving challenging, inverse problems across scientific domains.

A computational toolbox for the assembly yield of complex and heterogeneous structures

The self-assembly of complex structures from a set of non-identical building blocks is a hallmark of soft matter and biological systems, including protein complexes, colloidal clusters, and DNA-based assemblies. Predicting the dependence of the equilibrium assembly yield on the concentrations and interaction energies of building blocks is highly challenging, owing to the difficulty of computing the entropic contributions to the free energy of the many structures that compete with the ground state configuration. While these calculations yield well known results for spherically symmetric building blocks, they do not hold when the building blocks have internal rotational degrees of freedom. Here we present an approach for solving this problem that works with arbitrary building blocks, including proteins with known structure and complex colloidal building blocks. Our algorithm combines classical statistical mechanics with recently developed computational tools for automatic differentiation. Automatic differentiation allows efficient evaluation of equilibrium averages over configurations that would otherwise be intractable. We demonstrate the validity of our framework by comparison to molecular dynamics simulations of simple examples, and apply it to calculate the yield curves for known protein complexes and for the assembly of colloidal shells.

Evaluating magnetic fields using deep learning

PurposeThe purpose of this paper is to develop surrogate models, using deep learning (DL), that can facilitate the application of EM analysis software. In the current status quo, electrical systems can be found in an ever-increasing range of products that are part of everyone’s daily live. With the advances in technology, industries such as the automotive, communications and medical devices have been disrupted with new electrical and electronic systems. The innovation and development of such systems with increasing complexity over time has been supported by the increased use of electromagnetic (EM) analysis software. Such software enables engineers to virtually design, analyze and optimize EM systems without the need for building physical prototypes, thus helping to shorten the development cycles and consequently cut costs.Design/methodology/approachThe industry standard for simulating EM problems is using either the finite difference method or the finite element method (FEM). Optimization of the design process using such methods requires significant computational resources and time. With the emergence of artificial intelligence, along with specialized tools for automatic differentiation, the use of DL has become computationally much more efficient and cheaper. These advances in machine learning have ushered in a new era in EM simulations where engineers can compute results much faster while maintaining a certain level of accuracy.FindingsThis paper proposed two different models that can compute the magnetic field distribution in EM systems. The first model is based on a recurrent neural network, which is trained through a data-driven supervised learning method. The second model is an extension to the first with the incorporation of additional physics-based information to the authors’ model. Such a DL model, which is constrained by the laws of physics, is known as a physics-informed neural network. The solutions when compared with the ground truth, computed using FEM, show promising accuracy for the authors’ DL models while reducing the computation time and resources required, as compared to previous implementations in the literature.Originality/valueThe paper proposes a neural network architecture and is trained with two different learning methodologies, namely, supervised and physics-based. The working of the network along with the different learning methodologies is validated over several EM problems with varying levels of complexity. Furthermore, a comparative study is performed regarding performance accuracy and computational cost to establish the efficacy of different architectures and learning methodologies.

On the Development of an Implicit Discontinuous Galerkin Solver for Turbulent Real Gas Flows

The aim of this work is to describe an efficient implementation of cubic and multiparameter real gas models in an existing discontinuous Galerkin solver to extend its capabilities to the simulation of turbulent real gas flows. The adopted thermodynamic models are van der Waals, Peng–Robinson, and Span–Wagner, which differ from each other in terms of accuracy and computational cost. Convective numerical fluxes across elements interfaces are calculated with a thermodynamic consistent linearized Riemann solver, whereas for boundary conditions, a linearized expression of the generalized Riemann invariants is employed. Transport properties are treated as temperature- and density-dependent quantities through multiparameter correlations. An implicit time integration is adopted; Jacobian matrix and thermodynamic derivatives are obtained with the automatic differentiation tool Tapenade. The solver accuracy is assessed by computing both steady and unsteady real gas test cases available in the literature, and the effect of the mesh size and polynomial degree of approximation on the solution accuracy is investigated. A good agreement with experimental and numerical reference data is observed and specific non-classical phenomena are well reproduced by the solver.

Event-Based Automatic Differentiation of OpenMP with OpDiLib

We present the new software OpDiLib, a universal add-on for classical operator overloading AD tools that enables the automatic differentiation (AD) of OpenMP parallelized code. With it, we establish support for OpenMP features in a reverse mode operator overloading AD tool to an extent that was previously only reported on in source transformation tools. We achieve this with an event-based implementation ansatz that is unprecedented in AD. Combined with modern OpenMP features around OMPT, we demonstrate how it can be used to achieve differentiation without any additional modifications of the source code; neither do we impose a priori restrictions on the data access patterns, which makes OpDiLib highly applicable. For further performance optimizations, restrictions like atomic updates on adjoint variables can be lifted in a fine-grained manner. OpDiLib can also be applied in a semi-automatic fashion via a macro interface, which supports compilers that do not implement OMPT. We demonstrate the applicability of OpDiLib for a pure operator overloading approach in a hybrid parallel environment. We quantify the cost of atomic updates on adjoint variables and showcase the speedup and scaling that can be achieved with the different configurations of OpDiLib in both the forward and the reverse pass.

GPU Accelerated Automatic Differentiation With Clad

Automatic Differentiation (AD) is instrumental for science and industry. It is a tool to evaluate the derivative of a function specified through a computer program. The range of AD application domain spans from Machine Learning to Robotics to High Energy Physics. Computing gradients with the help of AD is guaranteed to be more precise than the numerical alternative and have a low, constant factor more arithmetical operations compared to the original function. Moreover, AD applications to domain problems typically are computationally bound. They are often limited by the computational requirements of high-dimensional parameters and thus can benefit from parallel implementations on graphics processing units (GPUs).Clad aims to enable differential analysis for C/C++ and CUDA and is a compiler-assisted AD tool available both as a compiler extension and in ROOT. Moreover, Clad works as a plugin extending the Clang compiler; as a plugin extending the interactive interpreter Cling; and as a Jupyter kernel extension based on xeus-cling.We demonstrate the advantages of parallel gradient computations on GPUs with Clad. We explain how to bring forth a new layer of optimization and a proportional speed up by extending Clad to support CUDA. The gradients of well-behaved C++ functions can be automatically executed on a GPU. The library can be easily integrated into existing frameworks or used interactively. Furthermore, we demonstrate the achieved application performance improvements, including (≈10x) in ROOT histogram fitting and corresponding performance gains from offloading to GPUs.

Multi-Row Turbomachinery Aerodynamic Design Optimization by an Efficient and Accurate Discrete Adjoint Solver

This paper proposes an approach that combines manual differentiation (MD) and automatic differentiation (AD) to develop an efficient and accurate multi-row discrete adjoint solver. In this approach, the structures of adjoint codes generated using an AD tool are first analyzed. Then, the AD-generated codes are manually adjusted to reduce memory and CPU time consumption. This manual adjustment is performed by replacing the automatically generated low-efficient differentiated codes with manually developed ones. To demonstrate the effectiveness of the proposed approach, the single-stage transonic compressor–NASA Stage 35 and the 1.5-stage Aachen turbine–are used. The solution information exchange at a rotor-stator/stator-rotor interface is achieved by a conservative, non-reflective, and robust discrete adjoint mixing plane method. The results show that the discrete adjoint solver developed by hybrid automatic and manual differentiation is more economical in computational cost than that developed purely by an AD tool and has higher sensitivity accuracy than the adjoint solver with the constant eddy viscosity (CEV) assumption. Moreover, the multi-row turbomachinery design optimizations can be efficiently performed by the discrete adjoint solver developed by the hybrid automatic and manual differentiation.

Moment Evolution Equations and Moment Matching for Stochastic Image EPDiff

Models of stochastic image deformation allow study of time-continuous stochastic effects transforming images by deforming the image domain. Applications include longitudinal medical image analysis with both population trends and random subject-specific variation. Focusing on a stochastic extension of the LDDMM models with evolutions governed by a stochastic EPDiff equation, we use moment approximations of the corresponding Itô diffusion to construct estimators for statistical inference in the full stochastic model. We show that this approach, when efficiently implemented with automatic differentiation tools, can successfully estimate parameters encoding the spatial correlation of the noise fields on the image.

Auto-eD: A visual learning tool for automatic differentiation

Auto-eD: A visual learning tool for automatic differentiation

Variational data assimilation to improve subsurface drainage model parameters

• Identification of the most influential model parameters using the global adjoint SA. • Variational data assimilation significantly reduces the water volume discrepancy. • The gradient-based stochastic procedure performs without the need for the prior. • A new hybrid BV method offers an automatic tool to select a robust parameter set. Variational data assimilation (VDA) has been implemented to enhance the estimation of the unknown input parameters of a new agricultural subsurface drainage model (SIDRA-RU) through assimilating drainage discharge observations. The adjoint model of SIDRA-RU has been successfully generated through the generic automatic differentiation tool (TAPENADE). First, the adjoint model is used to explore the local and global adjoint sensitivities of the valuable function defined over the drainage discharge simulations with respect to model input parameters. Next, the most influential parameters are estimated by applying the Variational DA approach. The performed sensitivity analysis shows that the most influential parameters on drainage discharge are those controlling the dynamics of the water table; the second most influential parameters manage the drainflow start of each drainage season. Compared to an alternative gradient-free calibration performance, the estimation of these governing parameters by the variational method improves the overall quality of the drainage discharge prediction, in particular in terms of the cumulative water volume. Improved parameters generate less than 5 mm (1%) of the discrepancy between simulated and observed water volumes, based on the five years of daily discharge observations on the Chantemerle agricultural parcels (36 ha). Preliminary numerical tests have shown the potential presence of multiple local minima, thus pointing out the equifinality issues. The latter can be highlighted by the self-compensation of both the physical soil parameters and the main conceptual parameters. For improving the robustness of the parameter estimates, a novel hybrid “Bayesian Variational” method is suggested. This method is based on the Bayesian averaging of an ensemble of optimal estimates.

Backpropagation through signal temporal logic specifications: Infusing logical structure into gradient-based methods

This paper presents a technique, named STLCG, to compute the quantitative semantics of Signal Temporal Logic (STL) formulas using computation graphs. STLCG provides a platform which enables the incorporation of logical specifications into robotics problems that benefit from gradient-based solutions. Specifically, STL is a powerful and expressive formal language that can specify spatial and temporal properties of signals generated by both continuous and hybrid systems. The quantitative semantics of STL provide a robustness metric, that is, how much a signal satisfies or violates an STL specification. In this work, we devise a systematic methodology for translating STL robustness formulas into computation graphs. With this representation, and by leveraging off-the-shelf automatic differentiation tools, we are able to efficiently backpropagate through STL robustness formulas and hence enable a natural and easy-to-use integration of STL specifications with many gradient-based approaches used in robotics. Through a number of examples stemming from various robotics applications, we demonstrate that STLCG is versatile, computationally efficient, and capable of incorporating human-domain knowledge into the problem formulation.

Source-to-Source Automatic Differentiation of OpenMP Parallel Loops

This article presents our work toward correct and efficient automatic differentiation of OpenMP parallel worksharing loops in forward and reverse mode. Automatic differentiation is a method to obtain gradients of numerical programs, which are crucial in optimization, uncertainty quantification, and machine learning. The computational cost to compute gradients is a common bottleneck in practice. For applications that are parallelized for multicore CPUs or GPUs using OpenMP, one also wishes to compute the gradients in parallel. We propose a framework to reason about the correctness of the generated derivative code, from which we justify our OpenMP extension to the differentiation model. We implement this model in the automatic differentiation tool Tapenade and present test cases that are differentiated following our extended differentiation procedure. Performance of the generated derivative programs in forward and reverse mode is better than sequential, although our reverse mode often scales worse than the input programs.

PyTorchDIA: a flexible, GPU-accelerated numerical approach to Difference Image Analysis

ABSTRACT We present a GPU-accelerated numerical approach for fast kernel and differential background solutions. The model image proposed in the Bramich Difference Image Analysis (DIA) algorithm is analogous to a very simple convolutional neural network (CNN), with a single convolutional filter (i.e. the kernel) and an added scalar bias (i.e. the differential background). Here, we do not solve for the discrete pixel array in the classical, analytical linear least-squares sense. Instead, by making use of PyTorch tensors (GPU compatible multidimensional matrices) and associated deep learning tools, we solve for the kernel via an inherently massively parallel optimization. By casting the DIA problem as a GPU-accelerated optimization that utilizes automatic differentiation tools, our algorithm is both flexible to the choice of scalar objective function, and can perform DIA on astronomical data sets at least an order of magnitude faster than its classical analogue. More generally, we demonstrate that tools developed for machine learning can be used to address generic data analysis and modelling problems.

Variational Quantum Generators: Generative Adversarial Quantum Machine Learning for Continuous Distributions

AbstractA hybrid quantum–classical approach to model continuous classical probability distributions using a variational quantum circuit is proposed. The architecture of this quantum generator consists of a quantum circuit that encodes a classical random variable into a quantum state and a parameterized quantum circuit trained to mimic the target distribution. The model allows for easy interfacing with a classical function, such as a neural network, and is trained using an adversarial learning approach. It is shown that the quantum generator is able to learn using either a classical neural network or a variational quantum circuit as the discriminator model. This implementation takes advantage of automatic differentiation tools to perform the optimization of the variational circuits employed. The framework presented here for the design and implementation of the variational quantum generators can serve as a blueprint for designing hybrid quantum–classical models for other machine learning tasks.

Path cuts

In scenes lit with sharp point-like light sources, light can bounce several times on specular materials before getting into our eyes, forming purely specular light paths. However, to our knowledge, rendering such multi-bounce pure specular paths has not been handled in previous work: while many light transport methods have been devised to sample various kinds of light paths, none of them are able to find multi-bounce pure specular light paths from a point light to a pinhole camera. In this paper, we present path cuts to efficiently render such light paths. We use a path space hierarchy combined with interval arithmetic bounds to prune non-contributing regions of path space, and to slice the path space into regions small enough to empirically contain at most one solution. Next, we use an automatic differentiation tool and a Newton-based solver to find an admissible specular path within a given path space region. We demonstrate results on several complex specular configurations, including RR, TT, TRT and TTTT paths.

Landau: A Language for Dynamical Systems with Automatic Differentiation

Most numerical solvers used to determine the free variables of dynamical systems rely on firstorder derivatives of the state of the system with respect to the free variables. The number of free variables can be fairly large. One of the approaches to obtaining these derivatives is the integration of the derivatives simultaneously with the dynamical equations, which is best done with automatic differentiation techniques. Even though there exist many automatic differentiation tools, none has been found to be scalable and usable for practical purposes of modeling dynamical systems. Landau is a Turing incomplete statically typed domain-specific language aimed to fill this gap. The Turing incompleteness allows for a sophisticated source code analysis and, as a result, a highly optimized compiled code. Among other things, the language syntax supports functions, compile-time ranged for loops, if/else branching constructions, real variables and arrays, and allows for manually discarding calculations where the automatic derivative values are expected to be negligibly small. In spite of reasonable restrictions, the language is rich enough to express and differentiate any cumbersome equation with practically no effort. Bibliography: 11 titles.

Adjoint optimization of pressurized membrane structures using automatic differentiation tools

This paper presents an adjoint-based method for solving optimization problems involving pressurized membrane structures subject to external pressure loads. Shape optimization of pressurized membranes is complicated by the fact that, lacking bending stiffness, their three-dimensional shape must be sustained by the internal pressure of the inflation medium. The proposed method treats the membrane structure as an immersed manifold and employs a total Lagrangian kinematic description with an analytical pressure–volume relationship for the inflating medium. To demonstrate the proposed method, this paper considers hydrostatically loaded inflatable barriers and develops an application-specific shape parametrization based on the analytical inhomogeneous solution for the inflated shape of cylindrical membranes. Coupling this shape parametrization approach with the adjoint method for computing the gradients of functionals enables a computationally efficient optimization of pressurized membrane structures. Numerical examples include minimization and minimax problems with inequality and state constraints, which are solved considering both plane strain and general plane stress conditions. The numerical implementation leverages the high-level mathematical syntax and automatic differentiation features of the finite-element library FEniCS and related library dolfin-adjoint. The overall techniques generalize to a broad range of structural optimization problems involving pressurized membrane and thin shell structures.

A multiphase CMAQ version 5.0 adjoint.

We present the development of a multiphase adjoint for the Community Multiscale Air Quality (CMAQ) model, a widely used chemical transport model. The adjoint model provides location- and time-specific gradients that can be used in various applications such as backward sensitivity analysis, source attribution, optimal pollution control, data assimilation, and inverse modeling. The science processes of the CMAQ model include gas-phase chemistry, aerosol dynamics and thermodynamics, cloud chemistry and dynamics, diffusion, and advection. Discrete adjoints are implemented for all the science processes, with an additional continuous adjoint for advection. The development of discrete adjoints is assisted with algorithmic differentiation (AD) tools. Particularly, the Kinetic PreProcessor (KPP) is implemented for gas-phase and aqueous chemistry, and two different automatic differentiation tools are used for other processes such as clouds, aerosols, diffusion, and advection. The continuous adjoint of advection is developed manually. For adjoint validation, the brute-force or finite-difference method (FDM) is implemented process by process with box- or column-model simulations. Due to the inherent limitations of the FDM caused by numerical round-off errors, the complex variable method (CVM) is adopted where necessary. The adjoint model often shows better agreement with the CVM than with the FDM. The adjoints of all science processes compare favorably with the FDM and CVM. In an example application of the full multiphase adjoint model, we provide the first estimates of how emissions of particulate matter (PM2.5) affect public health across the US.