Use Of Automatic Differentiation Research Articles

Differentiable and hardware-accelerated waveforms for gravitational wave data analysis

We propose the use of automatic differentiation through the programming framework for accelerating a variety of analysis tasks throughout gravitational wave (GW) science. Firstly, we demonstrate that complete waveforms which cover the inspiral, merger, and ringdown of binary black holes (i.e., IMRPhenomD) can be written in and demonstrate that the serial evaluation speed of the waveform (and its derivative) is similar to the implementation in . Moreover, allows for graphics processing unit–accelerated waveform calls which can be over an order of magnitude faster than serial evaluation on a CPU. We then focus on three applications where efficient and differentiable waveforms are essential. Firstly, we demonstrate how gradient descent can be used to optimize the ∼200 coefficients that are used to calibrate the waveform model. In particular, we demonstrate that the typical with numerical relativity waveforms can be improved by more than 50%. Secondly, we show that Fisher forecasting calculations can be sped up by ∼3–5× (on a CPU) with no loss in accuracy. This increased speed makes Fisher forecasting for a population of events substantially simpler. Finally, we show that gradient-based samplers like Hamiltonian Monte Carlo lead to significantly reduced autocorrelation values when compared to traditional Monte Carlo methods. Since differentiable waveforms have substantial advantages for a variety of tasks throughout GW science, we propose that waveform developers use to build new waveforms moving forward. Our waveform code, , can be found on GitHub website and will continue to be updated with new waveforms as they are implemented. Published by the American Physical Society 2024

Physics-Informed Graph-Mesh Networks for PDEs: A hybrid approach for complex problems

The recent rise of deep learning has led to numerous applications, including solving partial differential equations using Physics-Informed Neural Networks. This approach has proven highly effective in several academic cases. However, their lack of physical invariances, coupled with other significant weaknesses, such as an inability to handle complex geometries or their lack of generalization capabilities, make them unable to compete with classical numerical solvers in industrial settings. In this work, a limitation regarding the use of automatic differentiation in the context of physics-informed learning is highlighted. A hybrid approach combining physics-informed graph neural networks with numerical kernels from finite elements is introduced. After studying the theoretical properties of our model, we apply it to complex geometries, in two and three dimensions. Our choices are supported by an ablation study, and we evaluate the generalization capacity of the proposed approach.

Robust and efficient implementation of finite strain generalized continuum models for material failure: Analytical, numerical, and automatic differentiation with hyper-dual numbers

Generalized continuum models for representing nonlinear material behavior including material failure in the finite strain regime are commonly formulated based on scalar elastic and dissipation potential functions. The evolution of stresses and internal variables, i.e., the material state, is governed by partial derivatives of the potential functions with respect to deformation and stress measures. Furthermore, for application of such models in implicit nonlinear finite element analysis tangent operators, consistent with the numerical integration algorithm, are required. In this work, we study analytical, numerical and automatic differentiation schemes for the implementation of coupled, generalized continuum models considering finite inelastic deformations and fracture. This includes the application of finite difference approximations, complex-step derivative approximations and automatic differentiation based on hyper-dual numbers. For the use of automatic differentiation, a semi-analytical split approach is introduced for increasing the computational efficiency. Based on a comprehensive 2D and 3D finite element study, we demonstrate the superior properties of automatic differentiation compared to numerical differentiation methods with regards to accuracy and robustness. Furthermore, the additional computational cost for automatic differentiation using the proposed split approach becomes negligible for large problems.

Deep Neural Network-Based System Identification for Nonlinear MPC: Enhancements for Massive Multi-Output Systems and Experimental Validation with 1D Camera Image Outputs*

In this research, we investigate a system identification method based on deep neural networks for nonlinear Model Predictive Control (MPC), focusing on efficiently managing massive multi-output systems. This method involves the direct synthesis of state estimators and output predictors represented by neural networks from experimental data. The integration of these components with the Levenberg-Marquardt optimization method, coupled with the use of automatic differentiation, enables efficient realization of nonlinear MPC.In this research, we propose a specific architecture for the state estimator and output predictor, designed to suit multi-output systems. This approach is applied to a miniature four-wheeled vehicle equipped with a 1D camera, which generates 160-pixel image outputs. The experimental application to this test vehicle demonstrates the method's capability in effectively managing complex, multi-output systems.

Uncertainty propagation in orbital dynamics via Galerkin projection of the Fokker-Planck Equation

The Fokker–Planck equation is a partial differential equation that describes how the probability density function of an object state varies, when subject to deterministic and random forces. The solution to this equation is crucial in many space applications, such as space debris trajectory tracking and prediction, guidance navigation and control under uncertainties, space situational awareness, and mission analysis and planning. However, no general closed-form solutions are known and several methods exist to tackle its solution. In this work, we use a known technique to transform this equation into a set of linear ordinary differential equations in the context of orbital dynamics. In particular, we show the advantages of the applied methodology, which allows to decouple the time and state-dependent components and to retain the entire shape of the probability density function through time, in the presence of both deterministic and stochastic dynamics. With this approach, the probability density function values at future times and for different initial conditions can be computed without added costs, provided that some time-independent integrals are solved offline. We showcase the efficacy and use of this method on some orbital dynamics example, by also leveraging the use of automatic differentiation for efficiently computing the involved derivatives.

Exploring the free-energy landscape of a rotating superfluid.

The equilibrium state of a superfluid in a rotating cylindrical vessel is a vortex crystal-an array of vortex lines, which is stationary in the rotating frame. Experimental realizations of this behavior typically show a sequence of transient states before the free-energy-minimizing configuration is reached. Motivated by these observations, we construct a new method for a systematic exploration of the free-energy landscape via gradient-based optimization of a scalar loss function. Our approach is inspired by the pioneering numerical work of Campbell and Ziff [Phys. Rev. B. 20, 1886 (1979)] and makes use of automatic differentiation, which crucially allows us to include entire solution trajectories in the loss. We first use the method to converge thousands of low free-energy relative equilibria in the unbounded domain for vortex numbers in the range 10≤N≤30, which reveals an extremely dense set of mostly saddle-like solutions. As part of this search, we discover new continuous families of relative equilibria, which are often global minimizers of free energy. These continuous families all consist of crystals arranged in a double-ring configuration, and we assess which state from the family is most likely to be observed experimentally by computing energy-minimizing pathways from nearby local minima-identifying a common entry point into the family. The continuous families become discrete sets of equal-energy solutions when the wall is introduced in the problem. Finally, we develop an approach to compute homoclinic orbits and use it to examine the dynamics in the vicinity of the minimizing state by converging connections for low-energy saddles.

Flow reconstruction by multiresolution optimization of a discrete loss with automatic differentiation.

We present a potent computational method for the solution of inverse problems in fluid mechanics. We consider inverse problems formulated in terms of a deterministic loss function that can accommodate data and regularization terms. We introduce a multigrid decomposition technique that accelerates the convergence of gradient-based methods for optimization problems with parameters on a grid. We incorporate this multigrid technique to the Optimizing a DIscrete Loss (ODIL) framework. The multiresolution ODIL (mODIL) accelerates by an order of magnitude the original formalism and improves the avoidance of local minima. Moreover, mODIL accommodates the use of automatic differentiation for calculating the gradients of the loss function, thus facilitating the implementation of the framework. We demonstrate the capabilities of mODIL on a variety of inverse and flow reconstruction problems: solution reconstruction for the Burgers equation, inferring conductivity from temperature measurements, and inferring the body shape from wake velocity measurements in three dimensions. We also provide a comparative study with the related, popular Physics-Informed Neural Networks (PINNs) method. We demonstrate that mODIL has three to five orders of magnitude lower computational cost than PINNs in benchmark problems including simple PDEs and lid-driven cavity problems. Our results suggest that mODIL is a very potent, fast and consistent method for solving inverse problems in fluid mechanics.

QuOCS: The quantum optimal control suite

Quantum optimal control includes a family of pulse-shaping algorithms that aim to unlock the full potential of a variety of quantum technologies. The Quantum Optimal Control Suite (QuOCS) unites experimental focus and model-based approaches in a unified framework. Easy usage and installation presented here and the availability of various combinable optimization strategies is designed to improve the performance of many quantum technology platforms, such as color defects in diamond, superconducting qubits, atom- or ion-based quantum computers. It can also be applied to the study of more general phenomena in physics. In this paper, we describe the software and the toolbox of gradient-free and gradient-based algorithms. We then show how the user can connect it to their experiment. In addition, we provide illustrative examples where our optimization suite solves typical quantum optimal control problems, in both open- and closed-loop settings. Integration into existing experimental control software is already provided for the experiment control software Qudi (Binder et al., 2017 [41]), and further extensions are investigated and highly encouraged.QuOCS is available from GitHub, under Apache License 2.0, and can be found on the PyPI repository. Program summaryProgram Title: QuOCS - Quantum Optimal Control SuiteCPC Library link to program files:https://doi.org/10.17632/wjjch757fk.1Developer's repository link:https://github.com/Quantum-OCS/QuOCSLicensing provisions: Apache-2.0Programming language: PythonExternal routines: NumPy [1], SciPy [1], JAX [2]Nature of problem: Quantum systems are typically controlled by time-dependent electromagnetic fields to perform a certain set of quantum operations. Those operations may in turn provide building blocks for various quantum information processing tasks such as quantum computation, communication, simulation, sensing, and metrology. Numerous control strategies exist to design and improve such operations [3]. While some strategies are constructed to target a rather specific problem with high efficiency, others are more general to solve a wide range of applications [4,5]. To access the different algorithms, one has to download different optimization suites with different input and output parameters, making them hard to compare and combine. To benefit from the variety of algorithms, we have devised a customizable and intuitive optimization suite that simultaneously provides access to some of the most popular quantum optimal control algorithms.Solution method: We combine, in a unified framework, some of the frequently used optimal control algorithms which are the dressed Chopped Random Basis method (dCRAB) [6], and Gradient Ascent Pulse Engineering (GRAPE) [7], with an extension to make use of Automatic Differentiation (AD) [8]. The software is able to connect to both models of quantum dynamics in simulations and real-time quantum experiments to perform open- and closed-loop optimization, respectively. With minimal knowledge of optimal control theory, the user can manage to run optimizations of quantum processes using a variety of additional features such as stopping criteria and drift compensation. Logging and data management of the optimization progress and results are also handled by the suite. Its modular structure allows for extensions that accommodate customized algorithms and can be implemented by the user straightforwardly.Additional comments including unusual features: The connection to the experiments is performed by an extension that enables a direct integration to a laboratory control software Qudi [9].

Discrete adjoint for coupled conjugate heat transfer

The typical method to solve multi-physics problems such as Conjugate Heat Transfer (CHT) is the partitioned approach which couples separate solvers through boundary conditions. Effective gradient-based optimisation of partitioned CHT problems requires the adjoint of the coupling to maintain the efficiency of the original multi-physics coupling, which is a significant challenge. The use of automatic differentiation (AD) has the potential to ease this burden and leads to generic gradient computation methods. In this paper, we present how to automate the generation of adjoint fluid and solid solvers for a CHT adjoint using Automatic Differentiation (AD). The derivation of the adjoint of the loose coupling algorithms is shown for three fixed-point coupling algorithms. Application of the coupled adjoint algorithm is shown to two CHT optimisation benchmark cases for inverse design and shape optimisation. The results demonstrate that Robin-based coupling algorithms have faster runtimes and are an attractive option for CHT optimisation problems.

Evaluation of the degree of rate control via automatic differentiation

AbstractThe degree of rate control (DRC) quantitatively identifies the kinetically relevant (sometimes known as rate‐limiting) steps of a complex reaction network. This concept relies on derivatives which are commonly implemented numerically, for example, with finite differences (FDs). Numerical derivatives are tedious to implement, and can be problematic, and unstable or unreliable. In this study, we demonstrate the use of automatic differentiation (AD) in the evaluation of the DRC. AD libraries are increasingly available through modern machine learning frameworks. Compared with the FDs, AD provides solutions with higher accuracy with lower computational cost. We demonstrate applications in steady‐state and transient kinetics. Furthermore, we illustrate a hybrid local‐global sensitivity analysis method, the distributed evaluation of local sensitivity analysis, to assess the importance of kinetic parameters over an uncertain space. This method also benefits from AD to obtain high‐quality results efficiently.

Calibration of elastoplastic constitutive model parameters from full‐field data with automatic differentiation‐based sensitivities

AbstractWe present a framework for calibration of parameters in elastoplastic constitutive models that is based on the use of automatic differentiation (AD). The model calibration problem is posed as a partial differential equation‐constrained optimization problem where a finite element (FE) model of the coupled equilibrium equation and constitutive model evolution equations serves as the constraint. The objective function quantifies the mismatch between the displacement predicted by the FE model and full‐field digital image correlation data, and the optimization problem is solved using gradient‐based optimization algorithms. Forward and adjoint sensitivities are used to compute the gradient at considerably less cost than its calculation from finite difference approximations. Through the use of AD, we need only to write the constraints in terms of AD objects, where all of the derivatives required for the forward and inverse problems are obtained by appropriately seeding and evaluating these quantities. We present three numerical examples that verify the correctness of the gradient, demonstrate the AD approach's parallel computation capabilities via application to a large‐scale FE model, and highlight the formulation's ease of extensibility to other classes of constitutive models.

Categorical semantics of a simple differential programming language

With the increased interest in machine learning, and deep learning in particular, the use of automatic differentiation has become more wide-spread in computation. There have been two recent developments to provide the theoretical support for this types of structure. One approach, due to Abadi and Plotkin, provides a simple differential programming language. Another approach is the notion of a reverse differential category. In the present paper we bring these two approaches together. In particular, we show how an extension of reverse derivative categories models Abadi and Plotkin's language, and describe how this categorical model allows one to consider potential improvements to the operational semantics of the language.

Education

Education

Graphics Processing Unit-Accelerated Semiempirical Born Oppenheimer Molecular Dynamics Using PyTorch.

A new open-source high-performance implementation of Born Oppenheimer molecular dynamics based on semiempirical quantum mechanics models using PyTorch called PYSEQM is presented. PYSEQM was designed to provide researchers in computational chemistry with an open-source, efficient, scalable, and stable quantum-based molecular dynamics engine. In particular, PYSEQM enables computation on modern graphics processing unit hardware and, through the use of automatic differentiation, supplies interfaces for model parameterization with machine learning techniques to perform multiobjective training and prediction. The implemented semiempirical quantum mechanical methods (MNDO, AM1, and PM3) are described. Additional algorithms include a recursive Fermi-operator expansion scheme (SP2) and extended Lagrangian Born Oppenheimer molecular dynamics allowing for rapid simulations. Finally, benchmark testing on the nanostar dendrimer and a series of polyethylene molecules provides a baseline of code efficiency, time cost, and scaling and stability of energy conservation, verifying that PYSEQM provides fast and accurate computations.

Inverse Design of Photonic Crystals through Automatic Differentiation

Gradient-based inverse design in photonics has already achieved remarkable results in designing small-footprint, high-performance optical devices. The adjoint variable method, which allows for the efficient computation of gradients, has played a major role in this success. However, gradient-based optimization has not yet been applied to the mode-expansion methods that are the most common approach to studying periodic optical structures like photonic crystals. This is because, in such simulations, the adjoint variable method cannot be defined as explicitly as in standard finite-difference or finite-element time- or frequency-domain methods. Here, we overcome this through the use of automatic differentiation, which is a generalization of the adjoint variable method to arbitrary computational graphs. We implement the plane-wave expansion and the guided-mode expansion methods using an automatic differentiation library, and show that the gradient of any simulation output can be computed efficiently and in parallel with respect to all input parameters. We then use this implementation to optimize the dispersion of a photonic crystal waveguide, and the quality factor of an ultra-small cavity in a lithium niobate slab. This extends photonic inverse design to a whole new class of simulations, and more broadly highlights the importance that automatic differentiation could play in the future for tracking and optimizing complicated physical models.

Excited state mean-field theory without automatic differentiation.

We present a formulation of excited state mean-field theory in which the derivatives with respect to the wave function parameters needed for wave function optimization (not to be confused with nuclear derivatives) are expressed analytically in terms of a collection of Fock-like matrices. By avoiding the use of automatic differentiation and grouping Fock builds together, we find that the number of times we must access the memory-intensive two-electron integrals can be greatly reduced. Furthermore, the new formulation allows the theory to exploit the existing strategies for efficient Fock matrix construction. We demonstrate this advantage explicitly via the shell-pair screening strategy with which we achieve a cubic overall cost scaling. Using this more efficient implementation, we also examine the theory's ability to predict charge redistribution during charge transfer excitations. Using the coupled cluster as a benchmark, we find that by capturing orbital relaxation effects and avoiding self-interaction errors, excited state mean field theory out-performs other low-cost methods when predicting the charge density changes of charge transfer excitations.

Robust generation of constrained B-spline curves based on automatic differentiation and fairness optimization

This paper details the use of automatic differentiation in form parameter driven curve design by constrained optimization. Computer aided design, computer aided engineering (CAD/CAE), and particularly computer aided ship hull design (CASHD) are typically implemented as interactive processes in which the user obtains desired shapes by manipulation of control vertices. A fair amount of trial and error is needed to achieve the desired properties. In the variational form parameter approach taken here, the system computes vertices so that the resulting curve meets the specifications and is optimized with respect to a fairness criteria. Implementation of curve design as an optimization problem requires extensive derivative calculations. The paper illustrates how the programming burden can be eased through the use of automatic differentiation techniques. A variational curve design framework has been implemented in Python, and applications to CASHD curve design are shown. The new method is robust and allows great flexibility in the selection of constraints. Offsets, tangents, and curvature may be imposed anywhere along the curve. Form parameters may also be used to define straight segments within a curve, require the curve to enclose specified forms, or specify relationships between curve properties.

Motor Estimation using Heterogeneous Sets of Objects in Conformal Geometric Algebra

In this paper we present a novel method for nonlinear rigid body motion estimation from noisy data using heterogeneous sets of objects of the conformal model in geometric algebra. The rigid body motions are represented by motors. We employ state-of-the-art nonlinear optimization tools and compute gradients and Jacobian matrices using forward-mode automatic differentiation based on dual numbers. The use of automatic differentiation enables us to employ a wide range of cost functions in the estimation process. This includes cost functions for motor estimation using points, lines and planes. Moreover, we explain how these cost functions make it possible to use other geometric objects in the conformal model in the motor estimation process, e.g., spheres, circles and tangent vectors. Experimental results show that we are able to successfully estimate rigid body motions from synthetic datasets of heterogeneous sets of conformal objects including a combination of points, lines and planes.

Comprehensive framework for gradient-based optimization in closed-loop reservoir management

An efficient, robust, and flexible adjoint-based computational framework for performing closed-loop reservoir management is developed and applied. The methodology includes gradient-based production optimization and data assimilation (history matching). Flexibility is achieved through the use of automatic differentiation (AD) within the reservoir simulation, production optimization, and history matching modules. The use of AD will also facilitate the application of closed-loop reservoir management to physical models of higher complexity. A fast sequential convex programming (SCP) solver based on the method of moving asymptotes (MMA) is applied for the production optimization component of the closed-loop. This technique is shown to outperform the sequential quadratic programming (SQP) method, which is commonly used for production optimization computations. The history matching component of the workflow integrates both production data and proxy seismic measurements into a unified adjoint-based data assimilation framework. The effect of noisy data, and data of different types, on the accuracy of the history matching component is assessed. The overall closed-loop reservoir management methodology is tested using the well-documented Brugge model. Results demonstrate the efficient performance of the individual closed-loop components and the improvement in net present value that is achieved using these procedures.

Efficient aerodynamic shape optimization by structure exploitation

In this paper, we consider an optimization problem for the complete design chain of an airfoil. Starting with a parameter vector, one has to perform a three step procedure to evaluate the desired objective: Generate a grid around the airfoil, compute the flow around the airfoil, and compute the objective. Applying a gradient-based optimization method, one has to provide derivatives for this complex process. In the present paper, we propose the advanced use of automatic differentiation to compute the required gradient information. We report numerical results together with a mesh independency study and an analysis of the optimization process for an inviscid RAE2822 airfoil under transonic flight conditions.