Automatic Differentiation Technology Research Articles

Rethinking data-driven point spread function modeling with a differentiable optical model

In astronomy, upcoming space telescopes with wide-field optical instruments have a spatially varying point spread function (PSF). Specific scientific goals require a high-fidelity estimation of the PSF at target positions where no direct measurement of the PSF is provided. Even though observations of the PSF are available at some positions of the field of view (FOV), they are undersampled, noisy, and integrated into wavelength in the instrument’s passband. PSF modeling represents a challenging ill-posed problem, as it requires building a model from these observations that can infer a super-resolved PSF at any wavelength and position in the FOV. Current data-driven PSF models can tackle spatial variations and super-resolution. However, they are not capable of capturing PSF chromatic variations. Our model, coined WaveDiff, proposes a paradigm shift in the data-driven modeling of the point spread function field of telescopes. We change the data-driven modeling space from the pixels to the wavefront by adding a differentiable optical forward model into the modeling framework. This change allows the transfer of a great deal of complexity from the instrumental response into the forward model. The proposed model relies on efficient automatic differentiation technology and modern stochastic first-order optimization techniques recently developed by the thriving machine-learning community. Our framework paves the way to building powerful, physically motivated models that do not require special calibration data. This paper demonstrates the WaveDiff model in a simplified setting of a space telescope. The proposed framework represents a performance breakthrough with respect to the existing state-of-the-art data-driven approach. The pixel reconstruction errors decrease six-fold at observation resolution and 44-fold for a 3x super-resolution. The ellipticity errors are reduced at least 20 times, and the size error is reduced more than 250 times. By only using noisy broad-band in-focus observations, we successfully capture the PSF chromatic variations due to diffraction. WaveDiff source code and examples associated with this paper are available at this link .

Multi-output physics-informed neural network for one- and two-dimensional nonlinear time distributed-order models

<abstract><p>In this article, a physics-informed neural network based on the time difference method is developed to solve one-dimensional (1D) and two-dimensional (2D) nonlinear time distributed-order models. The FBN-$ \theta $, which is constructed by combining the fractional second order backward difference formula (BDF2) with the fractional Newton-Gregory formula, where a second-order composite numerical integral formula is used to approximate the distributed-order derivative, and the time direction at time $ t_{n+\frac{1}{2}} $ is approximated by making use of the Crank-Nicolson scheme. Selecting the hyperbolic tangent function as the activation function, we construct a multi-output neural network to obtain the numerical solution, which is constrained by the time discrete formula and boundary conditions. Automatic differentiation technology is developed to calculate the spatial partial derivatives. Numerical results are provided to confirm the effectiveness and feasibility of the proposed method and illustrate that compared with the single output neural network, using the multi-output neural network can effectively improve the accuracy of the predicted solution and save a lot of computing time.</p></abstract>

Automatic Differentiation of C++ Codes on Emerging Manycore Architectures with Sacado

Automatic differentiation (AD) is a well-known technique for evaluating analytic derivatives of calculations implemented on a computer, with numerous software tools available for incorporating AD technology into complex applications. However, a growing challenge for AD is the efficient differentiation of parallel computations implemented on emerging manycore computing architectures such as multicore CPUs, GPUs, and accelerators as these devices become more pervasive. In this work, we explore forward mode, operator overloading-based differentiation of C++ codes on these architectures using the widely available Sacado AD software package. In particular, we leverage Kokkos, a C++ tool providing APIs for implementing parallel computations that is portable to a wide variety of emerging architectures. We describe the challenges that arise when differentiating code for these architectures using Kokkos, and two approaches for overcoming them that ensure optimal memory access patterns as well as expose additional dimensions of fine-grained parallelism in the derivative calculation. We describe the results of several computational experiments that demonstrate the performance of the approach on a few contemporary CPU and GPU architectures. We then conclude with applications of these techniques to the simulation of discretized systems of partial differential equations.

Multi‐Net strategy: Accelerating physics‐informed neural networks for solving partial differential equations

AbstractPartial differential equations (PDEs) are the most ubiquitous tools for modeling natural science problems and have long received attention. Physics‐informed neural networks (PINNs) are emerging approaches to approximately solve PDEs. PINNs use automatic differentiation technology to construct the residual of PDEs in the loss function to encode physics conservation laws. We call this process the Single‐Net strategy. Due to the dependency of automatic differentiation among different orders of derivatives, the efficiency of PINNs under the Single‐Net strategy is limited. To address this issue, we propose the Multi‐Net strategy to decouple the dependency. Compared with the Single‐Net strategy, the Multi‐Net strategy reduces the training time of PINNs, and meanwhile, keeps the prediction accuracy. The effectiveness of the proposed strategy is demonstrated through time complexity analysis and a collection of experiments on Burgers equation, advection‐dispersion equation, Kdv equation, and Allen–Cahn equation.

TONR: An exploration for a novel way combining neural network with topology optimization

The rapid development of deep learning has opened a new door to the exploration of topology optimization methods. The combination of deep learning and topology optimization has become one of the hottest research fields at the moment. Different from most existing work, this paper conducts an in-depth study on the method of directly using neural networks (NN) to carry out topology optimization. Inspired by the idea from the field of “Inverting Representation of Image” and “Physics-Informed Neural Network”, a topology optimization via neural reparameterization framework (TONR) that can solve various topology optimization problems is formed. The core idea of TONR is Reparameterization, which means the update of the design variables (pseudo-density) in the conventional topology optimization method is transformed into the update of the NN’s parameters. The sensitivity analysis in the conventional topology optimization method is realized by automatic differentiation technology. With the update of NN’s parameters, the density field is optimized. Some strategies for dealing with design constraints, determining NN’s initial parameters, and accelerating training are proposed in the paper. In addition, the solution of the multi-constrained topology optimization problem is also embedded in the TONR framework. Numerical examples show that TONR can stably obtain optimized structures for different optimization problems, including the stress-constrained problem, structural natural frequency optimization problems, compliant mechanism design problems, heat conduction system design problems, and the optimization problem of hyperelastic structures. Compared with the existing methods that combine deep learning with topology optimization, TONR does not need to construct a dataset in advance and does not suffer from structural disconnection. The structures obtained by TONR can be comparable to the conventional methods.

Efficient (Partial) Determination of Derivative Matrices via Automatic Differentiation

In many scientific computing applications involving nonlinear systems or methods of optimization, a sequence of Jacobian or Hessian matrices is required. Automatic differentiation (AD) technology can be used to accurately determine these matrices, and it is well known that if these matrices exhibit a sparsity pattern (for all iterates), then not only can AD take advantage of this sparsity for significant efficiency gains, AD can also determine the sparsity pattern itself, with some additional work in the first iteration. Practical nonlinear systems and optimization problems often exhibit patterns beyond just “zero-nonzero.” For example, some elements may be duplicates of other elements at all iterates; some elements may be constant (not necessarily zero) for all iterates. Here we show how the popular graph-coloring approach to AD can be adapted to account for these cases as well, with resulting gains in efficiency. In addition, we address the problem of determining, by AD technology, a prescribed set of the entries of the Jacobian (or Hessian, in the optimization context) matrix.

Improved Power Flow Algorithm for VSC-HVDC System Based on High-Order Newton-Type Method

Voltage source converter (VSC) based high-voltage direct-current (HVDC) system is a new transmission technique, which has the most promising applications in the fields of power systems and power electronics. Considering the importance of power flow analysis of the VSC-HVDC system for its utilization and exploitation, the improved power flow algorithms for VSC-HVDC system based on third-order and sixth-order Newton-type method are presented. The steady power model of VSC-HVDC system is introduced firstly. Then the derivation solving formats of multivariable matrix for third-order and sixth-order Newton-type power flow method of VSC-HVDC system are given. The formats have the feature of third-order and sixth-order convergence based on Newton method. Further, based on the automatic differentiation technology and third-order Newton method, a new improved algorithm is given, which will help in improving the program development, computation efficiency, maintainability, and flexibility of the power flow. Simulations of AC/DC power systems in two-terminal, multi-terminal, and multi-infeed DC with VSC-HVDC are carried out for the modified IEEE bus systems, which show the effectiveness and practicality of the presented algorithms for VSC-HVDC system.

A secant method for nonlinear least-squares minimization

Quasi-Newton methods have played a prominent role, over many years, in the design of effective practical methods for the numerical solution of nonlinear minimization problems and in multi-dimensional zero-finding. There is a wide literature outlining the properties of these methods and illustrating their performance (e.g., Dennis and Schnabel, Numerical Methods for Unconstrained Optimization and Nonlinear Equations, 1996). In addition, most modern optimization libraries house a quasi-Newton collection of codes and they are widely used. The quasi-Newton contribution to practical nonlinear optimization is unchallenged. In this paper we propose and investigate an efficient quasi-Newton (secant) approach to the nonlinear least-squares problem, made practical due to the selective application of automatic differentiation (AD) technology. We also observe that AD technology can increase the efficiency of the standard quasi-Newton (positive definite secant) approach to the full nonlinear minimization approach to this problem and we compare these two AD-assisted methods. Finally, we compare the AD-assisted approaches to a standard globalized Gauss-Newton method.

Software supporting optimal experimental design: A case study of binary diffusion using EFCOSS

Methods for optimal experimental design aim at minimizing uncertainty in parameter estimation problems. Despite their long tradition in applied mathematics and importance in practical applications, they are currently not widely used in computational science and engineering. To make the techniques of optimal experimental design more accessible to a broader community, we introduce a novel software environment called EFCOSS and demonstrate its ease of use and versatility in two case studies of binary diffusion experiments. Through the use of a component-based software architecture, integration of automatic differentiation technology and facilitated interfacing to optimization algorithms, EFCOSS minimizes the computational overhead for the user who can thus focus on model development and analysis itself. The presented case studies focus on diffusion experiments in liquids since these experiments are typically very demanding. The use of optimal experimental design techniques allows to reduce experimental time and effort significantly.

Automatic differentiation for reduced sequential quadratic programming

Abstract In order to slove the large-scale nonlinear programming (NLP) problems efficiently, an efficient optimization algorithm based on reduced sequential quadratic programming (rSQP) and automatic differentiation (AD) is presented in this paper. With the characteristics of sparseness, relatively low degrees of freedom and equality constraints utilized, the nonlinear programming problem is solved by improved rSQP solver. In the solving process, AD technology is used to obtain accurate gradient information. The numerical results show that the combined algorithm, which is suitable for large-scale process optimization problems, can calculate more efficiently than rSQP itself.

Solving large-scale optimization problems with EFCOSS

Derivatives play a prominent role in many areas of scientific computing. Traditionally, divided differences are employed to approximate derivatives, leading often to results of dubious quality at great computational expense. Automatic differentiation (AD), by contrast, is a powerful technique for accurately evaluating derivatives of functions described in a high-level programming language. AD requires little human effort and produces derivatives without truncation error. Although there is no conceptual difference between small and large codes, applying AD to programs with hundreds of thousands of lines of code is still a challenging task and requires a robust AD tool. We report on recent accomplishments of AD applied to the general-purpose finite element package SEPRAN transforming approximately 400,000 lines of Fortran77, and its integration into a prototype problem solving environment called EFCOSS supporting interoperability of simulation codes with optimization software using AD technology.

ADMIT-1

ADMIT-1 enables the computation of sparse Jacobian and Hessian matrices, using automatic differentiation technology, from a MATLAB environment. Given a function to be differentiated, ADMIT-1 will exploit sparsity if present to yield sparse derivative matrices (in sparse MATLAB form). A generic automatic differentiation tool, subject to some functionality requirements, can be plugged into ADMIT-1; examples include ADOL-C (C/C++ target functions)and ADMAT (MATLAB target funcitons). ADMIT-1 also allows for the calculation of gradients and has several other related functions. This article provides an introduction to the design and usage of ADMIT-1.

AUTOMATIC DIFFERENTIATION FOR MORE EFFICIENT SYSTEM ANALYSIS AND OPTIMIZATION

This paper reports on the benefits of using automatic differentiation (AD) to improve the efficiency of multidisciplinary analysis and optimization processes. Automatic differentiation technology provides researchers with new opportunities to use sensitivities for analysis and optimization. This research confirms that analysis and optimization applications in which the use of sensitivity information had previously been avoided or ruled out, as computationally prohibitive, need to be revisited. In this research, two applications are investigated in which the use of AD is observed to improve efficiency. The efficiency of solution strategies for non-hierarchic (i.e., coupled) system analysis is improved using automatic differentiation generated sensitivities. Calculation of sensitivities via AD allows for efficient application of Newton's method and a modified form of Newton's method to converge non-hierarchic system analyses. The line search strategy employed within a generalized reduced gradient (GRG) optimizer is improved through the use of AD generated sensitivities. Using the AD-based line search strategy within the GRG method results in a significant reduction in the number of system analyses and the CPU time required for optimization