Forward Mode Of Automatic Differentiation Research Articles

Low-variance Forward Gradients using Direct Feedback Alignment and momentum

Supervised learning in deep neural networks is commonly performed using error backpropagation. However, the sequential propagation of errors during the backward pass limits its scalability and applicability to low-powered neuromorphic hardware. Therefore, there is growing interest in finding local alternatives to backpropagation. Recently proposed methods based on forward-mode automatic differentiation suffer from high variance in large deep neural networks, which affects convergence. In this paper, we propose the Forward Direct Feedback Alignment algorithm that combines Activity-Perturbed Forward Gradients with Direct Feedback Alignment and momentum. We provide both theoretical proofs and empirical evidence that our proposed method achieves lower variance than forward gradient techniques. In this way, our approach enables faster convergence and better performance when compared to other local alternatives to backpropagation and opens a new perspective for the development of online learning algorithms compatible with neuromorphic systems.

Automatic parameter tuning for plug-and-play algorithms using generalized cross validation and Stein's unbiased risk estimation for linear inverse problems in computational imaging

We propose two automatic parameter tuning methods for Plug-and-Play (PnP) algorithms that use CNN denoisers. We focus on linear inverse problems and propose an iterative algorithm to calculate generalized cross-validation (GCV) and Stein’s unbiased risk estimator (SURE) functions for a half-quadratic splitting-based PnP (PnP-HQS) algorithm that uses a state-of- the-art CNN denoiser. The proposed methods leverage forward mode automatic differentiation to calculate the GCV and SURE functions and tune the parameters of a PnP-HQS algorithm automatically by minimizing the GCV and SURE functions using grid search. Because linear inverse problems appear frequently in computational imaging, the proposed methods can be applied in various domains. Furthermore, because the proposed methods rely on GCV and SURE functions, they do not require access to the ground truth image and do not require collecting an additional training dataset, which is highly desirable for imaging applications for which acquiring data is costly and time-consuming. We evaluate the performance of the proposed methods on deblurring and MRI experiments and show that the GCV-based proposed method achieves comparable performance to that of the oracle tuning method that adjusts the parameters by maximizing the structural similarity index between the ground truth image and the output of the PnP algorithm. We also show that the SURE-based proposed method often leads to worse performance compared to the GCV-based proposed method.

You Only Linearize Once: Tangents Transpose to Gradients

Automatic differentiation (AD) is conventionally understood as a family of distinct algorithms, rooted in two “modes”—forward and reverse—which are typically presented (and implemented) separately. Can there be only one? Following up on the AD systems developed in the JAX and Dex projects, we formalize a decomposition of reverse-mode AD into (i) forward-mode AD followed by (ii) unzipping the linear and non-linear parts and then (iii) transposition of the linear part. To that end, we define a (substructurally) linear type system that can prove a class of functions are (algebraically) linear. Our main results are that forward-mode AD produces such linear functions, and that we can unzip and transpose any such linear function, conserving cost, size, and linearity. Composing these three transformations recovers reverse-mode AD. This decomposition also sheds light on checkpointing, which emerges naturally from a free choice in unzipping let expressions. As a corollary, checkpointing techniques are applicable to general-purpose partial evaluation, not just AD. We hope that our formalization will lead to a deeper understanding of automatic differentiation and that it will simplify implementations, by separating the concerns of differentiation proper from the concerns of gaining efficiency (namely, separating the derivative computation from the act of running it backward).

Efficient Dual-Numbers Reverse AD via Well-Known Program Transformations

Where dual-numbers forward-mode automatic differentiation (AD) pairs each scalar value with its tangent value, dual-numbers reverse-mode AD attempts to achieve reverse AD using a similarly simple idea: by pairing each scalar value with a backpropagator function. Its correctness and efficiency on higher-order input languages have been analysed by Brunel, Mazza and Pagani, but this analysis used a custom operational semantics for which it is unclear whether it can be implemented efficiently. We take inspiration from their use of linear factoring to optimise dual-numbers reverse-mode AD to an algorithm that has the correct complexity and enjoys an efficient implementation in a standard functional language with support for mutable arrays, such as Haskell. Aside from the linear factoring ingredient, our optimisation steps consist of well-known ideas from the functional programming community. We demonstrate the use of our technique by providing a practical implementation that differentiates most of Haskell98.

Estimation of Simulated Left Ventricle Elastance Using Lumped Parameter Modelling and Gradient-Based Optimization With Forward-Mode Automatic Differentiation Based on Synthetically Generated Noninvasive Data.

The present study evaluates a parameter discovery approach based on a lumped parameter model of the cardiovascular system in conjunction with optimization to approximate important cardiac parameters, including simulated left ventricle elastances. Important parameters pertaining to ventricular function were estimated using gradient optimization and synthetically generated measurements. Forward-mode automatic differentiation was used to estimate the cost function-parameter matrices and compared to the common finite differences approach. Synthetic data of healthy and diseased hearts were generated as proxies for noninvasive clinical measurements and used to evaluate the algorithm. Twelve parameters including left ventricle elastances were selected for optimization based on 99% explained variation in mean left ventricle pressure and volume. The hybrid optimization strategy yielded the best overall results compared to 1st order optimization with automatic differentiation and finite difference approaches, with mean absolute percentage errors ranging from 6.67% to 14.14%. Errors in left ventricle elastance estimates for simulated aortic stenosis and mitral regurgitation were smallest when including synthetic measurements for arterial pressure and valvular flow rate at approximately 2% and degraded to roughly 5% when including volume trends as well. However, the latter resulted in better tracking of the left ventricle pressure waveforms and may be considered when the necessary equipment is available.

Estimation of Pulmonary Arterial Pressure Using Simulated Non-Invasive Measurements and Gradient-Based Optimization Techniques

Reliable quantification of pulmonary arterial pressure is essential in the diagnostic and prognostic assessment of a range of cardiovascular pathologies, including rheumatic heart disease, yet an accurate and routinely available method for its quantification remains elusive. This work proposes an approach to infer pulmonary arterial pressure based on scientific machine learning techniques and non-invasive, clinically available measurements. A 0D multicompartment model of the cardiovascular system was optimized using several optimization algorithms subject to forward-mode automatic differentiation. Measurement data were synthesized from known parameters to represent the healthy, mitral regurgitant, aortic stenosed, and combined valvular disease situations with and without pulmonary hypertension. Eleven model parameters were selected for optimization based on 95% explained variation in mean pulmonary arterial pressure. A hybrid Adam and limited-memory Broyden–Fletcher–Goldfarb–Shanno optimizer yielded the best results with input data including valvular flow rates, heart chamber volume changes, and systematic arterial pressure. Mean absolute percentage errors ranged from 1.8% to 3.78% over the simulated test cases. The model was able to capture pressure dynamics under hypertensive conditions with pulmonary arterial systole, diastole, and mean pressure average percentage errors of 1.12%, 2.49%, and 2.14%, respectively. The low errors highlight the potential of the proposed model to determine pulmonary pressure for diseased heart valves and pulmonary hypertensive conditions.

Differential methods for assessing sensitivity in biological models.

Differential sensitivity analysis is indispensable in fitting parameters, understanding uncertainty, and forecasting the results of both thought and lab experiments. Although there are many methods currently available for performing differential sensitivity analysis of biological models, it can be difficult to determine which method is best suited for a particular model. In this paper, we explain a variety of differential sensitivity methods and assess their value in some typical biological models. First, we explain the mathematical basis for three numerical methods: adjoint sensitivity analysis, complex perturbation sensitivity analysis, and forward mode sensitivity analysis. We then carry out four instructive case studies. (a) The CARRGO model for tumor-immune interaction highlights the additional information that differential sensitivity analysis provides beyond traditional naive sensitivity methods, (b) the deterministic SIR model demonstrates the value of using second-order sensitivity in refining model predictions, (c) the stochastic SIR model shows how differential sensitivity can be attacked in stochastic modeling, and (d) a discrete birth-death-migration model illustrates how the complex perturbation method of differential sensitivity can be generalized to a broader range of biological models. Finally, we compare the speed, accuracy, and ease of use of these methods. We find that forward mode automatic differentiation has the quickest computational time, while the complex perturbation method is the simplest to implement and the most generalizable.

Higher Order Automatic Differentiation of Higher Order Functions

We present semantic correctness proofs of automatic differentiation (AD). We consider a forward-mode AD method on a higher order language with algebraic data types, and we characterise it as the unique structure preserving macro given a choice of derivatives for basic operations. We describe a rich semantics for differentiable programming, based on diffeological spaces. We show that it interprets our language, and we phrase what it means for the AD method to be correct with respect to this semantics. We show that our characterisation of AD gives rise to an elegant semantic proof of its correctness based on a gluing construction on diffeological spaces. We explain how this is, in essence, a logical relations argument. Throughout, we show how the analysis extends to AD methods for computing higher order derivatives using a Taylor approximation.

A stochastic subspace approach to gradient-free optimization in high dimensions

We present a stochastic descent algorithm for unconstrained optimization that is particularly efficient when the objective function is slow to evaluate and gradients are not easily obtained, as in some PDE-constrained optimization and machine learning problems. The algorithm maps the gradient onto a low-dimensional random subspace of dimension $$\ell$$ at each iteration, similar to coordinate descent but without restricting directional derivatives to be along the axes. Without requiring a full gradient, this mapping can be performed by computing $$\ell$$ directional derivatives (e.g., via forward-mode automatic differentiation). We give proofs for convergence in expectation under various convexity assumptions as well as probabilistic convergence results under strong-convexity. Our method provides a novel extension to the well-known Gaussian smoothing technique to descent in subspaces of dimension greater than one, opening the doors to new analysis of Gaussian smoothing when more than one directional derivative is used at each iteration. We also provide a finite-dimensional variant of a special case of the Johnson–Lindenstrauss lemma. Experimentally, we show that our method compares favorably to coordinate descent, Gaussian smoothing, gradient descent and BFGS (when gradients are calculated via forward-mode automatic differentiation) on problems from the machine learning and shape optimization literature.

Optimization of a dual mixed refrigerant process using a nonsmooth approach

This article uses a nonsmooth flowsheeting methodology to create simulation and optimization models for dual mixed refrigerant processes. New improved operating conditions are obtained using the primal-dual interior-point optimizer IPOPT, with sensitivity information calculated using new developments in nonsmooth analysis to obtain generalized derivative information using a nonsmooth generalization of the vector forward mode of automatic differentiation. Several optimization studies are performed with constraints on both the minimum temperature difference (ΔTmin) and total heat exchanger conductance (UAmax) used to represent the trade-offs between energy consumption and the required heat transfer area. In addition, comparison is made with the conventional process simulator Aspen HYSYS using particle swarm optimization. Results show that the nonsmooth model was able to reduce the required compression power by 14.4% compared to the initial feasible design for the dual mixed refrigerant process, and by 20.4–21.6% for the dual mixed refrigerant process with NGL extraction. Furthermore, the solutions obtained from the nonsmooth model were 1.9–8.1% better than the design obtained by particle swarm optimization. Multistart optimization also shows that IPOPT converges to the best known solution when starting from an initial feasible design.

Automatic differentiation using operator overloading (ADOO) for implicit resolution of hyperbolic single phase and two-phase flow models

Implicit time integration schemes are widely used in computational fluid dynamics to speed-up computations. Indeed, implicit schemes usually allow for less stringent time-step stability constraints than their explicit counterpart. The derivation of an implicit scheme is however a challenging and time-consuming task, increasing substantially with the model equations complexity since this method usually requires fairly accurate evaluation of the spatial scheme's matrix Jacobian. This article presents a flexible method to overcome the difficulties associated to the computation of the derivatives, based on the forward mode of automatic differentiation using operator overloading (ADOO). Flexibility and simplicity of the method are illustrated through implicit resolution of various flow models of increasing complexity such as the compressible Euler equations, a two-phase flow model in full equilibrium [28] and a symmetric variant [44] of the two-phase flow model of Baer and Nunziato [2] dealing with mixtures in total disequilibrium.

Efficient differentiable programming in a functional array-processing language

We present a system for the automatic differentiation (AD) of a higher-order functional array-processing language. The core functional language underlying this system simultaneously supports both source-to-source forward-mode AD and global optimisations such as loop transformations. In combination, gradient computation with forward-mode AD can be as efficient as reverse mode, and that the Jacobian matrices required for numerical algorithms such as Gauss-Newton and Levenberg-Marquardt can be efficiently computed.

Convergence and error analysis of an automatically differentiated finite volume based heat conduction code

Purpose This paper aims to investigate the convergence and error properties of a finite volume-based heat conduction code that uses automatic differentiation to evaluate derivatives of solutions outputs with respect to arbitrary solution input(s). A problem involving conduction in a plane wall with convection at its surfaces is used as a test problem, as it has an analytical solution, and the error can be evaluated directly. Design/methodology/approach The finite volume method is used to discretize the transient heat diffusion equation with constant thermophysical properties. The discretized problem is then linearized, which results in two linear systems; one for the primary solution field and one for the secondary field, representing the derivative of the primary field with respect to the selected input(s). Derivatives required in the formation of the secondary linear system are obtained by automatic differentiation using an operator overloading and templating approach in C++. Findings The temporal and spatial discretization error for the derivative solution follows the same order of accuracy as the primary solution. Second-order accuracy of the spatial and temporal discretization schemes is confirmed for both primary and secondary problems using both orthogonal and non-orthogonal grids. However, it has been found that for non-orthogonal cases, there is a limit to the error reduction, which is concluded to be a result of errors in the Gauss-based gradient reconstruction method. Originality/value The convergence and error properties of derivative solutions obtained by forward mode automatic differentiation of finite volume-based codes have not been previously investigated.

Computationally relevant generalized derivatives: theory, evaluation and applications

A new method for evaluating generalized derivatives in nonsmooth problems is reviewed. Lexicographic directional (LD-)derivatives are a recently developed tool in nonsmooth analysis for evaluating generalized derivative elements in a tractable and robust way. Applicable to problems in both steady-state and dynamic settings, LD-derivatives exhibit a number of advantages over current theory and algorithms. As highlighted in this article, the LD-derivative approach now admits a suitable theory for inverse and implicit functions, nonsmooth dynamical systems and optimization problems, among others. Moreover, this technique includes an extension of the standard vector forward mode of automatic differentiation (AD) and acts as the natural extension of classical calculus results to the nonsmooth case in many ways. The theory of LD-derivatives is placed in the context of state-of-the-art methods in nonsmooth analysis, with an application in multistream heat exchanger modelling and design used to illustrate the usefulness of the approach.

Motor Estimation using Heterogeneous Sets of Objects in Conformal Geometric Algebra

In this paper we present a novel method for nonlinear rigid body motion estimation from noisy data using heterogeneous sets of objects of the conformal model in geometric algebra. The rigid body motions are represented by motors. We employ state-of-the-art nonlinear optimization tools and compute gradients and Jacobian matrices using forward-mode automatic differentiation based on dual numbers. The use of automatic differentiation enables us to employ a wide range of cost functions in the estimation process. This includes cost functions for motor estimation using points, lines and planes. Moreover, we explain how these cost functions make it possible to use other geometric objects in the conformal model in the motor estimation process, e.g., spheres, circles and tangent vectors. Experimental results show that we are able to successfully estimate rigid body motions from synthetic datasets of heterogeneous sets of conformal objects including a combination of points, lines and planes.

A vector forward mode of automatic differentiation for generalized derivative evaluation

Numerical methods for non-smooth equation-solving and optimization often require generalized derivative information in the form of elements of the Clarke Jacobian or the B-subdifferential. It is shown here that piecewise differentiable functions are lexicographically smooth in the sense of Nesterov, and that lexicographic derivatives of these functions comprise a particular subset of both the B-subdifferential and the Clarke Jacobian. Several recently developed methods for generalized derivative evaluation of composite piecewise differentiable functions are shown to produce identical results, which are also lexicographic derivatives. A vector forward mode of automatic differentiation (AD) is presented for evaluation of these derivatives, generalizing established methods and combining their computational benefits. This forward AD mode may be applied to any finite composition of known smooth functions, piecewise differentiable functions such as the absolute value function, , and , and certain non-smooth functions which are not piecewise differentiable, such as the Euclidean norm. This forward AD mode may be implemented using operator overloading, does not require storage of a computational graph, and is computationally tractable relative to the cost of a function evaluation. An implementation in C is discussed.

Adjoint-based optimization of experimental designs with many control variables

We propose a method for an efficient optimization of experimental designs, using a combination of discrete adjoint computations, Taylor arithmetic and matrix calculus. Compared to the state of the art of using finite differences or the forward mode of automatic differentiation, our proposed approach leads to a reduction of the relative temporal complexity from linear to constant time in the number of control variables and measurement weights. We demonstrate that the advantageous complexity results are not only of theoretical nature, but lead to significant speedups in practice as well. With our implementation we are very close to the theoretical bound of the cheap gradient principle. We present one academic (spatially discretized heat equation) and two industrial application examples (biochemical process/Diesel-oxidation catalysis process) where we achieve speedups that range between 10 and 100. In addition to our core results, we also describe an efficient adjoint approach for the treatment of differential algebraic equations and present adjoint formulas for constrained least-squares problems.

An efficient overloaded method for computing derivatives of mathematical functions in MATLAB

An object-oriented method is presented that computes without truncation the error derivatives of functions defined by MATLAB computer codes. The method implements forward-mode automatic differentiation via operator overloading in a manner that produces a new MATLAB code that computes the derivatives of the outputs of the original function with respect to the differentiation variables. Because the derivative code has the same input as the original function code, the method can be used recursively to generate derivatives of any order desired. In addition, the approach developed in this article has the feature that the derivatives are generated by simply evaluating the function on an instance of the class, thus making the method straightforward to use while simultaneously enabling differentiation of highly complex functions. A detailed description of the method is presented and the approach is illustrated and shown to be efficient on four examples.

DNAD, a simple tool for automatic differentiation of Fortran codes using dual numbers

DNAD (dual number automatic differentiation) is a simple, general-purpose tool to automatically differentiate Fortran codes written in modern Fortran (F90/ 95/2003) or legacy codes written in previous version of the Fortran language. It implements the forward mode of automatic differentiation using the arithmetic of dual numbers and the operator overloading feature of F90/ 95/2003. Very minimum changes of the source codes are needed to compute the first derivatives of Fortran programs. The advantages of DNAD in comparison to other existing similar computer codes are its programming simplicity, extensibility, and computational efficiency. Specifically, DNAD is more accurate and efficient than the popular complex-step approximation. Several examples are used to demonstrate its applications and advantages. Program summaryProgram title: DNADCatalogue identifier: AEOS_v1_0Program summary URL:http://cpc.cs.qub.ac.uk/summaries/AEOS_v1_0.htmlProgram obtainable from: CPC Program Library, Queen’s University, Belfast, N. IrelandLicensing provisions: Standard CPC licence, http://cpc.cs.qub.ac.uk/licence/licence.htmlNo. of lines in distributed program, including test data, etc.: 3922No. of bytes in distributed program, including test data, etc.: 18275Distribution format: tar.gzProgramming language: Fortran 90/95/2003.Computer: All computers with a modern FORTRAN compiler.Operating system: All platforms with a modern FORTRAN compiler.Classification: 4.12, 6.2.Nature of problem:Derivatives of outputs with respect to inputs of a Fortran code are often needed in physics, chemistry, and engineering. The author of the analysis code may no longer be available and the user may not have a deep knowledge of the code. Thus a simple tool is necessary to automatically differentiate the code with very minimum change to the source codes. This can be achieved using dual number arithmetic and operator overloading.Solution method:A new data type is defined with the first scalar component holding the function value and the second array component holding the first derivatives. All the basic operations and functions are overloaded with the new definitions according to dual number arithmetic. To differentiate an existing code, all real numbers should be replaced with this new data type and the input/output of the code should also be modified accordingly.Running time:For each additional independent variable, DNAD takes less time than the running time of the original analysis code. However, the actual running time depends on the compiler, the computer, and the operations involved in the code to be differentiated.

Parallel fluid flow control and optimisation with lattice Boltzmann methods and automatic differentiation

A framework to solve fluid flow control and optimisation problems numerically is presented. Problems are formulated on a mesoscopic basis. In a side condition, the dynamics of a Newtonian fluid is described by a family of lattice Boltzmann equations, which are linked to an incompressible Navier–Stokes equation. It is proposed to solve the non-linear optimisation problem by a line search algorithm. The needed derivatives are obtained by adapting a forward-mode automatic differentiation technique for parallel lattice Boltzmann equations. Emphasis is placed to obtain a realisation which is highly generic but also efficient in the sense of a scalable parallelisation. This is managed by taking advantage of polymorphism realised by an operator overloading approach. The realisation of the approach is also illustrated in this article by means of an example: the implementation for the open source library OpenLB1http://www.openlb.net.1 which enables the simulation of a great variety of 2D and 3D fluid flows by lattice Boltzmann methods. Numerical and performance results are presented for a series of steady-state distributed control problems with up to 1.029 unknown control parameters obtained on a high performance computer with up to 128 processing units. The obtained results are presented and discussed.