Efficient Gradient Computation Research Articles

Identification of a nonlinear spring and damper characteristics of a motorcycle suspension using test ride data

AbstractDuring test rides of motorcycles modifications are made to the suspension. In order to quantify those changes, the nonlinear spring and damper characteristics must be determined. This is usually done on a test bench. However, measurements on a test bench are closely related to high costs and high time exposure. Hence, a parameter identification after a test run, formulated as an optimization task, seems to be an auspicious approach. For this purpose a cost function is defined, which is minimized by considering the dynamics of the system. The strength of the contribution is the efficient gradient computation using the adjoint variable approach. In order to approximate the nonlinear spring and damper characteristics cubic splines are used. The values of the spline functions at specified grid points (knots) are adjusted such that the deviation between simulation and measurement is minimal.

OCEAN: An On-Chip Incremental-Learning Enhanced Artificial Neural Network Processor With Multiple Gated-Recurrent-Unit Accelerators

This paper presents OCEAN: an artificial neural network processor designed for accelerating gated-recurrent-unit (GRU) inference and on-chip incremental learning for sequential modeling. Implemented in 65-nm CMOS with silicon area of $2.9 \times 3.5\,\,\mathrm {mm}^{2}$ , the OCEAN processor features a 32-bit reduced instruction set computing core, 64-KB on-chip SRAM, and eight 16-bit four-cell GRU accelerators for inference and gradient computation. Each GRU accelerator is optimized and enhanced for efficient gradient computation. The processor is measured to consume 155 mW at the peak clock rate of 400 MHz and the supply of 1.2 V or 6.6 mW at 20 MHz/0.8 V. Both inference and on-chip incremental learning are accomplished on well-known AI tasks such as handwritten digit recognition, semantic natural language processing, and biomedical waveform-based seizure detection.

Gradient-based optimization for regression in the functional tensor-train format

Predictive analysis of complex computational models, such as uncertainty quantification (UQ), must often rely on using an existing database of simulation runs. In this paper we consider the task of performing low-multilinear-rank regression on such a database. Specifically we develop and analyze an efficient gradient computation that enables gradient-based optimization procedures, including stochastic gradient descent and quasi-Newton methods, for learning the parameters of a functional tensor-train (FT). We compare our algorithms with 22 other nonparametric and parametric regression methods on 10 real-world data sets and show that for many physical systems, exploiting low-rank structure facilitates efficient construction of surrogate models. We use a number of synthetic functions to build insight into behavior of our algorithms, including the rank adaptation and group-sparsity regularization procedures that we developed to reduce overfitting. Finally we conclude the paper by building a surrogate of a physical model of a propulsion plant on a naval vessel.

An adjoint method for gradient-based optimization of stellarator coil shapes

We present a method for stellarator coil design via gradient-based optimization of the coil-winding surface. The REGCOIL (Landreman 2017 Nucl. Fusion 57 046003) approach is used to obtain the coil shapes on the winding surface using a continuous current potential. We apply the adjoint method to calculate derivatives of the objective function, allowing for efficient computation of analytic gradients while eliminating the numerical noise of approximate derivatives. We are able to improve engineering properties of the coils by targeting the root-mean-squared current density in the objective function. We obtain winding surfaces for W7-X and HSX which simultaneously decrease the normal magnetic field on the plasma surface and increase the surface-averaged distance between the coils and the plasma in comparison with the actual winding surfaces. The coils computed on the optimized surfaces feature a smaller toroidal extent and curvature and increased inter-coil spacing. A technique for computation of the local sensitivity of figures of merit to normal displacements of the winding surface is presented, with potential applications for understanding engineering tolerances.

Gradient-enhanced kriging for high-dimensional problems

Surrogate models provide an affordable alternative to the evaluation of expensive deterministic functions. However, the construction of accurate surrogate models with many independent variables is currently prohibitive because they require a large number of function evaluations for the desired accuracy. Gradient-enhanced kriging has the potential to reduce the number of evaluations when efficient gradient computation, such as an adjoint method, is available. However, current gradient-enhanced kriging methods do not scale well with the number of sampling points because of the rapid growth in the size of the correlation matrix, where new information is added for each sampling point in each direction of the design space. Furthermore, they do not scale well with the number of independent variables because of the increase in the number of hyperparameters that must be estimated. To address this issue, we develop a new gradient-enhanced surrogate model approach that drastically reduces the number of hyperparameters through the use of the partial least squares method to maintain accuracy. In addition, this method is able to control the size of the correlation matrix by adding only relevant points defined by the information provided by the partial least squares method. To validate our method, we compare the global accuracy of the proposed method with conventional kriging surrogate models on two analytic functions with up to 100 dimensions, as well as engineering problems of varied complexity with up to 15 dimensions. We show that the proposed method requires fewer sampling points than conventional methods to obtain the desired accuracy, or it provides more accuracy for a fixed budget of sampling points. In some cases, we get models that are over three times more accurate than previously developed surrogate models for the same computational time, and over 3200 times faster than standard gradient-enhanced kriging models for the same accuracy.

Scalable Backpropagation for Gaussian Processes using Celerite

This research note presents a derivation and implementation of efficient and scalable gradient computations using the celerite algorithm for Gaussian Process (GP) modeling. The algorithms are derived in a reverse accumulation or backpropagation framework and they can be easily integrated into existing automatic differentiation frameworks to provide a scalable method for evaluating the gradients of the GP likelihood with respect to all input parameters. The algorithm derived in this note uses less memory and is more efficient than versions using automatic differentiation and the computational cost scales linearly with the number of data points.

Multiscale gradient computation for flow in heterogeneous porous media

An efficient multiscale (MS) gradient computation method for subsurface flow management and optimization is introduced. The general, algebraic framework allows for the calculation of gradients using both the Direct and Adjoint derivative methods. The framework also allows for the utilization of any MS formulation that can be algebraically expressed in terms of a restriction and a prolongation operator. This is achieved via an implicit differentiation formulation. The approach favors algorithms for multiplying the sensitivity matrix and its transpose with arbitrary vectors. This provides a flexible way of computing gradients in a form suitable for any given gradient-based optimization algorithm. No assumption w.r.t. the nature of the problem or specific optimization parameters is made. Therefore, the framework can be applied to any gradient-based study. In the implementation, extra partial derivative information required by the gradient computation is computed via automatic differentiation. A detailed utilization of the framework using the MS Finite Volume (MSFV) simulation technique is presented. Numerical experiments are performed to demonstrate the accuracy of the method compared to a fine-scale simulator. In addition, an asymptotic analysis is presented to provide an estimate of its computational complexity. The investigations show that the presented method casts an accurate and efficient MS gradient computation strategy that can be successfully utilized in next-generation reservoir management studies.

Gradient-Based Optimization for Poroelastic and Viscoelastic MR Elastography

We describe an efficient gradient computation for solving inverse problems arising in magnetic resonance elastography (MRE). The algorithm can be considered as a generalized 'adjoint method' based on a Lagrangian formulation. One requirement for the classic adjoint method is assurance of the self-adjoint property of the stiffness matrix in the elasticity problem. In this paper, we show this property is no longer a necessary condition in our algorithm, but the computational performance can be as efficient as the classic method, which involves only two forward solutions and is independent of the number of parameters to be estimated. The algorithm is developed and implemented in material property reconstructions using poroelastic and viscoelastic modeling. Various gradient- and Hessian-based optimization techniques have been tested on simulation, phantom and in vivo brain data. The numerical results show the feasibility and the efficiency of the proposed scheme for gradient calculation.

GPU-accelerated adjoint algorithmic differentiation

Many scientific problems such as classifier training or medical image reconstruction can be expressed as minimization of differentiable real-valued cost functions and solved with iterative gradient-based methods. Adjoint algorithmic differentiation (AAD) enables automated computation of gradients of such cost functions implemented as computer programs. To backpropagate adjoint derivatives, excessive memory is potentially required to store the intermediate partial derivatives on a dedicated data structure, referred to as the “tape”. Parallelization is difficult because threads need to synchronize their accesses during taping and backpropagation. This situation is aggravated for many-core architectures, such as Graphics Processing Units (GPUs), because of the large number of light-weight threads and the limited memory size in general as well as per thread. We show how these limitations can be mediated if the cost function is expressed using GPU-accelerated vector and matrix operations which are recognized as intrinsic functions by our AAD software. We compare this approach with naive and vectorized implementations for CPUs. We use four increasingly complex cost functions to evaluate the performance with respect to memory consumption and gradient computation times. Using vectorization, CPU and GPU memory consumption could be substantially reduced compared to the naive reference implementation, in some cases even by an order of complexity. The vectorization allowed usage of optimized parallel libraries during forward and reverse passes which resulted in high speedups for the vectorized CPU version compared to the naive reference implementation. The GPU version achieved an additional speedup of 7.5±4.4, showing that the processing power of GPUs can be utilized for AAD using this concept. Furthermore, we show how this software can be systematically extended for more complex problems such as nonlinear absorption reconstruction for fluorescence-mediated tomography. Program summaryProgram title: AD-GPUCatalogue identifier: AEYX_v1_0Program summary URL: http://cpc.cs.qub.ac.uk/summaries/AEYX_v1_0.htmlProgram obtainable from: CPC Program Library, Queen’s University, Belfast, N. IrelandLicensing provisions: Standard CPC licence, http://cpc.cs.qub.ac.uk/licence/licence.htmlNo. of lines in distributed program, including test data, etc.: 16715No. of bytes in distributed program, including test data, etc.: 143683Distribution format: tar.gzProgramming language: C++ and CUDA.Computer: Any computer with a compatible C++ compiler and a GPU with CUDA capability 3.0 or higher.Operating system: Windows 7 or Linux.RAM: 16 GbyteClassification: 4.9, 4.12, 6.1, 6.5.External routines: CUDA 6.5, Intel MKL (optional) and routines from BLAS, LAPACK and CUBLASNature of problem: Gradients are required for many optimization problems, e.g. classifier training or nonlinear image reconstruction. Often, the function, of which the gradient is required, can be implemented as a computer program. Then, algorithmic differentiation methods can be used to compute the gradient. Depending on the approach this may result in excessive requirements of computational resources, i.e. memory and arithmetic computations. GPUs provide massive computational resources but require special considerations to distribute the workload onto many light-weight threads.Solution method: Adjoint algorithmic differentiation allows efficient computation of gradients of cost functions given as computer programs. The gradient can be theoretically computed using a similar amount of arithmetic operations as one function evaluation. Optimal usage of parallel processors and limited memory is a major challenge which can be mediated by the use of vectorization.Restrictions: To use the GPU-accelerated adjoint algorithmic differentiation method, the cost function must be implemented using the provided AD-GPU intrinsics for matrix and vector operations. Unusual features:GPU-acceleration.Additional comments: The code uses some features of C++11, e.g. std::shared ptr. Alternatively, the boost library can be used.Running time: The time to run the example program is a few minutes or up to a few hours to reproduce the performance measurements.

Parameters affecting laterally loaded piles in frozen soils by an efficient sensitivity analysis method

In cold regions, accurately simulating the nonlinear responses of frozen soil–structure interaction (SSI) systems is of significant importance for the seismic safety assessment. This paper presents a finite element (FE) simulation of nonlinear lateral responses of a real reinforced-concrete filled steel pipe pile embedded in frozen ground at an outdoor test site in Fairbanks, Alaska. A pressure-independent multi-yield surface J2 plasticity model is used to simulate the frozen soil behavior, while frame element with fiber section and nonlinear steel/concrete model are used for the pile. The FE analysis results agree well with the experiment results. Furthermore, the response sensitivities to various material parameters are computed by using an efficient and accurate gradient computation method, i.e., direct differentiation method (DDM), with limited additional computational cost. Based on DDM, the relative importance of material parameters on system responses is studied when the system is subjected to varying lateral deflections, respectively (corresponding to different levels of lateral loads such as earthquakes). Stiffness-related parameters are dominant when the system is subject to small deflections, while strength-related or post-yield parameters become dominant for large deflections. In addition, global responses are slightly more sensitive to material parameters of pile than the parameters of frozen soil. The response sensitivity to the unfrozen soil below the frozen ground crust is almost negligible. Finally, the response sensitivities to the soil parameters are observed to decrease when the distance of the soil from the pile increases. The sensitivity analysis methodology presented in this paper can be adapted to other frozen soil–pile interaction cases and the study results provide valuable insight for engineering practice.

SAGRAD: A Program for Neural Network Training with Simulated Annealing and the Conjugate Gradient Method.

SAGRAD (Simulated Annealing GRADient), a Fortran 77 program for computing neural networks for classification using batch learning, is discussed. Neural network training in SAGRAD is based on a combination of simulated annealing and Møller’s scaled conjugate gradient algorithm, the latter a variation of the traditional conjugate gradient method, better suited for the nonquadratic nature of neural networks. Different aspects of the implementation of the training process in SAGRAD are discussed, such as the efficient computation of gradients and multiplication of vectors by Hessian matrices that are required by Møller’s algorithm; the (re)initialization of weights with simulated annealing required to (re)start Møller’s algorithm the first time and each time thereafter that it shows insufficient progress in reaching a possibly local minimum; and the use of simulated annealing when Møller’s algorithm, after possibly making considerable progress, becomes stuck at a local minimum or flat area of weight space. Outlines of the scaled conjugate gradient algorithm, the simulated annealing procedure and the training process used in SAGRAD are presented together with results from running SAGRAD on two examples of training data.

Reservoir management optimization using well-specific upscaling and control switching

Optimizing objectives of long-term reservoir management typically requires a high number of reservoir simulations. Two important remedies to reduce the runtime (and make the optimization problem manageable) are model reduction/upscaling and efficient computation of gradients. Adjoint methods are generally considered to be the most efficient means for obtaining gradients. In this work, we employ a global approach to compute upscaled transmissibilities for a coarse model. The proposed coarsening strategy takes as input any number of fine-scale states (pressure fields from, e.g., a previous simulation), and produces a coarse-scale model calibrated to the specific flow scenario(s) dictated by the input. We argue that compared to traditional general-purpose upscaling approaches, more aggressive coarsening can be applied in this type of scenario-specific upscaling. Utilizing a fully-implicit, three-phase, black-oil simulator with adjoint capabilities, we investigate the performance of our methodology by optimizing the net-present-value (NPV) for a real-field model. For the optimization approach, we develop a control switching approach for efficient handling of output constraints. We consider moderately upscaled models, and conclude that for the case we consider, at least two orders of magnitude speed-up can be achieved whilst retaining sufficient accuracy.

Stochastic reduced order models for inverse problems under uncertainty

This work presents a novel methodology for solving inverse problems under uncertainty using stochastic reduced order models (SROMs). Given statistical information about an observed state variable in a system, unknown parameters are estimated probabilistically through the solution of a model-constrained, stochastic optimization problem. The point of departure and crux of the proposed framework is the representation of a random quantity using a SROM—a low dimensional, discrete approximation to a continuous random element that permits efficient and non-intrusive stochastic computations. Characterizing the uncertainties with SROMs transforms the stochastic optimization problem into a deterministic one. The non-intrusive nature of SROMs facilitates efficient gradient computations for random vector unknowns and relies entirely on calls to existing deterministic solvers. Furthermore, the method is naturally extended to handle multiple sources of uncertainty in cases where state variable data, system parameters, and boundary conditions are all considered random.The new and widely-applicable SROM framework is formulated for a general stochastic optimization problem in terms of an abstract objective function and constraining model. For demonstration purposes, however, we study its performance in the specific case of inverse identification of random material parameters in elastodynamics. We demonstrate the ability to efficiently recover random shear moduli given material displacement statistics as input data. We also show that the approach remains effective for the case where the loading in the problem is random as well.

Utilization of efficient gradient and Hessian computations in the force field optimization process of molecular simulations

Computer simulations of chemical systems, especially systems of condensed matter, are highly important for both scientific and industrial applications. Thereby, molecular interactions are modeled on a microscopic level in order to study their impact on macroscopic phenomena. To be capable of predicting physical properties quantitatively, accurate molecular models are indispensable. Molecular interactions are described mathematically by force fields, which have to be parameterized. Recently, an automated gradient-based optimization procedure was published by the authors based on the minimization of a loss function between simulated and experimental physical properties. The applicability of gradient-based procedures is not trivial at all because of two reasons: firstly, simulation data are affected by statistical noise, and secondly, the molecular simulations required for the loss function evaluations are extremely time-consuming. Within the optimization process, gradients and Hessians were approximated by finite differences so that additional simulations for the respective modified parameter sets were required. Hence, a more efficient approach to computing gradients and Hessians is presented in this work. The method developed here is based on directional instead of partial derivatives. It is compared with the classical computations with respect to computation time. Firstly, molecular simulations are replaced by fit functions that define a functional dependence between specific physical observables and force field parameters. The goal of these simulated simulations is to assess the new methodology without much computational effort. Secondly, it is applied to real molecular simulations of the three chemical substances phosgene, methanol and ethylene oxide. It is shown that up to 75% of the simulations can be avoided using the new algorithm.

Improvement of state profile accuracy in nonlinear dynamic optimization with the quasi‐sequential approach

AbstractQuasi‐sequential methods are efficient and flexible strategies for the solution of dynamic optimization problems. At the heart of these strategies lies the time discretization and approximation of dynamic systems for nonlinear optimization problems. To address this question, we employ a time derivative analysis within the quasi‐sequential approach and derive a finite element placement strategy. In addition, methods for direct error prediction are applied to this approach and extended with a proposed time derivative analysis. According to the information for current time derivatives, subintervals are introduced that improve accuracy of state profiles. Since this is only done in the simulation layer, the nonlinear programing solver need not be restarted. An efficient gradient computation is also derived for these subintervals; the resulting enhanced accuracy accelerates convergence performance and increases the robustness of the solution to initialization. A beer fermentation process case study is presented to demonstrate the effectiveness of the proposed approach. © 2011 American Institute of Chemical Engineers AIChE J, 2011

Highly Efficient Gradient Computation for Density-Constrained Analytical Placement

Recent analytical global placers use density constraints to approximate nonoverlap constraints, and these show very successful results. This paper unifies a wide range of density smoothing techniques called global smoothing and presents a highly efficient method for computing the gradient of such smoothed densities used in several well-known analytical placers. This method reduces the complexity of the gradient computation by a factor of n compared with a naive method, where n is the number of modules. Furthermore, with this efficient gradient computation, it is able to support an efficient nonlinear programming-based placement framework, which supersedes the existing force-directed placement methods. Experiments show that replacing the approximated gradient computation in mPL6 with the exact gradient computation improves wire length by 15% on the IBM-HB+ benchmark and by 3% on average on the modified International Symposium on Physical Design 2005 (ISPD'05) and ISPD'06 placement contest benchmarks with movable macros. The results also show that the augmented Lagrangian method outperforms the quadratic penalty method with the exact gradient computation.

New advances in three-dimensional controlled-source electromagnetic inversion

SUMMARY New techniques for improving both the computational and imaging performance of the three-dimensional (3-D) electromagnetic inverse problem are presented. A non-linear conjugate gradient algorithm is the framework of the inversion scheme. Full wave equation modelling for controlled sources is utilized for data simulation along with an efficient gradient computation approach for the model update. Improving the modelling efficiency of the 3-D finite difference (FD) method involves the separation of the potentially large modelling mesh, defining the set of model parameters, from the computational FD meshes used for field simulation. Grid spacings and thus overall grid sizes can be reduced and optimized according to source frequencies and source–receiver offsets of a given input data set. Further computational efficiency is obtained by combining different levels of parallelization. While the parallel scheme allows for an arbitrarily large number of parallel tasks, the relative amount of message passing is kept constant. Image enhancement is achieved by model parameter transformation functions, which enforce bounded conductivity parameters and thus prevent parameter overshoots. Further, a remedy for treating distorted data within the inversion process is presented. Data distortions simulated here include positioning errors and a highly conductive overburden, hiding the desired target signal. The methods are demonstrated using both synthetic and field data.

Response Gradients for Nonlinear Beam-Column Elements under Large Displacements

Accurate and efficient response gradient computations for nonlinear geometry are required in structural reliability, optimization, system identification, and response sensitivity analysis of frame structures that undergo large displacements. In this paper, the exact response gradient of beam-column finite elements under large displacements is derived considering uncertain material and geometric parameters. The element response formulation takes place in a corotational reference frame that displaces and rotates with the element, thus permitting the separation of nonlinear material from nonlinear geometric effects in the computation of the response, as well as of the gradient of the response. Relative to the corotational reference frame, small deformation theory suffices for all structural engineering applications. Thus, the proposed response gradient computations are applicable to most beam-column element formulations available in the literature, including force-based and mixed formulations.

Optimization of nonlinear wave function parameters

AbstractAn energy‐based optimization method is presented for our recently developed nonlinear wave function expansion form for electronic wave functions. This expansion form is based on spin eigenfunctions, using the graphical unitary group approach (GUGA). The wave function is expanded in a basis of product functions, allowing application to closed‐shell and open‐shell systems and to ground and excited electronic states. Each product basis function is itself a multiconfigurational function that depends on a relatively small number of nonlinear parameters called arc factors. The energy‐based optimization is formulated in terms of analytic arc factor gradients and orbital‐level Hamiltonian matrices that correspond to a specific kind of uncontraction of each of the product basis functions. These orbital‐level Hamiltonian matrices give an intuitive representation of the energy in terms of disjoint subsets of the arc factors, they provide for an efficient computation of gradients of the energy with respect to the arc factors, and they allow optimal arc factors to be determined in closed form for subspaces of the full variation problem. Timings for energy and arc factor gradient computations involving expansion spaces of >1024 configuration state functions are reported. Preliminary convergence studies and molecular dissociation curves are presented for some small molecules. © 2006 Wiley Periodicals, Inc. Int J Quantum Chem, 2006

On Efficient Computation of the Optimization Problem Arising in the Inverse Modeling of Non-Stationary Multiphase Multicomponent Flow Through Porous Media

In this paper we discuss a method of solving inverse problems in non-isothermal multiphase multicomponent flow through porous media. The conceptual model is described by a system of non-linear partial differential equations which involve unknown parameters. These parameters are to be determined using a set of observations at discrete points in space and time by an optimization method. It is based on a reduced Gauss-Newton iteration in combination with an efficient gradient computation which takes advantage of a recently developed efficient numerical simulation technique. A sensitivity analysis is carried out for the optimum parameter set. Numerical experiments are performed for a one dimensional column experiment carried out at the VEGAS, University of Stuttgart, Germany.