Gradient-based Optimization Techniques Research Articles

Differentiable Visual Computing: Challenges and Opportunities.

Classical algorithms typically contain domain-specific insights. This makes them often more robust, interpretable, and efficient. On the other hand, deep-learning models must learn domain-specific insight from scratch from a large amount of data using gradient-based optimization techniques. To have the best of both worlds, we should make classical visual computing algorithms differentiable to enable gradient-based optimization. Computing derivatives of classical visual computing algorithms is challenging: there can be discontinuities, and the computation pattern is often irregular compared to high-arithmetic intensity neural networks. In this article, we discuss the benefits and challenges of combining classical visual computing algorithms and modern data-driven methods, with particular emphasis to my thesis, which took one of the first steps toward addressing these challenges.

Tracking of a Fixed-Shape Moving Object Based on the Gradient Descent Method.

Tracking moving objects is one of the most promising yet the most challenging research areas pertaining to computer vision, pattern recognition and image processing. The challenges associated with object tracking range from problems pertaining to camera axis orientations to object occlusion. In addition, variations in remote scene environments add to the difficulties related to object tracking. All the mentioned challenges and problems pertaining to object tracking make the procedure computationally complex and time-consuming. In this paper, a stochastic gradient-based optimization technique has been used in conjunction with particle filters for object tracking. First, the object that needs to be tracked is detected using the Maximum Average Correlation Height (MACH) filter. The object of interest is detected based on the presence of a correlation peak and average similarity measure. The results of object detection are fed to the tracking routine. The gradient descent technique is employed for object tracking and is used to optimize the particle filters. The gradient descent technique allows particles to converge quickly, allowing less time for the object to be tracked. The results of the proposed algorithm are compared with similar state-of-the-art tracking algorithms on five datasets that include both artificial moving objects and humans to show that the gradient-based tracking algorithm provides better results, both in terms of accuracy and speed.

Forecasting in Big Data Environments: An Adaptable and Automated Shrinkage Estimation of Neural Networks (AAShNet)

This paper considers improved forecasting in possibly nonlinear dynamic settings, with high-dimension predictors (“big data” environments). To overcome the curse of dimensionality and manage data and model complexity, we examine shrinkage estimation of a back-propagation algorithm of a neural net with skip-layer connections. We expressly include both linear and nonlinear components. This is a high-dimensional learning approach including both sparsity \(L_1\) and smoothness \(L_2\) penalties, allowing high-dimensionality and nonlinearity to be accommodated in one step. This approach selects significant predictors as well as the topology of the neural network. We estimate optimal values of shrinkage hyperparameters by incorporating a gradient-based optimization technique resulting in robust predictions with improved reproducibility. The latter has been an issue in some approaches. This is statistically interpretable and unravels some network structure, commonly left to a black box. An additional advantage is that the nonlinear part tends to get pruned if the underlying process is linear. In an application to forecasting equity returns, the proposed approach captures nonlinear dynamics between equities to enhance forecast performance. It offers an appreciable improvement over current univariate and multivariate models by actual portfolio performance.

An inverse strategy to determine constitutive parameters of tubular materials for hydroforming processes

This paper is to determine the flow stress curve of 5049-O aluminium alloy by a tube hydraulic bulging test with fixed end-conditions. During this test, several tubular specimens are bulged under different internal pressures before their bursting, and the corresponding bulging height and wall thickness at the pole are measured. An inverse strategy is developed to determine the constitutive parameters of tubular materials based on experimental data, which combines the finite element method with gradient-based optimization techniques. In this scheme, the objective function is formulated with the sum of least squares of the error between numerical and experimental data, and finite difference approximation is used to calculate the gradient. The tubular material behavior is assumed to meet the von Mises yield criterion and Hollomon exponential hardening law. Then, constitutive parameters identification is performed by minimization of the objective function. In order to validate the performance of this framework, identified parameters are compared with those obtained by two types of theoretical models, and tensile tests are performed on specimens cut from the same tubes. The comparison shows that this inverse framework is robust and can achieve a more accurate parameter identification by eliminating mechanical and geometrical assumptions in classical theoretical analysis.

OPTIMIZED DEEP NEURAL NETWORKS ARCHITECTURE MODEL FOR BREAST CANCER DIAGNOSIS

Breast cancer has increasingly claimed the lives of women. Oncologists use digital mammograms as a viable source to detect breast cancer and classify it into benign and malignant based on the severity. The performance of the traditional methods on breast cancer detection could not be improved beyond a certain point due to the limitations and scope of computing. Moreover, the constrained scope of image processing techniques in developing automated breast cancer detection systems has motivated the researchers to shift their focus towards Artificial Intelligence based models. The Neural Networks (NN) have exhibited greater scope for the development of automated medical image analysis systems with the highest degree of accuracy. As NN model enables the automated system to understand the feature of problem-solving without being explicitly programmed. The optimization for NN offers an additional payoff on accuracy, computational complexity, and time. As the scope and suitability of optimization methods are data-dependent, the choice of selection of an appropriate optimization method itself is emerging as a prominent domain of research. In this paper, Deep Neural Networks (DNN) with different optimizers and Learning rates were designed for the prediction of breast cancer and its classification. Comparative performance analysis of five distinct first-order gradient-based optimization techniques, namely, Adaptive Gradient (Adagrad), Root Mean Square Propagation (RMSProp), Adaptive Delta (Adadelta), Adaptive Moment Estimation (Adam), and Stochastic Gradient Descent (SGD), is carried out to make predictions on the classification of breast cancer masses. For this purpose, the Mammographic Mass dataset was chosen for experimentation. The parameters determined for experiments were chosen on the number of hidden layers and learning rate along with hyperparameter tuning. The impacts of those optimizers were tested on the NN with One Hidden Layer (NN1HL), DNN with Three Hidden Layers (DNN4HL), and DNN with Eight Hidden Layers (DNN8HL). The experimental results showed that DNN8HL-Adam (DNN8HL-AM) had produced the highest accuracy of 91% among its counterparts. This research endorsed that the incorporation of optimizers in DNN contributes to an increased accuracy and optimized architecture for automated system development using neural networks.

Hamiltonian coordination primitives for decentralized multiagent navigation

We focus on decentralized navigation among multiple non-communicating agents in continuous domains without explicit traffic rules, such as sidewalks, hallways, or squares. Following collision-free motion in such domains requires effective mechanisms of multiagent behavior prediction. Although this prediction problem can be shown to be NP-hard, humans are often capable of solving it efficiently by leveraging sophisticated mechanisms of implicit coordination. Inspired by the human paradigm, we propose a novel topological formalism that explicitly models multiagent coordination. Our formalism features both geometric and algebraic descriptions enabling the use of standard gradient-based optimization techniques for trajectory generation but also symbolic inference over coordination strategies. In this article, we contribute (a) HCP (Hamiltonian Coordination Primitives), a novel multiagent trajectory-generation pipeline that accommodates spatiotemporal constraints formulated as symbolic topological specifications corresponding to a desired coordination strategy; (b) HCPnav, an online planning framework for decentralized collision avoidance that generates motion by following multiagent trajectory primitives corresponding to high-likelihood, low-cost coordination strategies. Through a series of challenging trajectory-generation experiments, we show that HCP outperforms a trajectory-optimization baseline in generating trajectories of desired topological specifications in terms of success rate and computational efficiency. Finally, through a variety of navigation experiments, we illustrate the efficacy of HCPnav in handling challenging multiagent navigation scenarios under homogeneous or heterogeneous agents across a series of environments of different geometry.

Achieving a flat, wideband frequency response of a loudspeaker unit by numerical optimization with requirements on its directivity.

This paper demonstrates how significant improvement in frequency response and directivity of a loudspeaker may be obtained by optimizing the local properties of the materials for the diaphragm and surround. Performance is investigated as the considered frequency range and off-axis requirements are progressively expanded. The results are generated by optimizing the values and layout of stiffness, mass, and damping of both the speaker diaphragm and surround. This is accomplished using a density and gradient-based optimization technique in conjunction with a fully coupled finite element model of the loudspeaker and the surrounding acoustic domain. The targeted frequency range is from 600 Hz up to 10 kHz and the range for the directivity is from 0° to 30°. The results show that a completely flat on-axis response is achievable even for very broad frequency ranges and that a reasonably flat response over a wide directivity can be obtained as well. The results presented in this research assume that complete design and production freedom are available.

Accelerated Optimization of Metasurfaces with the Woodbury Matrix Identity

Direct optimization of the impedances of a metasurface is difficult due to the large number of subwavelength elements comprising a metasurface. Gradient-based optimization techniques numerically approximate the gradient using finite differences requiring at least one cost evaluation for each element in the metasurface. If the cost evaluation is expensive or the number of elements is large, then the calculation of the gradient becomes time-consuming and impractical. When metasurfaces are modeled using integral equations, the cost function evaluation involves matrix inversions of the linear system constructed with the method of moments. Since each component of the gradient perturbs the linear system along a few diagonal elements of the impedance matrix, the matrix inversions can be accelerated using the Woodbury Matrix Identity. This accelerated solution of the linear system allows the gradient to be calculated efficiently permitting the direct optimization of metasurface impedances. In this paper, an acceleration technique for the gradient descent optimization of metasurfaces modeled with integral equations is presented. Numerical results are provided and show up to a 26.5 times improvement in computation time for the cases presented.

A Novel Reconstruction Method to Increase Spatial Resolution in Electron Probe Microanalysis

The spatial resolution of electron probe microanalysis (EPMA), a non-destructive method to determine the chemical composition of materials, is currently restricted to a pixel size larger than the volume of interaction between beam electrons and the material, as a result of limitations on the underlying k-ratio model. Using more sophisticated models to predict k-ratios while solving the inverse problem of reconstruction offers a possibility to increase the spatial resolution. Here, a k-ratio model based on the deterministic M1-model in Boltzmann Continuous Slowing-Down approximation (BCSD) will be utilized to present a reconstruction method for EPMA which is implemented as a PDE-constrained optimization problem. Iterative gradient-based optimization techniques are used in combination with the adjoint state method to calculate the gradient in order to solve the optimization problem efficiently. The accuracy of the spatial resolution still depends on the number and quality of the measured data, but in contrast to conventional reconstruction methods, an overlapping of the interaction volumes of different measurements is permissible without ambiguous solutions. The combination of k-ratios measured with various electron beam configurations is necessary for a high resolution. Attempts to reconstruct materials with synthetic data show challenges that occur with small reconstruction pixels, but also indicate the potential to improve the spatial resolution in EPMA using the presented method.

Stabilisation of distributed time-delay systems: a smoothed spectral abscissa optimisation approach

We address the stabilisation problem of linear time-invariant systems with pointwise and distributed delays. We first introduce the concept of the smoothed spectral abscissa, a robust stability measure previously studied for pointwise delay systems. We show that the new stability measure for distributed systems is smooth with respect to the system parameters, and robust in the sense that it provides a trade-off between the decay rate of the solution and the norm of the system. Next, we reformulate the stabilisation problem as an unconstrained minimisation problem considering the smoothed spectral abscissa as the objective function, and show that its properties enable solving the stabilisation problem by using standard gradient-based optimisation techniques. We show the potential of the proposed approach by using it for the stabilisation of a fleet of vehicles, and an oscillator with a class of time-varying delays in the input.

On the optimisation efficiency for the inverse identification of constitutive model parameters

The development of full-field measurement techniques paved the way for the design of new mechanical tests. However, because these mechanical tests provide heterogeneous strain fields, no closed-form solution exists between the measured deformation fields and the constitutive parameters. Therefore, inverse identification techniques should be used to calibrate constitutive models, such as the widely known finite element model updating (FEMU) and the virtual fields method (VFM). Although these inverse identification techniques follow distinct approaches to explore full-field measurements, they all require using an optimisation technique to find the optimum set of material parameters. Nonetheless, the choice of a suitable optimisation technique lacks attention and proper research. Most studies tend to use a least-squares gradient-based optimisation technique, such as the Levenberg-Marquardt algorithm. This work analyses optimisation algorithms, gradient-based and -free algorithms, for the inverse identification of constitutive model parameters. To avoid needless implementation and take advantage of highly developed programming languages, the optimisation algorithms available in optimisation libraries are used. A FEMU based approach is considered in the calibration of a thermoelastoviscoplastic model. The material parameters governing strain hardening, temperature and strain rate are identified. Results are discussed in terms of efficiency and the robustness of the optimisation processes.

Bayesian topology optimization for efficient design of origami folding structures

Bayesian optimization (BO) is a popular method for solving optimization problems involving expensive objective functions. Although BO has been applied across various fields, its use in structural optimization area is in its early stages. Origami folding structures provide a complex design space where the use of an efficient optimizer is critical. In this work for the first time we demonstrate the ability of BO to solve origami-inspired design problems. We use a Gaussian process (GP) as the surrogate model that is trained to mimic the response of the expensive finite element (FE) objective function. The ability of this BO-FE framework to find optimal designs is verified by applying it to well-known origami design problems. We compare the performance of the proposed approach to traditional gradient-based optimization techniques and genetic algorithm methods in terms of ability to discover designs and computational efficiency. BO has many user-defined components/parameters and intuitions for these for structural optimization are currently limited. In this work, we study the role of hyperparameter tuning and the sensitivity of Bayesian optimization to the quality and size of the initial training set. Taking a holistic view of the computational expense, we propose various heuristic approaches to reduce the overall cost of optimization. Our results show that Bayesian optimization is an efficient alternative to traditional methods. It allows for the discovery of optimal designs using fewer finite element solutions, which makes it an attractive choice for the non-convex design space of origami fold mechanics.

The Switch Point Algorithm

The Switch Point Algorithm is a new approach for solving optimal control problems whose solutions are either singular or bang-bang or both singular and bang-bang, and which possess a finite number of jump discontinuities in an optimal control at the points in time where the solution structure changes. Problems in this class can often be reduced to an optimization over the switching points. Formulas are derived for the derivative of the objective with respect to the switch points, the initial costate, and the terminal time. All these derivatives can be computed simultaneously in just one integration of the state and costate dynamics. Hence, gradient-based unconstrained optimization techniques, including the conjugate gradient method or quasi-Newton methods, can be used to compute an optimal control. The performance of the algorithm is illustrated using test problems with known solutions and comparisons with other algorithms from the literature.

MaxAFL: Maximizing Code Coverage with a Gradient-Based Optimization Technique

Evolutionary fuzzers generally work well with typical software programs because of their simple algorithm. However, there is a limitation that some paths with complex constraints cannot be tested even after long execution. Fuzzers based on concolic execution have emerged to address this issue. The concolic execution fuzzers also have limitations in scalability. Recently, the gradient-based fuzzers that use a gradient to mutate inputs have been introduced. Gradient-based fuzzers can be applied to real-world programs and achieve high code coverage. However, there is a problem that the existing gradient-based fuzzers require heavyweight analysis or sufficient learning time. In this paper, we propose a new type of gradient-based fuzzer, MaxAFL, to overcome the limitations of existing gradient-based fuzzers. Our approach constructs an objective function through fine-grained static analysis. After constructing a well-made objective function, we can apply the gradient-based optimization algorithm. We use a modified gradient-descent algorithm to minimize our objective function and propose some probabilistic techniques to escape local optimum. We introduce an adaptive objective function which aims to explore various paths in the program. We implemented MaxAFL based on the original AFL. MaxAFL achieved increase of code coverage per time compared with three other fuzzers in six open-source Linux binaries. We also measured cumulative code coverage per total execution, and MaxAFL outperformed the other fuzzers in this metric. Finally, MaxAFL can also find more bugs than the other fuzzers.

Truncation Error Based Mesh Optimization

AbstractTruncation error is used to drive mesh adaptation in order to reduce the discretization error in solutions to a variety of 1D and 2D flow problems. The adaptation is performed using r-adaptation to move the mesh nodes in the domain in an attempt to reduce the truncation error since it is the local source of discretization error. Here, we present a new set of r-adaptation methods called mesh optimization along with three different ways of performing this type of adaptation. Each of these techniques uses a finite difference gradient-based local optimization technique with different sets of design variables to create a mesh that minimizes a functional based on truncation error. These new truncation error based mesh optimization techniques are compared to a more common truncation error based mesh equidistribution technique. Some observations on the performance and behavior of the different adaptation methods and best practices for their use are presented. All of the optimization methods are shown to reduce the truncation error one or two orders of magnitude and the discretization error by roughly one order of magnitude for the 1D problems tested. In two dimensions, the optimization-based adaptation methods are able to reduce the discretization error by up to a factor of seven. Mesh equidistribution achieved similar levels of improvement for much less cost compared to the mesh optimization techniques.

Novel strategies for modal-based structural material identification

In this work, we present modal-based methods for model calibration in structural dynamics, and address several key challenges in the solution of gradient-based optimization problems with eigenvalues and eigenvectors, including the solution of singular Helmholtz problems encountered in sensitivity calculations, non-differentiable objective functions caused by mode swapping during optimization, and cases with repeated eigenvalues. Unlike previous literature that relied on direct solution of the eigenvector adjoint equations, we present a parallel iterative domain decomposition strategy (Adjoint Computation via Modal Superposition with Truncation Augmentation) for the solution of the singular Helmholtz problems. For problems with repeated eigenvalues we present a novel Mode Separation via Projection algorithm, and in order to address mode swapping between inverse iterations we present a novel Injective mode ordering metric. We present the implementation of these methods in a massively parallel finite element framework with the ability to use measured modal data to extract unknown structural model parameters from large complex problems. A series of increasingly complex numerical examples are presented that demonstrate the implementation and performance of the methods in a massively parallel finite element framework [7,5], using gradient-based optimization techniques in the Rapid Optimization Library (ROL) [21].

The Potential for Tidal Range Energy Systems to Provide Continuous Power: A UK Case Study

The extraction of tidal energy from head differences represents a predictable and flexible option for generating electricity. Here, we investigate the generation potential of prospective tidal power plants in the UK. Originally conceived as separate projects, operating these schemes as a cooperative system could prove beneficial. Combined with the inherent operational flexibility of tidal range-based schemes, a notable tidal phase difference in selected sites allows for the system to spread power generation over a larger proportion of the day. Using depth-averaged modelling and gradient-based optimisation techniques, we explore how a flexible cumulative operation schedule could be applied to provide a degree of continuous supply if desirable. While fully continuous operation is not achieved, a number of different optimisation schedules deliver cumulative continuous supply for over half of the year. The average minimum cumulative power output on these days is consistently over 500 MW out of a total installed capacity of 6195.3 MW. Furthermore, by introducing financial incentives associated with reliable, baseload supply, we provide an economic assessment of the tidal power plant system. The daily minimum cumulative power output determines income in the modelled idealised baseload market, while excess supply is traded in an hourly variable wholesale energy market. Results indicate that subsidies would be required in order to make a pursuit of continuous generation financially advantageous over energy maximisation strategies.

Minimizing control volatility for nonlinear systems with smooth piecewise-quadratic input signals

Abstract We consider a class of nonlinear optimal control problems in which the aim is to minimize control variation subject to an upper bound on the system cost. This idea of sacrificing some cost in exchange for less control volatility—thereby making the control signal easier and safer to implement—is explored in only a handful of papers in the literature, and then mainly for piecewise-constant (discontinuous) controls. Here we consider the case of smooth continuously differentiable controls, which are more suitable in some applications, including robotics and motion control. In general, the control signal’s total variation—the objective to be minimized in the optimal control problem—cannot be expressed in closed form. Thus, we introduce a smooth piecewise-quadratic discretization scheme and derive an analytical expression, which turns out to be rational and non-smooth, for computing the total variation of the approximate piecewise-quadratic control. This leads to a non-smooth dynamic optimization problem in which the decision variables are the knot points and shape parameters for the approximate control. We then prove that this non-smooth problem can be transformed into an equivalent smooth problem, which is readily solvable using gradient-based numerical optimization techniques. The paper includes a numerical example to verify the proposed approach.

Shape optimization of the X0-specimen: theory, numerical simulation and experimental verification

The paper deals with the gradient based shape optimization of the biaxial X0-specimen, which has been introduced and examined in various papers, under producibility restrictions and the related experimental verification. The original, engineering based design of the X0-specimen has been applied successfully to different loading conditions persisting the question if relevant stress states could be reached by optimizing the geometry. Specimens with the initial as well as with the two load case dependent optimized geometries have been fabricated of the aluminum alloy sheets (AlSi1MgMn; EN AW 6082-T6) and tested. The strain fields in critical regions of the specimens have been recorded by digital image correlation technique. In addition, scanning electron microscope analysis of the fracture surfaces clearly indicate the stress state dependent damage processes. Consequently, the presented gradient based optimization technique facilitated significant improvements to study the damage and fracture processes in a more purposeful way.

Efficient FEM-Based EM Optimization Technique Using Combined Lagrangian Method With Newton’s Method

Gradient-based optimization algorithms are popularly used in electromagnetic (EM)-based design optimizations. Among the gradient-based optimization algorithms, Newton’s method is not practically applicable to EM optimization because it is very time consuming to obtain the Hessian matrix containing the second-order derivatives of the EM responses with respect to the geometrical variables. This article addresses this situation and proposes an efficient gradient-based EM optimization technique using the combined Lagrangian method with Newton’s method. EM optimizations can be reformulated into constrained optimizations when the finite element method (FEM) is applied to perform EM simulations. In this article, we propose to elevate the Lagrangian method (i.e., a popular constrained optimization method) to EM optimization. By using the Lagrangian method to perform the EM optimization, the Hessian matrix can be obtained efficiently without the time-consuming evaluations of second-order derivatives of the EM responses with respect to the geometrical variables. With the efficiently calculated the Hessian matrix, Newton’s method can be applied. We derive new formulations of Newton’s method specifically for the EM optimization with the Lagrangian method. The proposed EM optimization using the combined Lagrangian method with Newton’s method can converge faster than direct EM optimizations with other gradient-based optimization methods. The proposed technique is demonstrated by two EM optimization examples of microwave components.