Gradient Of Objective Function Research Articles

Distributed Event-Triggered Algorithm Designs for Resource Allocation Problems via a Universal Scalar Function-Based Analysis.

In this article, we are concerned with distributed algorithm designs for resource allocation problems via event-triggered communication. The target is to search an optimal resource allocation scheme such that the summation of objective functions is minimized. Due to communication efficiency and privacy concerns, distributed algorithms with event-triggered communications are proposed in this article. The communication is only permitted or triggered if variation of gradient of the local objective function exceeds a threshold. By constructing a novel technical lemma and a universal scalar function, the convergence and linear convergence rates are established under some mild assumptions. Extensive numerical experiments on the IEEE 118-bus power system demonstrate that: Compared to the periodic algorithms, such as ADMM and Mirror-P-EXTRA, the proposed algorithms not only remarkably reduce the communication times but also have competitive convergence speed. The latter is striking that it implies there exist useless communications in the periodic algorithms that are censored by the proposed event-triggered strategy.

Gradient-Based Empirical Risk Minimization Using Local Polynomial Regression

In this paper, we consider the widely studied problem of empirical risk minimization (ERM) of strongly convex and smooth loss functions using iterative gradient-based methods. A major goal of the existing literature has been to compare different prototypical algorithms, such as batch gradient descent (GD) or stochastic gradient descent (SGD), by analyzing their rates of convergence to ϵ-approximate solutions with respect to the number of gradient computations, which is also known as the oracle complexity. For example, the oracle complexity of GD is [Formula: see text], where n is the number of training samples and p is the parameter space dimension. When n is large, this can be prohibitively expensive in practice, and SGD is preferred due to its oracle complexity of [Formula: see text]. Such standard analyses only utilize the smoothness of the loss function in the parameter being optimized. In contrast, we demonstrate that when the loss function is smooth in the data, we can learn the oracle at every iteration and beat the oracle complexities of GD, SGD, and their variants in important regimes. Specifically, at every iteration, our proposed algorithm, Local Polynomial Interpolation-based Gradient Descent (LPI-GD), first performs local polynomial regression with a virtual batch of data points to learn the gradient of the loss function and then estimates the true gradient of the ERM objective function. We establish that the oracle complexity of LPI-GD is [Formula: see text], where d is the data space dimension, and the gradient of the loss function is assumed to belong to an η-Hölder class with respect to the data. Our proof extends the analysis of local polynomial regression in nonparametric statistics to provide supremum norm guarantees for interpolation in multivariate settings and also exploits tools from the inexact GD literature. Unlike the complexities of GD and SGD, the complexity of our method depends on d. However, our algorithm outperforms GD, SGD, and their variants in oracle complexity for a broad range of settings where d is small relative to n. For example, with typical loss functions (such as squared or cross-entropy loss), when [Formula: see text] for any [Formula: see text] and [Formula: see text] is at the statistical limit, our method can be made to require [Formula: see text] oracle calls for any [Formula: see text], while SGD and GD require [Formula: see text] and [Formula: see text] oracle calls, respectively. Funding: This work was supported in part by the Office of Naval Research [Grant N000142012394], in part by the Army Research Office [Multidisciplinary University Research Initiative Grant W911NF-19-1-0217], and in part by the National Science Foundation [Transdisciplinary Research In Principles Of Data Science, Foundations of Data Science].

Three-dimensional finite-memory quasi-Newton inversion of the magnetotelluric based on unstructured grids

Abstract Simulation optimization of complex geological bodies is a necessary means to improve inversion accuracy and computational efficiency; thus, inversion of magnetotelluric (MT) based on unstructured grids has become a research hotspot in recent years. This article realizes the three-dimensional (3D) finite element forward modeling of MT based on the magnetic vector potential-electric scalar potential method, using unstructured grids as the forward modeling grid, which improves computational efficiency. The inversion uses the limited-memory Broyden–Fletcher–Goldfarb–Shanno (LBFGS) method, and in the process of calculating the objective function gradient, the quasi-forward method is used to avoid solving the Jacobian matrix, which has the advantages of requiring small storage space and fast computational efficiency. Finally, the 3D LBFGS inversion algorithm of MT based on unstructured grids was realized, and the inversion studies of classic and complex models verified the effectiveness and the reliability of the algorithm proposed in this article.

A Second-Order Finite-Difference Method for Derivative-Free Optimization

In this paper, a second-order finite-difference method is proposed for finding the second-order stationary point of derivative-free nonconvex unconstrained optimization problems. The forward-difference or the central-difference technique is used to approximate the gradient and Hessian matrix of objective function, respectively. The traditional trust-region framework is used, and we minimize the approximation trust region subproblem to obtain the search direction. The global convergence of the algorithm is given without the fully quadratic assumption. Numerical results show the effectiveness of the algorithm using the forward-difference and central-difference approximations.

ЗАДАЧА ОПТИМАЛЬНОГО КЕРУВАННЯ ТОЧКАМИ НАВАНТАЖУВАННЯ ТА ЇХНІМИ ФУНКЦІЯМИ РЕАКЦІЇ ДЛЯ ПАРАБОЛІЧНОГО РІВНЯННЯ

We consider the problem of optimal control of loading points and the corresponding reaction functions described by a loaded parabolic equation. Optimality conditions for control actions are obtained. The objective functional gradient formulas contained in these conditions are used in the algorithm for numerically solving the problem of optimization of loading points and reaction functions based on first-order optimization methods. The results of numerical experiments are provided. Keywords: distributed-parameter system, loaded differential equation, necessary optimality condition, functional gradient.

Layout and cross-sectional size optimization of truss structures with mixed design variables based on gradient method

The object of the study was truss-type rod structures, which were investigated for the purpose of finding the optimal design solution in a mixed (continuous and discrete) space of variables. The parameters of the geometric scheme of the truss, as well as the dimensions of the cross-sections of its elements, were considered as design variables. The stated optimization problem is represented as a nonlinear programming task, in which the objective function and nonlinear constraints of the mathematical model are continuously differentiable functions of the design variables. The system of constraints includes strength and stability inequalities, formulated for the design cross-sections of the rod elements of the structure, which is subject to the effect of the design load combinations of the ultimate limit states. As a part of the system of constraints, the displacement constraints formulated for the specified structural nodes, which is subject to the action of design load combinations of the serviceability limit states, are considered. To solve the stated optimization problem, a method of the objective function gradient on the surface of active constraints was used, with the simultaneous elimination of residuals in the violated restrictions. For design variables, the variation of which must be performed according to a given set of possible discrete values, a discretization procedure for the optimal solution obtained in the continuous space of design variables is proposed. A comparison of the proposed optimization methodology with alternative metaheuristic methods and algorithms reported in the literature was performed. On the considered problem of parametric optimization of a 47-span tower structure, a design solution with a weight of 835,403 kg was obtained, which is 1.53...4.6 % better than the optimal solutions obtained by other authors

Investigation of influence of objective function valley ratio on the determination error of its minimum coordinates

Objectives. A valley is a region of an objective function landscape in which the function varies along one direction more slowly than along other directions. In order to determine the error of the objective function minimum location in such regions, it is necessary to analyze relations of valley parameters.Methods. A special test function was used in numerical experiments to model valleys with variables across wide ranges of parameters. The position and other valley parameters were defined randomly. Valley dimensionality and ratio were estimated from eigenvalues of the approximated Hessian of objective function in the termination point of minimum search. The error was defined as the Euclidian distance between the known minimum position and the minimum search termination point. Linear regression analysis and approximation with an artificial neural network model were used for statistical processing of experimental data.Results. A linear relation of logarithm of valley ratio to logarithm of minimum position error was obtained. Here, the determination coefficient R2 was ~0.88. By additionally taking into account the Euclidian norm of the objective function gradient in the termination point, R2 can be augmented to ~0.95. However, by using the artificial neural network model, an approximation R2 ~ 0.97 was achieved.Conclusions. The obtained relations may be used for estimating the expected error of extremum coordinates in optimization problems. The described method can be extended to functions having a valley dimensionality of more than one and to other types of hard-to-optimize algorithms regions of objective function landscapes.

An accelerated computational platform for optimal field developments with reduced footprint

Reducing the environmental footprint and increasing efficiency of hydrocarbon production as well as geological sequestration of carbon dioxide (CO2) through optimally selected and operated fewer wells is crucial for cleaner and more sustainable energy transition. We describe a parallel subsurface field-development optimization (SFDO) platform with novel components. SFDO enables the optimization of well-locations/-trajectories (WLO), controls (WCO), and both jointly (WLO–WCO). All objective function computations are performed in parallel at a given iteration in SFDO.The novel distributed quasi-Newton (DQN) optimizer is an important new element in accelerating SFDO. DQN is developed using a multi-threaded optimization paradigm that utilizes high-performance computing (HPC) to distribute an ensemble of optimization threads. By operating its threads in parallel within a single optimization run, DQN efficiently identifies multiple distinct local optima of the objective function. DQN tracks and updates a common input-output (training) data set by recording the responses from successful simulation runs. The support-vector regression (SVR) method is used to approximate the Hessian matrix while objective function gradients are estimated analytically using SVR-computed sensitivities.The DQN implementation inside SFDO is extensively tested and validated in this work. We first validate DQN using a synthetic test case with a pre-computed solution. DQN incurs fewest number of simulations, and thus, the shortest run time while successfully finding the global optimum. For the first time in the literature, we show that DQN is more effective than alternative methods in problems involving WCO and WLO–WCO. Then, we report on DQN's performance against alternative techniques in four realistic applications, namely on a WLO problem for environmental footprint reduction and a novel WLO–WCO application for the geological sequestration of CO2 involving a novel take on WLO with deforming well trajectories. Finally, we describe two realistic WLO problems in which the well count is also optimized in addition to the well locations. One of these applications is for heater-well pattern optimization for an innovative unconventional thermal recovery method. DQN finds local optima by incurring a significantly lower computational cost in comparison to alternative methods. Results of the field tests support the advantageous qualities of DQN previously observed in synthetic tests. We conclude that SFDO accelerated with DQN helps increase the efficiency and reduce the footprint of realistic field-scale developments.

A single timescale stochastic quasi-Newton method for stochastic optimization

In this paper, we propose a single timescale stochastic quasi-Newton method for solving the stochastic optimization problems. The objective function of the problem is a composition of two smooth functions and their derivatives are not available. The algorithm sets to approximate sequences to estimate the gradient of the composite objective function and the inner function. The matrix correction parameters are given in BFGS update form for avoiding the assumption that Hessian matrix of objective is positive definite. We show the global convergence of the algorithm. The algorithm achieves the complexity to find an approximate stationary point and ensure that the expectation of the squared norm of the gradient is smaller than the given accuracy tolerance ϵ. The numerical results of nonconvex binary classification problem using the support vector machine and a multicall classification problem using neural networks are reported to show the effectiveness of the algorithm.

Hierarchical Nash equilibrium seeking strategies of quadratic time‐varying games with Euler–Lagrange players

AbstractThis article investigates the distributed Nash equilibrium seeking problem of quadratic time‐varying games with Euler–Lagrange (EL) players, where external disturbances and parametric uncertainties are involved. A gradient‐based hierarchical algorithm consisting of a game layer and a control layer is proposed. Specifically, in the game layer, EL players communicate with neighbors through a graph to reach the consensus on potential aggregate values, which will be employed to calculate the gradient of each player's objective function, and then, a gradient‐based sliding mode controller is developed to track time‐varying gradient in the control layer. Thus, the convergence results are hierarchically obtained through the Lyapunov stability method. In addition, the hierarchical control strategy is extended to address the constrained problems through the utilization of a smooth penalty function. By appropriately choosing control parameters, the Nash equilibrium seeking errors can be arbitrarily small. The relation between the optimal solutions of the original problem and the dual one is further discussed. Finally, the proposed methods are numerically verified.

A discrete adjoint framework coupled with adaptive PCE for robust aerodynamic optimization of turbomachinery under flow uncertainty

Flow uncertainty is commonly encountered in turbomachinery. To mitigate the negative effects caused by the flow uncertainty, a framework coupled with adaptive polynomial chaos expansion (PCE) and discrete adjoint method for robust aerodynamic design optimization (RADO) of turbomachinery blades is developed. The optimization framework relies on gradient algorithms, which effectively avoids “dimensional curse” problem. Adaptive PCE method improves the efficiency of uncertainty quantification and only once PCE model need to be constructed in per optimization step, making the framework highly efficient. The PCE matrix and chain rule are employed to determine the gradient of the RADO objective function. This framework is employed for optimizing turbine blade, considering both inviscid and viscous flow. Monte Carlo simulation and finite difference PCE are used to validate the reliability of uncertainty quantification and gradient propagation methods. A comparative analysis of the optimization results obtained by both RADO and deterministic aerodynamic design optimization (DADO) is demonstrated. The higher robustness brought by RADO indicates the practicality of this framework in robust optimization of turbomachinery blades.

Augmented Lagrangian index-3 semi-recursive formulations with projections

AbstractSensitivity analysis represents a powerful tool for the optimization of multibody system dynamics. The performance of a gradient-based optimization algorithm is strongly tied to the dynamic and the sensitivity formulations considered. The accuracy and efficiency are critical to any optimization problem, thus they are key factors in the selection of the dynamic and sensitivity analysis approaches used to compute an objective function gradient. Semi-recursive methods usually outperform global methods in terms of computational time, even though they involve sometimes demanding recursive procedures. Semi-recursive methods are well suited to be combined with different constraints enforcement schemes as the augmented Lagrangian index-3 formulation with velocity and acceleration projections (ALI3-P), taking advantage of the robustness, accurate fulfillment of constraint equations and the low computational burden. The sensitivity analysis of the semi-recursive ALI3-P formulation is studied in this document by means of the direct differentiation method. As a result, a semi-recursive ALI3-P sensitivity formulation is developed for an arbitrary reference point selection, and then two particular versions are unfolded and implemented in the general purpose multibody library MBSLIM, using as reference point the center of mass (RTdyn0) or the global origin of coordinates (RTdyn1). Besides, the detailed derivatives of the recursive terms are provided, which will be useful not only for the direct sensitivity formulation presented herein, but also for other sensitivity formulations relying on the same recursive expressions. The implementation has been tested in two numerical experiments, a five-bar benchmark problem and a buggy vehicle.

Aeroacoustic and Aerodynamic Adjoint-Based Shape Optimization of an Axisymmetric Aero-Engine Intake

A continuous adjoint-based aeroacoustic optimization, based on a hybrid model including the Ffowcs Williams–Hawkings (FW–H) acoustic analogy, to account for the multidisciplinary design of aero-engine intakes with an axisymmetric geometry, is presented. To optimize such an intake, the generatrix of its lips is parameterized using B-Splines, and the energy contained in the sound pressure spectrum, at the blade passing frequency at receivers located axisymmetrically around the axis of the engine, is minimized. The engine is not included in the optimization and manifests its presence through an independently computed time-series of static pressure over the annular boundary of the simulation domain that corresponds to the inlet to the fan. Taking advantage of the case axisymmetry, the steady 3D RANS equations are solved in the rotating frame of reference and post-processed to compute the flow quantities’ time-series required by the FW–H analogy. The numerical solution of the unsteady flow equations and the otherwise excessive overall cost of the optimization are, thus, avoided. The objective function gradient is computed using the continuous adjoint method, coupled with the analytical differentiation of the FW–H analogy. The adjoint equations are also solved in the rotating frame via steady solver.

Parametric optimization of bar structures with discrete and continuous design variables using improved gradient projection method

The paper considers a parametric optimization problem for the bar structures formulated as nonlinear programming task, where the purpose function and non-linear constraints of the mathematical model are continuously differentiable functions of the design variables. The method of the objective function gradient projection onto the active constraints surface with simultaneous correction of the constraints violations has been used to solve the parametric optimization problem. A discretization technique for the design variables that should vary discretely has been proposed. The discretization of the optimal design solution obtained in the continuous space of the design variables is performed by the purposefully selecting discrete points around the point of the continuous optimum. The comparison of the optimization results presented by the paper demonstrates that improved gradient method together with proposed discretization technique for the discrete design variables converges to better solutions of the problem comparing to the meta-heuristic algorithms.

An Early-Stop Adjoint Transient Sensitivity Analysis Method for Objective Functions Associated with Many Time Points

Transient sensitivity analysis aims to obtain the gradients of objective functions (circuit performance) with respect to design or variation parameters in a simulator, which can be widely used in yield analysis and circuit optimization, among others. However, the traditional method has a computational complexity of <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$O(N^{2})$</tex> for objective functions containing circuit states at <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$N$</tex> time points. The computational complexity is too expensive for large <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$N$</tex> , especially in time-frequency transform. This paper proposes a many-time-point sensitivity method to reduce the computational complexity to <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$O(N)$</tex> in multiparameter many-time-point cases. The paper demonstrates a derivation process that improves efficiency by weighting the transfer chain and multiplexing the backpropagation process. We also proposed an early-stop method to improve efficiency further under the premise of ensuring accuracy. The algorithm enables sensitivity calculation of performances involving thousands of time points, such as signal-to-noise and distortion ratio and total harmonic distortion, with significant speed improvements.

Constant Power Psychoacoustic Spectrum Optimization for Loudness and Sharpness with Application to Vehicle Interiors

&lt;div&gt;Sound quality assessments are an integral part of vehicle design. Especially now, as manufacturers move towards electrification, vehicle sounds are fundamentally changing. By improving the quality of the interior sounds of a vehicle, consumers’ subjective evaluation of it can be increased. Therefore, the field of psychoacoustics, which is the study of human perception of sound, is broadly applicable here. In fact, the perceived quality of a sound signal is influenced by several psychoacoustic indicators, including loudness, sharpness, and roughness. Of particular utility is identifying in advance how to distribute audible frequency content in a way that optimizes psychoacoustic metrics as this can help automotive engineers obtain specific design targets that optimize vehicle noise, vibration, and harshness (NVH).&lt;/div&gt; &lt;div&gt;In this article, a novel modified gradient-based optimization technique (MGOT) is developed to optimize psychoacoustic loudness and sharpness. The new technique is applied to identify targeted adjustments to a measured vehicle interior sound signal that keep the signal energy constant but reduce loudness and/or sharpness. The MGOT numerically approximates the objective function gradient for small changes in the signal power distribution for which constant overall signal power is maintained. These gradient calculations identify power spectrum one-third octave band trades that minimize a sound signal metric that is a weighted sum of loudness and sharpness while conserving the total signal power. A trade consists of a reduction of power content from a one-third octave band designated as a source together with a simultaneous addition of that power to another receiver one-third octave band. In the MGOT, a one-third octave band that is at any time identified as a source can never later become a receiver of power. The MGOT results and execution times are compared with two widely available general-purpose optimization routines (a standard gradient-based optimizer and a “genetic,” non-gradient optimizer) are used to achieve identical optimization objectives. In comparison to existing optimization techniques, MGOT is found to identify spectrum modifications that produce a superior minimization of the objective function for comparable or even reduced execution times. The resultant sound spectrum modifications can guide vehicle structural or calibration design recommendations that realize a preferred frequency distribution for enhancing the vehicle driving experience.&lt;/div&gt;

Airfoil inverse design based on laminar compressible adjoint lattice Boltzmann method

AbstractA new optimization techniques based on the adjoint lattice Boltzmann method is derived for airfoil inverse design in laminar compressible flows. In this study, the developed adjoint lattice Boltzmann scheme based on the circular function (CF) is extended for airfoil inverse design problems in laminar incompressible and compressible flows. New mathematical derivation based on compressible lattice Boltzmann equations (LBE) is developed which can find target shape of an airfoil with available desired pressure distribution. The adjoint lattice Boltzmann method is extended for both the incompressible and compressible flows by selecting the circular function idea for calculating the equilibrium distribution functions. So, the adjoint equation is also expanded based on CF idea for calculation of objective function gradient vector. The steepest decent technique is utilized as gradient optimizer. Also, a novel solution is presented to remove singularity problem of the adjoint boundary condition. In order to validate the developed optimization algorithm, results are presented for both incompressible and compressible inverse problem in steady and unsteady flow and accurate results are obtained.

Adaptive sampling strategies for risk-averse stochastic optimization with constraints

Abstract We introduce adaptive sampling methods for stochastic programs with deterministic constraints. First, we propose and analyze a variant of the stochastic projected gradient method, where the sample size used to approximate the reduced gradient is determined on-the-fly and updated adaptively. This method is applicable to a broad class of expectation-based risk measures, and leads to a significant reduction in the individual gradient evaluations used to estimate the objective function gradient. Numerical experiments with expected risk minimization and conditional value-at-risk minimization support this conclusion, and demonstrate practical performance and efficacy for both risk-neutral and risk-averse problems. Second, we propose an SQP-type method based on similar adaptive sampling principles. The benefits of this method are demonstrated in a simplified engineering design application, featuring risk-averse shape optimization of a steel shell structure subject to uncertain loading conditions and model uncertainty.

Gradient Approximation and Multivariable Derivative-Free Optimization Based on Noncommutative Maps

In this article, multivariable derivative-free optimization algorithms for unconstrained optimization problems are developed. A novel procedure for approximating the gradient of multivariable objective functions based on noncommutative maps is introduced. The procedure is based on the construction of an exploration sequence to specify where the objective function is evaluated and the definition of so-called gradient generating functions which are composed with the objective function, such that the procedure mimics a gradient descent algorithm. Various theoretical properties of the proposed class of algorithms are investigated and numerical examples are presented.

Fixed‐time distributed optimization for multi‐agent systems with external disturbances over directed networks

AbstractThis article considers the fixed‐time distributed optimization problem of multi‐agent systems with external disturbances, in which the global optimization objective is a convex combination of local objective functions. To solve this issue, a directed communication network is carefully designed, and an integral sliding mode control protocol is proposed based on the gradient of global objective function first. Moreover, two distributed optimal protocols are designed by using the gradient and the Hessian matrix of local objective function, respectively. By employing Lyapunov stability theory, graph theory, convex analysis, and inequality techniques, we prove that all proposed protocols can make agents achieve consensus and converge accurately to the optimal solution of the considered problem in some fixed‐time intervals. Finally, some numerical simulations are given to verify the feasibility of the theoretical results.