Zeroth-order Methods Research Articles

Highly Smooth Zeroth-Order Methods for Solving Optimization Problems under the PL Condition

Highly Smooth Zeroth-Order Methods for Solving Optimization Problems under the PL Condition

Nonconvex Zeroth-Order Stochastic ADMM Methods with Lower Function Query Complexity.

Zeroth-order (a.k.a, derivative-free) methods are a class of effective optimization methods for solving complex machine learning problems, where gradients of the objective functions are not available or computationally prohibitive. Recently, although many zeroth-order methods have been developed, these approaches still have two main drawbacks: 1) high function query complexity; 2) not being well suitable for solving the problems with complex penalties and constraints. To address these challenging drawbacks, in this paper, we propose a class of faster zeroth-order stochastic alternating direction method of multipliers (ADMM) methods (ZO-SPIDER-ADMM) to solve the nonconvex finite-sum problems with multiple nonsmooth penalties. Moreover, we prove that the ZO-SPIDER-ADMM methods can achieve a lower function query complexity of [Formula: see text] for finding an ϵ-stationary point, which improves the existing best nonconvex zeroth-order ADMM methods by a factor of [Formula: see text], where n and d denote the sample size and data dimension, respectively. At the same time, we propose a class of faster zeroth-order online ADMM methods (ZOO-ADMM+) to solve the nonconvex online problems with multiple nonsmooth penalties. We also prove that the proposed ZOO-ADMM+ methods achieve a lower function query complexity of [Formula: see text], which improves the existing best result by a factor of [Formula: see text]. Extensive experimental results on the structure adversarial attack on black-box deep neural networks demonstrate the efficiency of our new algorithms.

Zeroth-Order Methods for Online Distributed Optimization with Strongly Pseudoconvex Cost Functions

Zeroth-Order Methods for Online Distributed Optimization with Strongly Pseudoconvex Cost Functions

ZO-JADE: Zeroth-Order Curvature-Aware Distributed Multi-Agent Convex Optimization

In this work we address the problem of convex optimization in a multi-agent setting where the objective is to minimize the mean of local cost functions whose derivatives are not available (e.g. black-box models). Moreover agents can only communicate with local neighbors according to a connected network topology. Zeroth-order (ZO) optimization has recently gained increasing attention in federated learning and multi-agent scenarios exploiting finite-difference approximations of the gradient using from $2$ (directional gradient) to $2d$ (central difference full gradient) evaluations of the cost functions, where $d$ is the dimension of the problem. The contribution of this work is to extend ZO distributed optimization by estimating the curvature of the local cost functions via finite-difference approximations. In particular, we propose a novel algorithm named ZO-JADE, that by adding just one extra point, i.e. $2d+1$ in total, allows to simultaneously estimate the gradient and the diagonal of the local Hessian, which are then combined via average tracking consensus to obtain an approximated Jacobi descent. Guarantees of semi-global exponential stability are established via separation of time-scales. Extensive numerical experiments on real-world data confirm the efficiency and superiority of our algorithm with respect to several other distributed zeroth-order methods available in the literature based on only gradient estimates.

Distributed Evolution Strategies for Black-Box Stochastic Optimization

This work concerns the evolutionary approaches to distributed stochastic black-box optimization, in which each worker can individually solve an approximation of the problem with nature-inspired algorithms. We propose a distributed evolution strategy (DES) algorithm grounded on a proper modification to evolution strategies, a family of classic evolutionary algorithms, as well as a careful combination with existing distributed frameworks. On smooth and nonconvex landscapes, DES has a convergence rate competitive to existing zeroth-order methods, and can exploit the sparsity, if applicable, to match the rate of first-order methods. The DES method uses a Gaussian probability model to guide the search and avoids the numerical issue resulted from finite-difference techniques in existing zeroth-order methods. The DES method is also fully adaptive to the problem landscape, as its convergence is guaranteed with any parameter setting. We further propose two alternative sampling schemes which significantly improve the sampling efficiency while leading to similar performance. Simulation studies on several machine learning problems suggest that the proposed methods show much promise in reducing the convergence time and improving the robustness to parameter settings.

Fair and Energy-Efficient Coverage Optimization for UAV Placement Problem in the Cellular Network

Unmanned Aerial Vehicle (UAV) Base Station (BS) placement optimization is an essential operational task to improve the Quality of Service (QoS) in UAV-aided wireless cellular networks. The existing approaches are almost zeroth order methods, and the few first order methods mainly ignore the allocation fairness, computational efficiency, and backhaul constraints. In this paper, we formulate the UAV placement problem as a constrained optimization problem, with the objective of maximizing the fair coverage versus energy consumption while satisfying the backhaul constraints at different time nodes. To guarantee fair QoS allocation, we introduce a novel fairness index to ensure fair communication opportunity and the novel region coverage ratio to avoid excess QoS on covered spots. An accurate and efficient proximal stochastic gradient descent based alternating algorithm that iteratively executes two optimization steps is proposed to optimize the UAV locations, which enables the fast single point-based first order methods to solve the complex problems with constraints. Experiment results manifest that the proposed algorithm performs well both in synthetic data scenario and in real city scenario. Furthermore, the proposed first order algorithm is more efficient than the existing zeroth order algorithm, typically referring to the meta-heuristic method.

Distributed Randomized Gradient-Free Mirror Descent Algorithm for Constrained Optimization

This article is concerned with the multiagent optimization problem. A distributed randomized gradient-free mirror descent (DRGFMD) method is developed by introducing a randomized gradient-free oracle in the mirror descent scheme where the non-Euclidean Bregman divergence is used. The classical gradient descent method is generalized without using subgradient information of objective functions. The proposed algorithms are the first distributed non-Euclidean zeroth-order methods, which achieve an approximate <inline-formula><tex-math notation="LaTeX">$O(\frac{1}{\sqrt{T}})$</tex-math></inline-formula> <inline-formula><tex-math notation="LaTeX">$T$</tex-math></inline-formula>-rate of convergence, recovering the best known optimal rate of distributed nonsmooth constrained convex optimization. Moreover, a decentralized reciprocal weighted averaging (RWA) approximating sequence is first investigated, the convergence for RWA sequence is shown to hold over time-varying graph. Rates of convergence are comprehensively explored for the algorithm with RWA (DRGFMD-RWA). The technique on constructing the decentralized RWA sequence provides new insight in searching for minimizers in distributed algorithms.

Zeroth-order methods for noisy Hölder-gradient functions

In this paper, we prove new complexity bounds for zeroth-order methods in non-convex optimization with inexact observations of the objective function values. We use the Gaussian smoothing approach of Nesterov and Spokoiny(Found Comput Math 17(2): 527–566, 2015. https://doi.org/10.1007/s10208-015-9296-2) and extend their results, obtained for optimization methods for smooth zeroth-order non-convex problems, to the setting of minimization of functions with Hölder-continuous gradient with noisy zeroth-order oracle, obtaining noise upper-bounds as well. We consider finite-difference gradient approximation based on normally distributed random Gaussian vectors and prove that gradient descent scheme based on this approximation converges to the stationary point of the smoothed function. We also consider convergence to the stationary point of the original (not smoothed) function and obtain bounds on the number of steps of the algorithm for making the norm of its gradient small. Additionally we provide bounds for the level of noise in the zeroth-order oracle for which it is still possible to guarantee that the above bounds hold. We also consider separately the case of \(\nu = 1\) and show that in this case the dependence of the obtained bounds on the dimension can be improved.

Metrology of Nanostructures by Tomographic Mueller-Matrix Scatterometry

The development of necessary instrumentation and metrology at the nanoscale, especially fast, low-cost, and nondestructive metrology techniques, is of great significance for the realization of reliable and repeatable nanomanufacturing. In this work, we present the application of a homemade novel optical scatterometer called the tomographic Mueller-matrix scatterometer (TMS), for the measurement of photoresist gratings. The TMS adopts a dual rotating-compensator configuration and illuminates the nanostructure sequentially under test conditions by a plane wave, with varying illumination directions and records. For each illumination direction, the polarized scattered field along various directions of observation can be seen in the form of scattering Mueller matrices. That more scattering information is collected by TMS than conventional optical scatterometry ensures that it achieves better measurement sensitivity and accuracy. We also show the capability of TMS for determining both grating pitch and other structural parameters, which is incapable by current zeroth-order methods such as reflectometry- or ellipsometry-based scatterometry.

Stochastic First- and Zeroth-Order Methods for Nonconvex Stochastic Programming

In this paper, we introduce a new stochastic approximation type algorithm, namely, the randomized stochastic gradient (RSG) method, for solving an important class of nonlinear (possibly nonconvex) stochastic programming problems. We establish the complexity of this method for computing an approximate stationary point of a nonlinear programming problem. We also show that this method possesses a nearly optimal rate of convergence if the problem is convex. We discuss a variant of the algorithm which consists of applying a postoptimization phase to evaluate a short list of solutions generated by several independent runs of the RSG method, and we show that such modification allows us to improve significantly the large-deviation properties of the algorithm. These methods are then specialized for solving a class of simulation-based optimization problems in which only stochastic zeroth-order information is available.

Optimum Design of Yagi–Uda Antennas Using Computational Intelligence

Optimization of Yagi-Uda antennas is a challenging design problem, since the antenna characteristics such as gain, input impedance, maximum sidelobe level etc., are known to be extremely sensitive to the design variables viz., element lengths and their spacings. This corresponds to a highly nonlinear and multimodal function space with functional and slope discontinuities that limit the use of conventional gradient based optimization approaches. Although, stochastic, zeroth-order methods like genetic algorithm and evolutionary algorithm are attractive choices for such classes of problems, their successful application requires scaling and aggregating parameters to handle constraints and objectives that may not be easy to provide. In this paper, we introduce a stochastic, zeroth-order optimization algorithm that handles constraints and objectives separately via Pareto ranking that eliminates the problem of scaling and aggregation. The algorithm is based on principles of learning and is embedded with three key learning strategies that control whom to learn from (i.e., leader identification and leader selection) and what to learn (i.e., information acquisition) in order to better guide the search. The leader identification mechanism partitions the individuals into a set of leaders and a set of followers. The followers interact with the leader and move toward the better performing leaders in search for better solutions. As the algorithm does not require parameters for scaling or aggregation, it provides the designer the true flexibility that is necessary to handle various forms of the design problem effectively and at a computational cost that is comparable to existing stochastic optimization methods. Results of three single objective antenna design examples (a four-element, a 15-element and a fixed boom length 22-element design) are presented and compared with published results to illustrate the behavior of the proposed algorithm and highlight its benefits in solving a wide variety of antenna design problems. A new set of results are presented for a multiobjective formulation of the design problem.

Noniterative algorithms for finding quantum optimal controls

Iterative methods are generally necessary for solving the design equations to identify optimal quantum controls. Since iteration can be computationally intensive, it is significant to develop good approximate noniterative methods. In this paper we present noniterative techniques for achieving quantum optimal control over the expectation value of positive semidefinite operators. The noniterative methods are characterized by an order index. Zeroth-order methods involving no feedback from the objective are generally found to be inadequate. A proposed first-order noniterative algorithm is expected to often be a good approximation. Numerical tests verify the noniterative capabilities of the algorithm.