Derivative-free Method Research Articles

Higher order Traub–Steffensen type methods and their convergence analysis in Banach spaces

Abstract In this paper, we consider two-step fourth-order and three-step sixth-order derivative free iterative methods and study their convergence in Banach spaces to approximate a locally-unique solution of nonlinear equations. Study of convergence analysis provides radius of convergence, error bounds and estimates on the uniqueness of the solution. Such estimates are not provided in the approaches that use Taylor expansions using higher order derivatives. Furthermore, in quest of fast algorithms, a generalized q-step scheme with increasing convergence order 2q + 2 is proposed and analyzed. Novelty of the q-step algorithm is that, in each step, order of convergence is increased by an amount of two at the cost of only one additional function evaluation. To maximize the computational efficiency, the optimal number of steps is calculated. Theoretical results regarding convergence and computational efficiency are verified through numerical experimentation.

Model-Based Derivative-Free Methods for Convex-Constrained Optimization

We present a model-based derivative-free method for optimization subject to general convex constraints, which we assume are unrelaxable and accessed only through a projection operator that is cheap to evaluate. We prove global convergence and a worst-case complexity of $O(\epsilon^{-2})$ iterations and objective evaluations for nonconvex functions, matching results for the unconstrained case. We introduce new, weaker requirements on model accuracy compared to existing methods. As a result, sufficiently accurate interpolation models can be constructed only using feasible points. We develop a comprehensive theory of interpolation set management in this regime for linear and composite linear models. We implement our approach for nonlinear least-squares problems and demonstrate strong practical performance compared to general-purpose solvers.

A derivative-free memoryless BFGS hyperplane projection method for solving large-scale nonlinear monotone equations

A derivative-free memoryless BFGS hyperplane projection method for solving large-scale nonlinear monotone equations

A derivative-free Milstein type approximation method for SPDEs covering the non-commutative noise case

We propose a derivative-free Milstein type scheme to approximate the mild solution of stochastic partial differential equations (SPDEs) that do not need to fulfill a commutativity condition for the noise term. The newly developed derivative-free Milstein type scheme differs significantly from schemes that are appropriate for the case of commutative noise. As a key result, the new derivative-free Milstein type scheme needs only two stages that are specifically tailored based on a technique that, compared to the original Milstein scheme, allows for a reduction of the computational complexity by one order of magnitude. Moreover, the proposed derivative-free Milstein scheme can flexibly be combined with some approximation method for the involved iterated stochastic integrals. As the main result, we prove the strong L^2-convergence of the introduced derivative-free Milstein type scheme, especially if it is combined with any suitable approximation algorithm for the necessary iterated stochastic integrals. We carry out a rigorous analysis of the error versus computational cost and derive the effective order of convergence for the derivative-free Milstein type scheme in the case that the truncated Fourier series algorithm for the approximation of the iterated stochastic integrals is applied. As a further novelty, we show that the use of approximations of iterated stochastic integrals based on truncated Fourier series together with the proposed derivative-free Milstein type scheme improves the effective order of convergence compared to that of the Euler scheme and the original Milstein scheme. This result is contrary to well known results in the finite dimensional SDE case where the use of merely truncated Fourier series does not improve the effective order of convergence in the L^2-sense compared to that of the Euler scheme.

A class of three-term derivative-free methods for large-scale nonlinear monotone system of equations and applications to image restoration problems

A class of three-term derivative-free methods for large-scale nonlinear monotone system of equations and applications to image restoration problems

Unifying information propagation models on networks and influence maximization.

Information propagation on networks is a central theme in social, behavioral, and economic sciences, with important theoretical and practical implications, such as the influence maximization problem for viral marketing. Here we consider a model that unifies the classical independent cascade models and the linear threshold models, and generalize them by considering continuous variables and allowing feedback in the dynamics. We then formulate its influence maximization as a mixed integer nonlinear programming problem and adopt derivative-free methods. Furthermore, we show that the problem can be exactly solved in the special case of linear dynamics, where the selection criterion is closely related to the Katz centrality, and propose a customized direct search method with local convergence. We then demonstrate the close to optimal performance of the customized direct search numerically on both synthetic and real networks.

A family of inertial derivative-free projection methods for constrained nonlinear pseudo-monotone equations with applications

A family of inertial derivative-free projection methods for constrained nonlinear pseudo-monotone equations with applications

Gradient flow structure and convergence analysis of the ensemble Kalman inversion for nonlinear forward models

The ensemble Kalman inversion (EKI) is a particle based method which has been introduced as the application of the ensemble Kalman filter to inverse problems. In practice it has been widely used as derivative-free optimization method in order to estimate unknown parameters from noisy measurement data. For linear forward models the EKI can be viewed as gradient flow preconditioned by a certain sample covariance matrix. Through the preconditioning the resulting scheme remains in a finite dimensional subspace of the original high-dimensional (or even infinite dimensional) parameter space and can be viewed as optimizer restricted to this subspace. For general nonlinear forward models the resulting EKI flow can only be viewed as gradient flow in approximation. In this paper we discuss the effect of applying a sample covariance as preconditioning matrix and quantify the gradient flow structure of the EKI by controlling the approximation error through the spread in the particle system. The ensemble collapse on the one side leads to an accurate gradient approximation, but on the other side to degeneration in the preconditioning sample covariance matrix. In order to ensure convergence as optimization method we derive lower as well as upper bounds on the ensemble collapse. Furthermore, we introduce covariance inflation without breaking the subspace property intending to reduce the collapse rate of the ensemble such that the convergence rate improves. In a numerical experiment we apply EKI to a nonlinear elliptic boundary-value problem and illustrate the dependence of EKI as derivative-free optimizer on the choice of the initial ensemble.

Periodic solutions of nonlinear ordinary differential equations computed by a boundary shape function method and a generalized derivative-free Newton method

In the paper, the period of an n-dimensional nonlinear dynamical system is computed by a formula derived in an (n+1)-dimensional augmented state space. The periodic conditions and nonlinear first-order ordinary differential equations constitute a specific periodic boundary value problem within a time interval, whose length is an unknown finite constant. Two periodic problems are considered: (I) boundary values are given and (II) boundary values are unknown. A boundary shape function method (BSFM), using the derived shape functions, is devised to an initial value problem with the initial values of new variables given, whereas the terminal values and period are determined by iterative algorithms. The periodic solutions obtained by the BSFM satisfy the periodic conditions automatically. For the sake of comparison, the iterative algorithms based on the shooting method are developed, directly implementing the Poincaré map with the fictitious time integration method to determine the periodic solutions, where the periodic conditions are transformed to a mathematically equivalent scalar equation. Owing to the implicit, non-differentiable and nonlinear property of the scalar equation, we develop a generalized derivative-free Newton method (GDFNM) to solve the periodic problem of case (I), which can pick up very accurate period through a few iterations. In numerical examples the computed order of convergence displays the merit of the proposed iterative algorithms. The BSFM and GDFNM are better than the shooting method from the aspects of convergence speed, accuracy and stability. A conventional Poincaré mapping method is introduced to solve the periodic problems with the same parameters. The BSFM converges faster and more accurate than the Poincaré mapping method and is less sensitive to the initial guesses of initial values and period.

Taking Care of the Discretization Problem: A Comprehensive Study of the Discretization Problem and a Black-Box Adversarial Attack in Discrete Integer Domain

Neural network based (NN-based) classifiers are known vulnerable against adversarial examples, namely, adding slight perturbations to a benign image cause a classifier to make a false prediction. To evaluate the robustness of NN-based classifiers against adversarial examples, numerous adversarial attacks with high success rates have been proposed recently. NN-based image classifiers usually normalize valid images (e.g., RGB image where the value at each coordinate is an integer between 0 and 255) into a real continuous domain (e.g., 3-dimensional matrix where the value at each coordinate is a real number between 0 and 1) and make classification decisions on the normalized images. However, adversarial examples crafted in a real continuous domain may become benign once they are denormalized back into the corresponding discrete integer domain, known as the discretization problem. This problem has been mentioned in some prior works but received relatively limited attention. In this work, we report the first comprehensive study of existing works to understand the impacts of the discretization problem. By analyzing 35 representative methods and empirically studying 20 representative open source tools, we found 29/35 (theoretically) and 14/20 (empirically) are affected by the discretization problem, e.g., the success rate could dramatically drop from 100 to 10 percent after the domain transformation. As the first step towards addressing this problem in a black-box scenario, we propose a novel derivative-free optimization method, which can directly craft adversarial examples in the discrete integer domain. Experimental results show that the method achieves nearly 100 percent attack success rates for both targeted and untargeted attacks, comparable to the most popular white-box methods (FGSM, BIM and C&W), and significantly outperforms representative black-box methods (ZOO, AutoZOOM, NES-PGD, Bandits, FD, FD-PSO and GenAttack). Our results suggest that the discretization problem should be treated more seriously, and the discrete optimization algorithms show a promising future in crafting effective black-box attacks.

Field-Development Optimization Method Benchmarked, Field Tested

_ This article, written by JPT Technology Editor Chris Carpenter, contains highlights of paper SPE 206267, “Benchmarking and Field Testing of the Distributed Quasi-Newton Derivative-Free Optimization Method for Field Development Optimization,” by Faruk Alpak, SPE, Yixuan Wang, SPE, and Guohua Gao, SPE, Shell, et al. The paper has not been peer reviewed. _ Recently, a novel distributed quasi-Newton (DQN) derivative-free optimization (DFO) method was developed for generic reservoir-performance optimization problems, including well-location optimization (WLO) and well-control optimization (WCO). DQN is designed to locate multiple local optima of highly nonlinear optimization problems effectively. However, its performance has been neither validated by realistic applications nor compared with other DFO methods. Field-testing results reinforce the auspicious computational attributes of DQN. Background An optimization problem is posed as the minimization or maximization of an objective function by modifying the control variables (x) within a search domain. The objective function is a highly nonlinear function of x and may have multiple local optima. In this paper, the objective function is assumed to be twice differentiable. DFO methods can be classified into local search methods and global search methods. In the complete paper, the authors’ goal is to locate multiple local optima of the objective function. Their focus is on local search DFO optimization methods, which include direct search methods and model-based methods. Current DFO methods reviewed in the complete paper have a common feature: Only one best approximation to the solution is updated in each iteration, and only one optimal solution is identified in the last iteration. Therefore, they are referred to as single-thread DFO methods. They do not represent an efficient approach because simulation results obtained by one optimization task starting from one initial guess are not shared with other optimization tasks that start from different initial guesses. The distributed Gauss-Newton method and the DQN method benchmarked in the paper are referred to as multiple-thread optimization methods. The authors have integrated DQN into a versatile field-development optimization platform designed specifically for iterative work flows enabled through distributed-parallel flow simulations.

Square-root cubature Kalman filter based power system dynamic state estimation

The state estimator’s real-time state information plays a vital role in monitoring and control of the power system. As a result, one of the primary requirements of the estimator is to provide accurate state estimates with good convergence under varied system operating conditions. In this context, dynamic state estimation (DSE) consistently tracks the power system dynamics and delivers real-time system state evolution. In literature, this dynamic state estimation (SE) problem is generally solved using an extended Kalman filter (EKF). The EKF approach neglects the non-linear mathematical functions of network equations by linearly approximating the measurement equations. This leads to degradation of SE accuracy during a sudden change in load conditions. In this paper, a new derivative-free method based on square-root cubature Kalman filter (SCKF) is proposed to overcome the limitations of the EKF technique and achieves improved performance. The proposed SCKF technique has been tested on IEEE 14-bus, IEEE 30-bus, and IEEE 57-bus systems under various operating conditions. The effectiveness of the SCKF technique has been compared with the traditional EKF and cubature Kalman filter (CKF) techniques using different performance indices. The obtained results demonstrate the versatility of the proposed SCKF technique.

A family of inertial-relaxed DFPM-based algorithms for solving large-scale monotone nonlinear equations with application to sparse signal restoration

In this paper, combining the inertial-relaxed technique and the Armijo line search technique, we propose a family of inertial-relaxed derivative-free projection methods (DFPMs) for large-scale monotone nonlinear equations. The global convergence of the proposed family is established without the Lipschitz continuity of the underlying mapping. To the best of our knowledge, this is the first convergence result for embedding the inertial-relaxed technique into DFPMs for solving monotone nonlinear equations. Moreover, we propose two inertial-relaxed DFPM-based algorithms with convergence guarantee by embedding two specific search directions into the family. The numerical results on standard monotone nonlinear equations show that our proposed methods are efficient and competitive. Finally, we illustrate the applicability and encouraging efficiency of the proposed methods via applying them to solve sparse signal restoration.

Anisotropic Diffusion in Consensus-Based Optimization on the Sphere

In this paper we are concerned with the global minimization of a possibly non-smooth and non-convex objective function constrained on the unit hypersphere by means of a multi-agent derivative-free method. The proposed algorithm falls into the class of the recently introduced Consensus-Based Optimization. In fact, agents move on the sphere driven by a drift towards an instantaneous consensus point, which is computed as a convex combination of agent locations, weighted by the cost function according to Laplaces principle, and it represents an approximation to a global minimizer. The dynamics is further perturbed by an anisotropic random vector field to favor exploration. The main results of this paper are about the proof of convergence of the numerical scheme to global minimizers provided conditions of well-preparation of the initial datum. The proof of convergence combines a mean-field limit result with a novel asymptotic analysis, and classical convergence results of numerical methods for SDE. The main innovation with respect to previous work is the introduction of an anisotropic stochastic term, which allows us to ensure the independence of the parameters of the algorithm from the dimension and to scale the method to work in very high dimension. We present several numerical experiments, which show that the algorithm proposed in the present paper is extremely versatile and outperforms previous formulations with isotropic stochastic noise.

A Derivative-Free MZPRP Projection Method for Convex Constrained Nonlinear Equations and Its Application in Compressive Sensing

Nonlinear systems of equations are widely used in science and engineering and, therefore, exploring efficient ways to solve them is paramount. In this paper, a new derivative-free approach for solving a nonlinear system of equations with convex constraints is proposed. The search direction of the proposed method is derived based on a modified conjugate gradient method, in such a way that it is sufficiently descent. It is worth noting that, unlike many existing methods that require a monotonicity assumption to prove the convergence result, our new method needs the underlying function to be pseudomonotone, which is a weaker assumption. The performance of the proposed algorithm is demonstrated on a set of some test problems and applications arising from compressive sensing. The obtained results confirm that the proposed method is effective compared to some existing algorithms in the literature.

Complete vehicle energy management using adaptive primal–dual operator splitting

Complete vehicle energy management (CVEM) aims at minimizing the energy consumption of all subsystems inside a vehicle. When the subsystems consist of energy buffers with linear dynamics and/or power converters with quadratic power losses, the CVEM becomes a large-scale and ill-conditioned optimal control problem for which well-known solution approaches (e.g. Pontryagin’s maximum principle, dynamic programming, convex and derivative-free optimization methods) are not effective. In this paper, we design an efficient solution algorithm for this problem by leveraging the framework of operator splitting methods. To facilitate its implementation, we propose a spectral method that automatically tunes the step sizes at every iteration and, in practice, accelerates convergence. Finally, we consider a case study for a parallel-hybrid vehicle and we illustrate the numerical performance of our method by showing that we can efficiently solve a CVEM problem with very large horizons.

A derivative-free scaling memoryless DFP method for solving large scale nonlinear monotone equations

A derivative-free scaling memoryless DFP method for solving large scale nonlinear monotone equations

A globally convergent derivative-free projection algorithm for signal processing

In this paper, a hybrid derivative-free conjugate gradient method that inherits the structures of two conjugate gradient methods is introduced to recover sparse signal in compressive sensing by solving the nonlinear convex constrained equations. The global convergence of the proposed method is proved, under some appropriate assumptions. Numerical experiments and comparisons suggest that the proposed algorithm is an efficient approach for sparse signal and image reconstruction in compressive sensing.

Deep reinforcement learning of viscous incompressible flow

We develop a new computational method to solve viscous incompressible fluid flow. This method is an extension of the Derivative-Free Loss Method, which solves elliptic and parabolic partial differential equations using Brownian motion, the Feynman-Kac formula, and reinforcement learning. Herein we derive a martingale condition and directly incorporate a projection of the velocity to solve the incompressible Navier-Stokes equations. Our method is mesh-free, scalable, and flexible, as demonstrated with several test cases.

Numerical Solution for the Single-Impulse Flyby Co-Orbital Spacecraft Problem

The traversal inspection of satellites in satellite constellations or geosynchronous orbits has been a focus of research. A large number of variable orbit requirements in the “single-to-single” mode severely affects the efficiency of inspections. To address this problem, this study investigated the problem of a single-impulse flyby co-orbiting two spacecraft and proposed a derivative-free numerical solution method that used the geometric relationship between the two intersections of the target and transfer orbits of the flyby problem in order to transform them into a nonlinear equation in a single variable for a given impulse time. The validity of the proposed method was verified using numerical examples. While the Lambert problem is one of the bases for solving the variable orbit problem, on-star intelligent control also raises the requirements for speed. To address this problem, this study also investigated the Lambert problem in a single-impulse flyby co-orbiting two spacecraft and determined the iterative initial value by constructing a quadratic interpolation equation between the inverse of the transfer time and the vertical component of the eccentric vector, the derivative-free quadratic interpolation cut-off method was proposed. Using 100,000 random tests showed that computational efficiency was improved by more than one order of magnitude compared with commonly used methods, with a calculation error of less than 10−6.