Derivative-free Method Research Articles

An interior point method for nonlinear constrained derivative-free optimization

In this paper, we consider constrained optimization problems where both the objective and constraint functions are of the black-box type. Furthermore, we assume that the nonlinear inequality constraints are non-relaxable, i.e. the values of the objective function and constraints cannot be computed outside of the feasible region. This situation happens frequently in practice especially in the black-box setting where function values are typically computed by means of complex simulation programs which may fail to execute if the considered point is outside of the feasible region. For such problems, we propose a new derivative-free optimization method which is based on the use of a merit function that handles inequality constraints by means of a log-barrier approach and equality constraints by means of an exterior penalty approach. We prove the convergence of the proposed method to KKT stationary points of the problem under standard assumptions (we do not require any convexity assumption). Furthermore, we also carry out a preliminary numerical experience on standard test problems and comparison with state-of-the-art solvers showing the efficiency of the proposed method.

An accelerated double-step derivative-free projection method based algorithm using Picard-Mann iterative process for solving convex constrained nonlinear equations

An accelerated double-step derivative-free projection method based algorithm using Picard-Mann iterative process for solving convex constrained nonlinear equations

Continual Learning From a Stream of APIs.

Continual learning (CL) aims to learn new tasks without forgetting previous tasks. However, existing CL methods require a large amount of raw data, which is often unavailable due to copyright considerations and privacy risks. Instead, stakeholders usually release pre-trained machine learning models as a service (MLaaS), which users can access via APIs. This paper considers two practical-yet-novel CL settings: data-efficient CL (DECL-APIs) and data-free CL (DFCL-APIs), which achieve CL from a stream of APIs with partial or no raw data. Performing CL under these two new settings faces several challenges: unavailable full raw data, unknown model parameters, heterogeneous models of arbitrary architecture and scale, and catastrophic forgetting of previous APIs. To overcome these issues, we propose a novel data-free cooperative continual distillation learning framework that distills knowledge from a stream of APIs into a CL model by generating pseudo data, just by querying APIs. Specifically, our framework includes two cooperative generators and one CL model, forming their training as an adversarial game. We first use the CL model and the current API as fixed discriminators to train generators via a derivative-free method. Generators adversarially generate hard and diverse synthetic data to maximize the response gap between the CL model and the API. Next, we train the CL model by minimizing the gap between the responses of the CL model and the black-box API on synthetic data, to transfer the API's knowledge to the CL model. Furthermore, we propose a new regularization term based on network similarity to prevent catastrophic forgetting of previous APIs. Our method performs comparably to classic CL with full raw data on the MNIST and SVHN datasets in the DFCL-APIs setting. In the DECL-APIs setting, our method achieves 0.97×, 0.75× and 0.69× performance of classic CL on the more challenging CIFAR10, CIFAR100, and MiniImageNet, respectively.

Bayesian nights: Optimizing night photography rendering with Bayesian derivative-free methods

Bayesian nights: Optimizing night photography rendering with Bayesian derivative-free methods

An inertial hybrid DFPM-based algorithm for constrained nonlinear equations with applications

The derivative-free projection method (DFPM) is an effective and classic approach for solving the system of nonlinear monotone equations with convex constraints, but the global convergence or convergence rate of the DFPM is typically analyzed under the Lipschitz continuity. This observation motivates us to propose an inertial hybrid DFPM-based algorithm, which incorporates a modified conjugate parameter utilizing a hybridized technique, to weaken the convergence assumption. By integrating an improved inertial extrapolation step and the restart procedure into the search direction, the resulting direction satisfies the sufficient descent and trust region properties, which independent of line search choices. Under weaker conditions, we establish the global convergence and Q-linear convergence rate of the proposed algorithm. To the best of our knowledge, this is the first analysis of the Q-linear convergence rate under the condition that the mapping is locally Lipschitz continuous. Finally, by applying the Bayesian hyperparameter optimization technique, a series of numerical experiment results demonstrate that the new algorithm has advantages in solving nonlinear monotone equation systems with convex constraints and handling compressed sensing problems.

A derivative-free projection method with double inertial effects for solving nonlinear equations

Recent research has highlighted the significant performance of multi-step inertial extrapolation in a wide range of algorithmic applications. This paper introduces a derivative-free projection method (DFPM) with a double-inertial extrapolation step for solving large-scale systems of nonlinear equations. The proposed method's global convergence is established under the assumption that the underlying mapping is Lipschitz continuous and satisfies a certain generalized monotonicity assumption (e.g., it can be pseudo-monotone). This is the first convergence result for a DFPM with double inertial step to solve nonlinear equations. Numerical experiments are conducted using well-known test problems to show the proposed method's effectiveness and robustness compared to two existing methods in the literature.

A Novel Noise-Aware Classical Optimizer for Variational Quantum Algorithms

A key component of variational quantum algorithms (VQAs) is the choice of classical optimizer employed to update the parameterization of an ansatz. It is well recognized that quantum algorithms will, for the foreseeable future, necessarily be run on noisy devices with limited fidelities. Thus, the evaluation of an objective function (e.g., the guiding function in the quantum approximate optimization algorithm (QAOA) or the expectation of the electronic Hamiltonian in variational quantum eigensolver (VQE)) required by a classical optimizer is subject not only to stochastic error from estimating an expected value but also to error resulting from intermittent hardware noise. Model-based derivative-free optimization methods have emerged as popular choices of a classical optimizer in the noisy VQA setting, based on empirical studies. However, these optimization methods were not explicitly designed with the consideration of noise. In this work we adapt recent developments from the “noise-aware numerical optimization” literature to these commonly used derivative-free model-based methods. We introduce the key defining characteristics of these novel noise-aware derivative-free model-based methods that separate them from standard model-based methods. We study an implementation of such noise-aware derivative-free model-based methods and compare its performance on demonstrative VQA simulations to classical solvers packaged in scikit-quant. History: Accepted by Giacomo Nannicini, Area Editor for Quantum Computing and Operations Research. Accepted for Special Issue. Funding: This material is based upon work supported by the U.S. Department of Energy, Office of Science, National Quantum Information Science Research Centers and the Office of Advanced Scientific Computing Research, Accelerated Research for Quantum Computing program under contract number DE-AC02-06CH11357. Supplemental Material: The software that supports the findings of this study is available within the paper and its Supplemental Information ( https://pubsonline.informs.org/doi/suppl/10.1287/ijoc.2023.0177 ) as well as from the IJOC GitHub software repository ( https://github.com/INFORMSJoC/2023.0177 ). The complete IJOC Software and Data Repository is available at https://informsjoc.github.io/ .

Efficient structural optimization under transient impact loads using multilayer perceptron and genetic algorithms

Currently, there is a lack of structural optimization methods when a structure is suffering transient impact load, considering nonlinear factors such as material plasticity and strain rate effect. The utilization of derivative-free algorithms also increases computational expenses. Accordingly, this paper presents a novel structure transient optimization approach based on multilayer perceptron combined with a derivative-free optimization method, which can efficiently solve the transient structure dynamics optimization problem with multiple optimization objects. In this approach, the area to be optimized is cut by several closed B-spline curves whose shapes are controlled by several parameters and can be varied and merged at each optimization iteration step. The structure's transient analysis process is replaced by a machine learning based surrogate model, which is trained by thousands of results from the FEM explicit transient simulation. Additionally, nearly orthogonal Latin hypercube sampling is utilized to simplify parameter dimensionality, reduce the training data set, save calculation time, and give the training data set a more comprehensive range of design parameters. In our optimization process, our design target is to minimize the peak reaction force while with the constrain of ensuring enough stiffness and mass. The results demonstrate our proposed methods could efficiently handle transient optimization problems without sensitivity calculations, exhibiting strong generalization capabilities.

Bi-attribute utility preference robust optimization: A continuous piecewise linear approximation approach

In this paper, we consider a bi-attribute decision making problem where the decision maker’s (DM’s) objective is to maximize the expected utility of outcomes with two attributes but where the true utility function which captures the DM’s risk preference is ambiguous. To tackle this ambiguity, we propose a maximin bi-attribute utility preference robust optimization (BUPRO) model where the optimal decision is based on the worst-case utility function in an ambiguity set of plausible utility functions constructed using partially available information such as the DM’s specific preference for certain lotteries. Specifically, we consider a BUPRO model with two attributes, where the DM’s risk attitude is bivariate risk-averse and the ambiguity set is defined by a linear system of inequalities represented by the Lebesgue–Stieltjes integrals of the DM’s utility functions. To solve the inner infinite-dimensional minimization problem, we propose a continuous piecewise linear approximation approach to approximate the DM’s unknown true utility. Unlike the univariate case, we partition the domain of the utility function into a set of small non-overlapping rectangles and then divide each rectangle into two triangles by either the main diagonal (Type-1) or the counter diagonal (Type-2). The inner minimization problem based on the piecewise linear utility function can be reformulated as a mixed-integer linear program and the outer maximization problem can be solved efficiently by the derivative-free method. In the case that all the small triangles are partitioned either in Type-1 or in Type-2, the inner minimization can be formulated as a finite dimensional linear program and the overall maximin as a single mixed-integer program. To quantify the approximation errors, we derive, under some mild conditions, the error bound for the difference between the BUPRO model and the approximate BUPRO model in terms of the ambiguity set, the optimal value and the optimal solutions. Finally, we carry out some numerical tests to examine the performance of the proposed models and computational schemes. The results demonstrate the efficiency of the computational schemes and highlight the stability of the BUPRO model against data perturbations.

A unified approach to the construction of higher-order derivative-free iterative methods for solving systems of nonlinear equations

In this article, we introduce a unified approach to constructing a higher-order derivative-free scheme based on the approximations of F'(zk)-1. A family of order p=6,7 derivative-free method is proposed and compared to some well-known methods. The necessary and sufficient condition for p-th order of convergence are given in terms of parameter matrices τ(k) and α(k) . Some good choices of and are offered. Numerical experiments were carried out to confirm the theoretical results.

A subspace derivative-free projection method for convex constrained nonlinear equations

A subspace derivative-free projection method for convex constrained nonlinear equations

Efficient, multimodal, and derivative-free bayesian inference with Fisher–Rao gradient flows

Abstract In this paper, we study efficient approximate sampling for probability distributions known up to normalization constants. We specifically focus on a problem class arising in Bayesian inference for large-scale inverse problems in science and engineering applications. The computational challenges we address with the proposed methodology are: (i) the need for repeated evaluations of expensive forward models; (ii) the potential existence of multiple modes; and (iii) the fact that gradient of, or adjoint solver for, the forward model might not be feasible.

While existing Bayesian inference methods meet some of these challenges individually, we propose a framework that tackles all three systematically. Our approach builds upon the Fisher-Rao gradient flow in probability space, yielding a dynamical system for probability densities that converges towards the target distribution at a uniform exponential rate. This rapid convergence is advantageous for the computational burden outlined in (i). We apply Gaussian mixture approximations with operator splitting techniques to simulate the flow numerically; the resulting approximation can capture multiple modes thus addressing (ii). Furthermore, we employ the Kalman methodology to facilitate a derivative-free update of these Gaussian components and their respective weights, addressing the issue in (iii).

The proposed methodology results in an efficient derivative-free posterior approximation method, flexible enough to handle multi-modal distributions: Gaussian Mixture Kalman Inversion (GMKI). The effectiveness of GMKI is demonstrated both theoretically and numerically in several experiments with multimodal target distributions, including proof-of-concept and two-dimensional examples, as well as a large-scale application: recovering the Navier-Stokes initial condition from solution data at positive times.



Hybridized Brazilian–Bowein type spectral gradient projection method for constrained nonlinear equations

This paper proposes a hybridized Brazilian and Bowein derivative-free spectral gradient projection method for solving systems of convex-constrained nonlinear equations. The method avoids solving any subproblems in each iteration. Global convergence is established under appropriate assumptions on the functions involved. Additionally, numerical experiments are conducted to evaluate the algorithm’s performance, providing evidence of its efficiency compared to similar algorithms from the existing literature. The results demonstrate that the method outperforms some existing approaches in terms of the number of iterations, function evaluations, and time required to obtain a solution based on the examples considered.

Neural dynamical operator: Continuous spatial-temporal model with gradient-based and derivative-free optimization methods

Data-driven modeling techniques have been explored in the spatial-temporal modeling of complex dynamical systems for many engineering applications. However, a systematic approach is still lacking to leverage the information from different types of data, e.g., with different spatial and temporal resolutions, and the combined use of short-term trajectories and long-term statistics. In this work, we build on the recent progress of neural operator and present a data-driven modeling framework called neural dynamical operator that is continuous in both space and time. A key feature of the neural dynamical operator is the resolution-invariance with respect to both spatial and temporal discretizations, without demanding abundant training data in different temporal resolutions. To improve the long-term performance of the calibrated model, we further propose a hybrid optimization scheme that leverages both gradient-based and derivative-free optimization methods and efficiently trains on both short-term time series and long-term statistics. We investigate the performance of the neural dynamical operator with three numerical examples, including the viscous Burgers' equation, the Navier–Stokes equations, and the Kuramoto–Sivashinsky equation. The results confirm the resolution-invariance of the proposed modeling framework and also demonstrate stable long-term simulations with only short-term time series data. In addition, we show that the proposed model can better predict long-term statistics via the hybrid optimization scheme with a combined use of short-term and long-term data.

A two-step relaxed-inertial derivative-free projection based algorithm for solving standard nonlinear pseudo-monotone equations and logistic regression problems

This paper explores a two-step inertial derivative-free projection method with a relaxation factor γ∈(0,2) for solving nonlinear pseudo-monotone equations. Unlike existing inertial algorithms for the system of nonlinear pseudo-monotone equations, the inertial step of our method involves the current iteration point and the previous two iteration points. In particular, one of the inertial parameters is nonpositive. In the proposed algorithm, the search direction possesses not only the sufficient descent property but also the trust region property, independent of the line search technique. Moreover, we also establish the global convergence and the convergence rate of the algorithm without the Lipschitz continuity of the underlying mapping. Finally, our method provides competitive results on standard nonlinear monotone and pseudo-monotone equations and logistic regression problems compared with two inertial algorithms existing in the literature.

A derivative free projection method for the singularities of vector fields with convex constraints on Hadamard manifolds

ABSTRACT The objective of this paper is to introduce a derivative free projection method designed to find the singularities of pseudomonotone vector fields with convex constraints on Hadamard manifolds. This innovative approach combines the hyperplane projection method with a novel search direction. The global convergence of the proposed method is established under certain conditions. Our method improves some existing results in the literature on Hadamard manifolds. Additionally, illustrative numerical examples are provided to demonstrate the practical efficacy of our method.

A stochastic approach for elliptic problems in perforated domains

A wide range of applications in science and engineering involve a PDE model in a domain with perforations, such as perforated metals or air filters. Solving such perforated domain problems suffers from computational challenges related to resolving the scale imposed by the geometries of perforations. We propose a neural network-based mesh-free approach for perforated domain problems. The method is robust and efficient in capturing various configuration scales, including the averaged macroscopic behavior of the solution that involves a multiscale nature induced by small perforations. The new approach incorporates the derivative-free loss method that uses a stochastic representation or the Feynman-Kac formulation. In particular, we implement the Neumann boundary condition for the derivative-free loss method to handle the interface between the domain and perforations. A suite of stringent numerical tests is provided to support the proposed method's efficacy in handling various perforation scales.

An IDFPM-based algorithm without Lipschitz continuity to constrained nonlinear equations for sparse signal and blurred image restoration problems

The derivative-free projection method (DFPM) is widely used to solve constrained nonlinear equations. To guarantee the convergence of the derivative-free projection method, the mapping should be Lipschitz continuous, which is a strict requirement in theory. Hence, it is interesting to design the new DFPM that possesses nice convergence under weaker theoretical hypothesis. In this paper, we propose an inertial DFPM-based algorithm (named IDFPM), in which the inertial extrapolation step is embedded in the design for the search direction. The global convergence of the proposed algorithm is obtained without the Lipschitz continuity of the mapping. Numerical experiments are carried out for two kinds of problems. The one consists of eight test problems from classical literature and the compressed sensing model. The numerical results show that the proposed algorithm is promising.

Derivative free processes of high order for nondifferentiable equations in Banach spaces

In this article, we consider two parametric classes of derivative free iterative methods of high order of convergence in Banach spaces. We study local and semilocal convergence results in Banach spaces for both families of iterative processes. We carry out a numerical application to solve a nonlinear and nondifferentiable Fredholm integral equation in the Banach space of continuous functions with the maximum norm.

Q-fully quadratic modeling and its application in a random subspace derivative-free method

Q-fully quadratic modeling and its application in a random subspace derivative-free method