Bound-constrained Optimization Problems Research Articles

Derivative-free bound-constrained optimization for solving structured problems with surrogate models

We propose and analyze a model-based derivative-free (DFO) algorithm for solving bound-constrained optimization problems where the objective function is the composition of a smooth function and a vector of black-box functions. We assume that the black-box functions are smooth and the evaluation of them is the computational bottleneck of the algorithm. The distinguishing feature of our algorithm is the use of approximate function values at interpolation points which can be obtained by an application-specific surrogate model that is cheap to evaluate. As an example, we consider the situation in which a sequence of related optimization problems is solved and present a regression-based approximation scheme that uses function values that were evaluated when solving prior problem instances. In addition, we propose and analyze a new algorithm for obtaining interpolation points that handles unrelaxable bound constraints. Our numerical results show that our algorithm outperforms a state-of-the-art DFO algorithm for solving a least-squares problem from a chemical engineering application when a history of black-box function evaluations is available.

Discrete Army Ant Search Optimizer-Based Target Coverage Enhancement in Directional Sensor Networks

Coverage of interest points is one of the most critical issues in directional sensor networks. However, considering the remote or inhospitable environment and the limitation of the perspective of directional sensors, it is easy to form perception blind after random deployment. The intension of our research is to deal with the bound-constrained optimization problem of maximizing the coverage of target points. A coverage enhancement strategy based on a discrete army ant search optimizer (DAASO) is proposed to solve the above problem, which is inspired by the biological habits of army ants. A set of experiments are conducted using different sensor parameters. Experimental results verify the effectiveness of the DAASO in coverage effect when compared to the existing methods.

Exploiting Problem Structure in Derivative Free Optimization

A structured version of derivative-free random pattern search optimization algorithms is introduced, which is able to exploit coordinate partially separable structure (typically associated with sparsity) often present in unconstrained and bound-constrained optimization problems. This technique improves performance by orders of magnitude and makes it possible to solve large problems that otherwise are totally intractable by other derivative-free methods. A library of interpolation-based modelling tools is also described, which can be associated with the structured or unstructured versions of the initial pattern search algorithm. The use of the library further enhances performance, especially when associated with structure. The significant gains in performance associated with these two techniques are illustrated using a new freely-available release of the Brute Force Optimizer (BFO) package firstly introduced in [Porcelli and Toint 2017 ], which incorporates them. An interesting conclusion of the numerical results presented is that providing global structural information on a problem can result in significantly less evaluations of the objective function than attempting to building local Taylor-like models.

An Active Set Trust-Region Method for Bound-Constrained Optimization

This paper discusses an active set trust-region algorithm for bound-constrained optimization problems. A sufficient descent condition is used as a computational measure to identify whether the function value is reduced or not. To get our complexity result, a critical measure is used which is computationally better than the other known critical measures. Under the positive definiteness of approximated Hessian matrices restricted to the subspace of non-active variables, it will be shown that unlimited zigzagging cannot occur. It is shown that our algorithm is competitive in comparison with the state-of-the-art solvers for solving an ill-conditioned bound-constrained least-squares problem.

Local Convergence Analysis of a Primal–Dual Method for Bound-Constrained Optimization Without SOSC

We propose a local convergence analysis of a primal–dual interior point algorithm for the solution of a bound-constrained optimization problem. The algorithm includes a regularization technique to prevent singularity of the matrix of the linear system at each iteration, when the second-order sufficient conditions do not hold at the solution. These conditions are replaced by a milder assumption related to a local error-bound condition. This new condition is a generalization of the one used in unconstrained optimization. We show that by an appropriate updating strategy of the barrier parameter and of the regularization parameter, the proposed algorithm owns a superlinear rate of convergence. The analysis is made thanks to a boundedness property of the inverse of the Jacobian matrix arising in interior point algorithms. An illustrative example is given to show the behavior and the gain obtained by this regularization strategy.

Evolutionary Framework With Reinforcement Learning-Based Mutation Adaptation

Although several multi-operator and multi-method approaches for solving optimization problems have been proposed, their performances are not consistent for a wide range of optimization problems. Also, the task of ensuring the appropriate selection of algorithms and operators may be inefficient since their designs are undertaken mainly through trial and error. This research proposes an improved optimization framework that uses the benefits of multiple algorithms, namely, a multi-operator differential evolution algorithm and a co-variance matrix adaptation evolution strategy. In the former, reinforcement learning is used to automatically choose the best differential evolution operator. To judge the performance of the proposed framework, three benchmark sets of bound-constrained optimization problems (73 problems) with 10, 30 and 50 dimensions are solved. Further, the proposed algorithm has been tested by solving optimization problems with 100 dimensions taken from CEC2014 and CEC2017 benchmark problems. A real-world application data set has also been solved. Several experiments are designed to analyze the effects of different components of the proposed framework, with the best variant compared with a number of state-of-the-art algorithms. The experimental results show that the proposed algorithm is able to outperform all the others considered.

A class of Benders decomposition methods for variational inequalities

We develop new variants of Benders decomposition methods for variational inequality problems. The construction is done by applying the general class of Dantzig–Wolfe decomposition of Luna et al. (Math Program 143(1–2):177–209, 2014) to an appropriately defined dual of the given variational inequality, and then passing back to the primal space. As compared to previous decomposition techniques of the Benders kind for variational inequalities, the following improvements are obtained. Instead of rather specific single-valued monotone mappings, the framework includes a rather broad class of multi-valued maximally monotone ones, and single-valued nonmonotone. Subproblems’ solvability is guaranteed instead of assumed, and approximations of the subproblems’ mapping are allowed (which may lead, in particular, to further decomposition of subproblems, which may otherwise be not possible). In addition, with a certain suitably chosen approximation, variational inequality subproblems become simple bound-constrained optimization problems, thus easier to solve.

An Efficient Nonmonotone Method for State-Constrained Elliptic Optimal Control Problems

This paper presents a novel numerical strategy based on combination of an adaptive semismooth Newton (ASN) method and the Lavrentiev regularization technique for the solution of elliptic optimal control problems with state constraints. Using the global convergence proof for a nonmonotone semismooth Newton method, we will exploit an adaptive nonmonotone line search method such that the nonmonotonicity degree of this method can be increased when the results are far from the optimum solution and it can be reduced when they are close to the optimizer. In this strategy, the role of the Lavrentiev regularization technique is converting the original optimal control problem to a regularized optimal control problem. Using the finite difference discretization scheme and a Newton–Cotes rule, the regularized optimal control problem is converted to a bound constrained optimization problem (BCOP). Then the ASN method is implemented to solve the resulting BCOP. Numerical results show the efficiency of the proposed procedure.

A novel meta-heuristic algorithm: Dynamic Virtual Bats Algorithm

Nature-inspired algorithms are a very important part of meta-heuristics. A novel nature inspired algorithm called the Dynamic Virtual Bats Algorithm (DVBA) is presented in this paper. DVBA is inspired by a bat’s ability to manipulate frequency and wavelength of the emitted sound waves when hunting. A role based search has been developed to improve the diversification and intensification capability of Bat Algorithm. In the DVBA, there are only two bats: explorer and exploiter bat. While the explorer bat explores the search space, the exploiter bat makes an intensive search of the local with the highest probability of locating the desired target. Depending on their location, bats exchange the roles dynamically.The performance of the DVBA is extensively evaluated on a suite of 30 bound-constrained optimization problems from CEC 2014 and compared favorably with other well-known meta-heuristics algorithms. The experimental results demonstrated that the proposed DVBA outperform, or is comparable to, its competitors in terms of the quality of final solution and its convergence rates.

An augmented Lagrangian interior-point approach for large-scale NLP problems on graphics processing units

The demand for fast solution of nonlinear optimization problems, coupled with the emergence of new concurrent computing architectures, drives the need for parallel algorithms to solve challenging nonlinear programming (NLP) problems. In this paper, we propose an augmented Lagrangian interior-point approach for general NLP problems that solves in parallel on a Graphics processing unit (GPU). The algorithm is iterative at three levels. The first level replaces the original problem by a sequence of bound-constrained optimization problems using an augmented Lagrangian method. Each of these bound-constrained problems is solved using a nonlinear interior-point method. Inside the interior-point method, the barrier sub-problems are solved using a variation of Newton's method, where the linear system is solved using a preconditioned conjugate gradient (PCG) method, which is implemented efficiently on a GPU in parallel. This algorithm shows an order of magnitude speedup on several test problems from the COPS test set.

A scalable algorithm for MAP estimators in Bayesian inverse problems with Besov priors

We present a scalable solver for approximating the maximum a posteriori (MAP) point of Bayesian inverse problems with Besov priors based on wavelet expansions with random coefficients. It is a subspace trust region interior reflective Newton conjugate gradient method for bound constrained optimization problems. The method combines the rapid locally-quadratic convergence rate properties of Newton's method, the effectiveness of trust region globalization for treating ill-conditioned problems, and the Eisenstat--Walker idea of preventing oversolving. We demonstrate the scalability of the proposed method on two inverse problems: a deconvolution problem and a coefficient inverse problem governed by elliptic partial differential equations. The numerical results show that the number of Newton iterations is independent of the number of wavelet coefficients $n$ and the computation time scales linearly in $n$. It will be numerically shown, under our implementations, that the proposed solver is two times faster than the split Bregman approach, and it is an order of magnitude less expensive than the interior path following primal-dual method. Our results also confirm the fact that the Besov $\mathbb{B}_{11}^1$ prior is sparsity promoting, discretization-invariant, and edge-preserving for both imaging and inverse problems governed by partial differential equations.

An efficient interval computing technique for bound-constrained uncertain optimization problems

In this article, a competent interval-oriented approach is proposed to solve bound-constrained uncertain optimization problems. This new class of problems is considered here as an extension of the classical bound-constrained optimization problems in an inexact environment. The proposed technique is nothing but an imitation of the well-known interval analysis-based branch-and-bound optimization approach. Efficiency of this technique is strongly dependent on division, bounding, selection/rejection and termination criteria. The technique involves a multisection division criterion of the accepted/proposed search region. Then, we have employed the interval-ranking definitions with respect to the pessimistic decision makers’ point of view given by Mahato and Bhunia [Interval-arithmetic-oriented interval computing technique for global optimization, Appl. Math. Res. Express 2006 (2006), pp. 1–19] to compare the interval-valued objectives calculated in each subregion and also to select the subregion containing the best interval objective value. The process is continued until the interval width for each variable in the accepted subregion is negligible and ultimately the global or close-to-global interval-valued optimal solution is obtained. The proposed technique has been evaluated numerically using a wide set of newly introduced univariate/multivariate test problems. Finally, to compare the computational results obtained by the proposed method, the graphical representation for some test problems is given.

Derivative-free optimization: a review of algorithms and comparison of software implementations

AbstractThis paper addresses the solution of bound-constrained optimization problems using algorithms that require only the availability of objective function values but no derivative information. We refer to these algorithms as derivative-free algorithms. Fueled by a growing number of applications in science and engineering, the development of derivative-free optimization algorithms has long been studied, and it has found renewed interest in recent time. Along with many derivative-free algorithms, many software implementations have also appeared. The paper presents a review of derivative-free algorithms, followed by a systematic comparison of 22 related implementations using a test set of 502 problems. The test bed includes convex and nonconvex problems, smooth as well as nonsmooth problems. The algorithms were tested under the same conditions and ranked under several criteria, including their ability to find near-global solutions for nonconvex problems, improve a given starting point, and refine a near-optimal solution. A total of 112,448 problem instances were solved. We find that the ability of all these solvers to obtain good solutions diminishes with increasing problem size. For the problems used in this study, , , and are better, on average, than other derivative-free solvers in terms of solution quality within 2,500 function evaluations. These global solvers outperform local solvers even for convex problems. Finally, , , and show superior performance in terms of refining a near-optimal solution.

Stopping rules and backward error analysis for bound-constrained optimization

Termination criteria for the iterative solution of bound-constrained optimization problems are examined in the light of backward error analysis. It is shown that the problem of determining a suitable perturbation on the problem’s data corresponding to the definition of the backward error is analytically solvable under mild assumptions. Moreover, a link between existing termination criteria and this solution is clarified, indicating that some standard measures of criticality may be interpreted in the sense of backward error analysis. The backward error problem is finally considered from the multicriteria optimization point of view and some numerical illustration is provided.

A local-best harmony search algorithm with dynamic subpopulations

This article presents a local-best harmony search algorithm with dynamic subpopulations (DLHS) for solving the bound-constrained continuous optimization problems. Unlike existing harmony search algorithms, the DLHS algorithm divides the whole harmony memory (HM) into many small-sized sub-HMs and the evolution is performed in each sub-HM independently. To maintain the diversity of the population and to improve the accuracy of the final solution, information exchange among the sub-HMs is achieved by using a periodic regrouping schedule. Furthermore, a novel harmony improvisation scheme is employed to benefit from good information captured in the local best harmony vector. In addition, an adaptive strategy is developed to adjust the parameters to suit the particular problems or the particular phases of search process. Extensive computational simulations and comparisons are carried out by employing a set of 16 benchmark problems from the literature. The computational results show that, overall, the proposed DLHS algorithm is more effective or at least competitive in finding near-optimal solutions compared with state-of-the-art harmony search variants.

Projected Pseudotransient Continuation

We propose and analyze a pseudotransient continuation algorithm for dynamics on subsets of $R^N$. Examples include certain flows on manifolds and the dynamic formulation of bound-constrained optimization problems. The method gets its global convergence properties from the dynamics and inherits its local convergence properties from any fast locally convergent iteration.

On the Convergence of Multiplicative Update Algorithms for Nonnegative Matrix Factorization

Nonnegative matrix factorization (NMF) is useful to find basis information of nonnegative data. Currently, multiplicative updates are a simple and popular way to find the factorization. However, for the common NMF approach of minimizing the Euclidean distance between approximate and true values, no proof has shown that multiplicative updates converge to a stationary point of the NMF optimization problem. Stationarity is important as it is a necessary condition of a local minimum. This paper discusses the difficulty of proving the convergence. We propose slight modifications of existing updates and prove their convergence. Techniques invented in this paper may be applied to prove the convergence for other bound-constrained optimization problems.

A coordinate gradient descent method for nonsmooth separable minimization

We consider the problem of minimizing the sum of a smooth function and a separable convex function. This problem includes as special cases bound-constrained optimization and smooth optimization with l1-regularization. We propose a (block) coordinate gradient descent method for solving this class of nonsmooth separable problems. We establish global convergence and, under a local Lipschitzian error bound assumption, linear convergence for this method. The local Lipschitzian error bound holds under assumptions analogous to those for constrained smooth optimization, e.g., the convex function is polyhedral and the smooth function is (nonconvex) quadratic or is the composition of a strongly convex function with a linear mapping. We report numerical experience with solving the l1-regularization of unconstrained optimization problems from More et al. in ACM Trans. Math. Softw. 7, 17–41, 1981 and from the CUTEr set (Gould and Orban in ACM Trans. Math. Softw. 29, 373–394, 2003). Comparison with L-BFGS-B and MINOS, applied to a reformulation of the l1-regularized problem as a bound-constrained optimization problem, is also reported.

A general-purpose tunable landscape generator

The research literature on metaheuristic and evolutionary computation has proposed a large number of algorithms for the solution of challenging real-world optimization problems. It is often not possible to study theoretically the performance of these algorithms unless significant assumptions are made on either the algorithm itself or the problems to which it is applied, or both. As a consequence, metaheuristics are typically evaluated empirically using a set of test problems. Unfortunately, relatively little attention has been given to the development of methodologies and tools for the large-scale empirical evaluation and/or comparison of metaheuristics. In this paper, we propose a landscape (test-problem) generator that can be used to generate optimization problem instances for continuous, bound-constrained optimization problems. The landscape generator is parameterized by a small number of parameters, and the values of these parameters have a direct and intuitive interpretation in terms of the geometric features of the landscapes that they produce. An experimental space is defined over algorithms and problems, via a tuple of parameters for any specified algorithm and problem class (here determined by the landscape generator). An experiment is then clearly specified as a point in this space, in a way that is analogous to other areas of experimental algorithmics, and more generally in experimental design. Experimental results are presented, demonstrating the use of the landscape generator. In particular, we analyze some simple, continuous estimation of distribution algorithms, and gain new insights into the behavior of these algorithms using the landscape generator

Algorithm 856

APPSPACK is software for solving unconstrained and bound-constrained optimization problems. It implements an asynchronous parallel pattern search method that has been specifically designed for problems characterized by expensive function evaluations. Using APPSPACK to solve optimization problems has several advantages: No derivative information is needed; the procedure for evaluating the objective function can be executed via a separate program or script; the code can be run serially or in parallel, regardless of whether the function evaluation itself is parallel; and the software is freely available. We describe the underlying algorithm, data structures, and features of APPSPACK version 4.0, as well as how to use and customize the software.