Unconstrained Optimization Algorithm Research Articles

The analysis of p-term algorithms for unconstrained optimization

Consider the multidimensional unconstrained minimization problem in a case of continuously differentiable function. An iterative algorithm for solving such a problem is called a multi-term if in order to find the next approximation to the optimal point we need to compute values of the function or its gradient in two or more previous points. So that, conjugate gradient algorithm is a two-term algorithm. The aim of this paper is to study a generalized $p$-term method for unconstrained optimization. One substantiates the properties of this algorithm for quadratic functions and proves that it relates to conjugate direction methods. The goal of computational experiment was to compare the results of minimization with different number of terms $p$ and find the ``optimal'' value for the $p$. The numerical results for some well-known test functions are given.

Forward–backward quasi-Newton methods for nonsmooth optimization problems

The forward-backward splitting method (FBS) for minimizing a nonsmooth composite function can be interpreted as a (variable-metric) gradient method over a continuously differentiable function which we call forward-backward envelope (FBE). This allows to extend algorithms for smooth unconstrained optimization and apply them to nonsmooth (possibly constrained) problems. Since the FBE and its gradient can be computed by simply evaluating forward-backward steps, the resulting methods rely on the very same black-box oracle as FBS. We propose an algorithmic scheme that enjoys the same global convergence properties of FBS when the problem is convex, or when the objective function possesses the Kurdyka-{\L}ojasiewicz property at its critical points. Moreover, when using quasi-Newton directions the proposed method achieves superlinear convergence provided that usual second-order sufficiency conditions on the FBE hold at the limit point of the generated sequence. Such conditions translate into milder requirements on the original function involving generalized second-order differentiability. We show that BFGS fits our framework and that the limited-memory variant L-BFGS is well suited for large-scale problems, greatly outperforming FBS or its accelerated version in practice. The analysis of superlinear convergence is based on an extension of the Dennis and Mor\'e theorem for the proposed algorithmic scheme.

Constrained dynamic multi-objective evolutionary optimization for operational indices of beneficiation process

Operational indices optimization of beneficiation process is a dynamic optimization problem in nature. It is difficult to solve because the related dynamic models of operational indices cannot be achieved easily. Focusing on the operational indices optimization under uncertain environments in production process, this paper first formulates a constrained dynamic multi-objective optimization problem based on the collected data, which considers the changing factors in production and the constraints of operational and production indices, and takes the production indices as optimization objectives and the operational indices as decision variables. To solve the established constrained dynamic multi-objective problem, a prediction with modification mechanism based dynamic multi-objective evolutionary optimization algorithm is proposed. The algorithm first divides the population into several sub-populations and then predicts each sub-population center of new environment independently. New population is generated by Gaussian and uniform distribution based on the estimated centers to improve the convergence speed. At the same time, to ensure the population diversity, a modification strategy is adopted to detect which reference point has no individual associated and produces some individuals around it. The proposed algorithm is applied to solve the dynamic operational indices optimization problem and compared with a constrained and a modified unconstrained dynamic multi-objective optimization algorithm. The statistical results demonstrate the efficiency and effectiveness of the proposed algorithm to solve the real-world dynamic operational indices optimization problem.

Algorithms for unconstrained global optimization of nonlinear (polynomial) programming problems: The single and multi-segment polynomial B-spline approach

We investigate the use of the polynomial B-spline form for unconstrained global optimization of multivariate polynomial nonlinear programming problems. We use the B-spline form for higher order approximation of multivariate polynomials. We first propose a basic algorithm for global optimization that uses several accelerating algorithms such as cut-off test and monotonicity test. We then propose an improved algorithm consisting of several additional ingredients, such as a new subdivision point selection rule and a modified subdivision direction selection rule. The performances of the proposed basic and improved algorithms are tested and compared on a set of 14 test problems under two test conditions. The results of the tests show the superiority of the improved algorithm with multi-segment B-spline over that of the single segment B-spline, in terms of the chosen performance metrics. We also compare the quality of the set of all global minimizers found using the proposed algorithms (basic & improved) with those using well-known solvers BARON and Gloptipoly, on a smaller set of four test problems. The problems in the latter set have multiple global minimizers. The results show the superiority of the proposed algorithms, in that they are able to capture all the global minimizers, whereas Gloptipoly and BARON fail to do so in some of the test problems.

A subspace conjugate gradient algorithm for large-scale unconstrained optimization

In this paper, a subspace three-term conjugate gradient method is proposed. The search directions in the method are generated by minimizing a quadratic approximation of the objective function on a subspace. And they satisfy the descent condition and Dai-Liao conjugacy condition. At each iteration, the subspace is spanned by the current negative gradient and the latest two search directions. Thereby, the dimension of the subspace should be 2 or 3. Under some appropriate assumptions, the global convergence result of the proposed method is established. Numerical experiments show the proposed method is competitive for a set of 80 unconstrained optimization test problems.

A Vector Angle-Based Evolutionary Algorithm for Unconstrained Many-Objective Optimization

Taking both convergence and diversity into consideration, this paper suggests a vector angle-based evolutionary algorithm for unconstrained (with box constraints only) many-objective optimization problems. In the proposed algorithm, the maximum-vector-angle-first principle is used in the environmental selection to guarantee the wideness and uniformity of the solution set. With the help of the worse-elimination principle, worse solutions in terms of the convergence (measured by the sum of normalized objectives) are allowed to be conditionally replaced by other individuals. Therefore, the selection pressure toward the Pareto-optimal front is strengthened. The proposed method is compared with other four state-of-the-art many-objective evolutionary algorithms on a number of unconstrained test problems with up to 15 objectives. The experimental results have shown the competitiveness and effectiveness of our proposed algorithm in keeping a good balance between convergence and diversity. Furthermore, it was shown by the results on two problems from practice (with irregular Pareto fronts) that our method significantly outperforms its competitors in terms of both the convergence and diversity of the obtained solution sets. Notably, the new algorithm has the following good properties: 1) it is free from a set of supplied reference points or weight vectors; 2) it has less algorithmic parameters; and 3) the time complexity of the algorithm is low. Given both good performance and nice properties, the suggested algorithm could be an alternative tool when handling optimization problems with more than three objectives.

A benchmark study on the efficiency of unconstrained optimization algorithms in 2D-aerodynamic shape design

Optimization algorithms are used in various engineering applications to identify optimal shapes. In this work, we benchmark several unconstrained optimization algorithms (Nelder–Mead, Quasi-Newton, steepest descent) under variation of gradient estimation schemes (adjoint equations, finite differences). Flow fields are computed by solving the Reynolds-Averaged Navier–Stokes equations using the open source computational fluid dynamics code OpenFOAM. Design variables vary from N=2 to N=364. The efficiency of the optimization algorithms are benchmarked in terms of: (a) computation time, and (b) applicability and ease of use. Results for lift optimizations are presented for airfoils at a Reynolds number of Re=50,000. As a result, we find for a small number of design variables N≈5 or less, the computational efficiency of all optimization algorithms to be similar, while the ease of use of the Nelder–Mead algorithm makes it a perfect choice for a low number of design variables. For intermediate and large number o...

Constrained Laplacian biogeography-based optimization algorithm

Biogeography-based optimization (BBO) is a relatively new nature inspired optimization technique proposed by Dan Simon for unconstrained optimization, which was later generalized and improved by Happing Ma and Dan Simon for constrained optimization, called blended biogeography-based optimization. In an earlier paper, the authors have proposed a Laplacian biogeography-based optimization algorithm (LX-BBO) for unconstrained optimization. The purpose of the present paper is to generalize the LX-BBO from the unconstrained case to the constrained case. This is done by using the Deb’s constrained handling method. In order to evaluate the performance of the proposed constrained LX-BBO for constrained optimization problems, five different constrained optimization problems and popular CEC 2006 benchmark collection is used. Based on the analysis of results it is shown that the proposed Constrained LX-BBO outperforms Blended BBO for constrained optimization.

Line search fixed point algorithms based on nonlinear conjugate gradient directions: application to constrained smooth convex optimization

This paper considers the fixed point problem for a nonexpansive mapping on a real Hilbert space and proposes novel line search fixed point algorithms to accelerate the search. The termination conditions for the line search are based on the well-known Wolfe conditions that are used to ensure the convergence and stability of unconstrained optimization algorithms. The directions to search for fixed points are generated by using the ideas of the steepest descent direction and conventional nonlinear conjugate gradient directions for unconstrained optimization. We perform convergence as well as convergence rate analyses on the algorithms for solving the fixed point problem under certain assumptions. The main contribution of this paper is to make a concrete response to an issue of constrained smooth convex optimization; that is, whether or not we can devise nonlinear conjugate gradient algorithms to solve constrained smooth convex optimization problems. We show that the proposed fixed point algorithms include ones with nonlinear conjugate gradient directions which can solve constrained smooth convex optimization problems. To illustrate the practicality of the algorithms, we apply them to concrete constrained smooth convex optimization problems, such as constrained quadratic programming problems and generalized convex feasibility problems, and numerically compare them with previous algorithms based on the Krasnosel’skiĭ-Mann fixed point algorithm. The results show that the proposed algorithms dramatically reduce the running time and iterations needed to find optimal solutions to the concrete optimization problems compared with the previous algorithms.

New Conjugate Gradient Method for Unconstrained Optimization with Logistic Mapping

In this paper , we suggested a new conjugate gradient algorithm for unconstrained optimization based on logistic mapping, descent condition and sufficient descent condition for our method are provided. Numerical results show that our presented algorithm is more efficient for solving nonlinear unconstrained optimization problems comparing with (DY) .

A Competitive Divide-and-Conquer Algorithm for Unconstrained Large-Scale Black-Box Optimization

This article proposes a competitive divide-and-conquer algorithm for solving large-scale black-box optimization problems for which there are thousands of decision variables and the algebraic models of the problems are unavailable. We focus on problems that are partially additively separable, since this type of problem can be further decomposed into a number of smaller independent subproblems. The proposed algorithm addresses two important issues in solving large-scale black-box optimization: (1) the identification of the independent subproblems without explicitly knowing the formula of the objective function and (2) the optimization of the identified black-box subproblems. First, a Global Differential Grouping (GDG) method is proposed to identify the independent subproblems. Then, a variant of the Covariance Matrix Adaptation Evolution Strategy (CMA-ES) is adopted to solve the subproblems resulting from its rotation invariance property. GDG and CMA-ES work together under the cooperative co-evolution framework. The resultant algorithm, named CC-GDG-CMAES, is then evaluated on the CEC’2010 large-scale global optimization (LSGO) benchmark functions, which have a thousand decision variables and black-box objective functions. The experimental results show that, on most test functions evaluated in this study, GDG manages to obtain an ideal partition of the index set of the decision variables, and CC-GDG-CMAES outperforms the state-of-the-art results. Moreover, the competitive performance of the well-known CMA-ES is extended from low-dimensional to high-dimensional black-box problems.

Optimization of multilayer cylindrical cloaks using genetic algorithms and NEWUOA

The problem of minimizing the scattering from a multilayer cylindrical cloak is studied. Both TM and TE polarizations are considered. A two-stage optimization procedure using genetic algorithms and NEWUOA (new unconstrained optimization algorithm) is adopted for realizing the cloak using homogeneous isotropic layers. The layers are arranged such that they follow a repeated pattern of alternating DPS and DNG materials. The results show that a good level of invisibility can be realized using a reasonable number of layers. Maintaining the cloak performance over a finite range of frequencies without sacrificing the level of invisibility is achieved.

A cubic regularization algorithm for unconstrained optimization using line search and nonmonotone techniques

In recent years, cubic regularization algorithms for unconstrained optimization have been defined as alternatives to trust-region and line search schemes. These regularization techniques are based on the strategy of computing an (approximate) global minimizer of a cubic overestimator of the objective function. In this work we focus on the adaptive regularization algorithm using cubics (ARC) proposed in Cartis et al. [Adaptive cubic regularisation methods for unconstrained optimization. Part I: motivation, convergence and numerical results, Mathematical Programming A 127 (2011), pp. 245–295]. Our purpose is to design a modified version of ARC in order to improve the computational efficiency preserving global convergence properties. The basic idea is to suitably combine a Goldstein-type line search and a nonmonotone accepting criterion with the aim of advantageously exploiting the possible good descent properties of the trial step computed as (approximate) minimizer of the cubic model. Global convergence properties of the proposed nonmonotone ARC algorithm are proved. Numerical experiments are performed and the obtained results clearly show satisfactory performance of the new algorithm when compared to the basic ARC algorithm.

Part orientation optimisation for the additive layer manufacture of metal components

In a number of additive layer manufacturing processes, particularly for metals, additional support structure is required during the build process to act as scaffolding for overhanging features and to dissipate process heat. Such structures use valuable raw materials and their removal adds to post processing time. The objective of this study was to investigate whether a simple, single objective optimisation technique could be used to find the best orientation of the part, that would minimise the volume of support needed during the build. Not only reducing waste but potentially providing an effective and consistent approach for inexperienced users to orient components during manufacture. Software was developed using MatLab with an unconstrained optimisation algorithm implemented to search the different rotations of the part and identify the configuration with the least requirement for support volume. The algorithm was gradient based, and so multiple starting points were used to identify a global minimum. The efficacy of the algorithm is illustrated with three different case studies of increasing complexity. Additionally, the component of the final study was manufactured, which allowed a comparison between the algorithm’s results and the orientations chosen by experienced operatives. In two of the three case studies, the software was able to find good solutions for the support volume minimisation. For the manufactured part, only one of the results matched the orientation chosen by the operators, the other was orientated in a similar way but the difference added significantly to the required support volume. Future developments of the software would benefit from incorporating the expertise of the manufacturing operative.

Solving Large-Deflection Problem of Spatial Beam with Circular Cross-Section Using an Optimization-Based Runge–Kutta Method

Abstract Based on the Bernoulli–Euler beam theory, the nonlinear governing differential equations (GDEs) for a spatially deflected beam with circular cross-section are formulated, which are then reduced to first-order differential equations to be compatible with Runge–Kutta method. With the boundary conditions of a spatial beam, the governing equations are treated as an initial value problem (IVP) of ordinary differential equations. A Runge–Kutta method combined with an unconstrained optimization algorithm (RKUO) is presented to solve the IVP. The approach for determining the orientation of the cross-section plane at any position on the deflected beam is also provided. Finally, the comparison between the RKUO results and those achieved using nonlinear finite element (NFE) analysis and spatial pseudo-rigid-body model validate the accuracy and effectiveness of RKUO. The results also demonstrated the unique capabilities of RKUO to solve large spatial deflection problems that are outside the range of nonlinear finite element model.

On the worst-case complexity of nonlinear stepsize control algorithms for convex unconstrained optimization

A Nonlinear Stepsize Control (NSC) framework has been proposed by Toint [Nonlinear stepsize control, trust regions and regularizations for unconstrained optimization, Optim.Methods Softw. 28 (2013), pp. 82–95] for unconstrained optimization, generalizing many trust-region and regularization algorithms. More recently, worst-case complexity bounds for the generic NSC framework were proved by Grapiglia et al. [On the convergence and worst-case complexity of trust-region and regularization methods for unconstrained optimization, Math. Program. 152 (2015), pp. 491–520] in the context of non-convex problems. In this paper, improved complexity bounds are obtained for convex and strongly convex objectives.

A Kriging-Based Unconstrained Global Optimization Algorithm

Abstract Efficient Global Optimization (EGO) algorithm with Kriging model is stable and effective for an expensive black-box function. However, How to get a more global optimal point on the basis of surrogates has been concerned in simulation-based design optimization. In order to better solve a black-box unconstrained optimization problem, this paper introduces a new EGO method named improved generalized EGO (IGEGO), in which two targets will be achieved: using Kriging surrogate model and guiding the optimal searching direction into more promising regions. Kriging modeling which can fast construct an approximation model is the premise ofperforming optimization. Next, a new infill sampling criterion (ISC) called improved generalized expected improvement which round off Euclidean norm on variation of the optimal solutions ofparameter θ to replace parameter g can effectively balance global and local search in IGEGO method. Twelve numerical tests and an engineering example are given to illustrate the reliability, applicability and effectiveness of the present method

A second-order globally convergent direct-search method and its worst-case complexity

Direct-search algorithms form one of the main classes of algorithms for smooth unconstrained derivative-free optimization, due to their simplicity and their well-established convergence results. They proceed by iteratively looking for improvement along some vectors or directions. In the presence of smoothness, first-order global convergence comes from the ability of the vectors to approximate the steepest descent direction, which can be quantified by a first-order criticality (cosine) measure. The use of a set of vectors with a positive cosine measure together with the imposition of a sufficient decrease condition to accept new iterates leads to a convergence result as well as a worst-case complexity bound. In this paper, we present a second-order study of a general class of direct-search methods. We start by proving a weak second-order convergence result related to a criticality measure defined along the directions used throughout the iterations. Extensions of this result to obtain a true second-order optimality one are discussed, one possibility being a method using approximate Hessian eigenvectors as directions (which is proved to be truly second-order globally convergent). Numerically guaranteeing such a convergence can be rather expensive to ensure, as it is indicated by the worst-case complexity analysis provided in this paper, but turns out to be appropriate for some pathological examples.

A memory based differential evolution algorithm for unconstrained optimization

In optimization, the performance of differential evolution (DE) and their hybrid versions exist in the literature is highly affected by the inappropriate choice of its operators like mutation and crossover. In general practice, during simulation DE does not employ any strategy of memorizing the so-far-best results obtained in the initial part of the previous generation. In this paper, a new “Memory based DE (MBDE)” presented where two “swarm operators” have been introduced. These operators based on the pBEST and gBEST mechanism of particle swarm optimization. The proposed MBDE is employed to solve 12 basic, 25 CEC 2005, and 30 CEC 2014 unconstrained benchmark functions. In order to further test its efficacy, five different test system of model order reduction (MOR) problem for single-input and single-output system are solved by MBDE. The results of MBDE are compared with state-of-the-art algorithms that also solved those problems. Numerical, statistical, and graphical analysis reveals the competency of the proposed MBDE.

A new filled function method applied to unconstrained global optimization

In this paper, we propose a new filled function, and give an efficient criterion to choose its two parameters appropriately. An algorithm for unconstrained global optimization is developed from this new filled function. numerical results with a comparison on many test problems show that this algorithm is efficient and reliable.