Class Of Unconstrained Optimization Problems Research Articles

Adaptive Evolution Strategies for Stochastic Zeroth-Order Optimization

We consider solving a class of unconstrained optimization problems in which only stochastic estimates of the objective functions are available. Existing stochastic optimization methods are mainly extended from gradient-based methods, faced with the challenges of noisy function evaluations, hardness in choosing step-sizes, and probably ill-conditioned landscapes. This paper presents a stochastic evolution strategy (SES) framework and several adaptation schemes to avoid these challenges. The SES framework combines the ideas of population sampling and minibatch sampling in exploiting the zeroth-order gradient information, efficiently reducing the noise in both data selection and gradient approximation. In addition, it admits approximating the gradients using a non-isotropic Gaussian distribution to better capture the curvature information of the landscapes. Based on this framework, we implement a step-size adaptation rule and two covariance matrix adaptation rules, where the former can automatically tune the step-sizes and the latter are intended to cope with ill-conditioning. For SES with certain fixed step-sizes, we establish a nearly optimal convergence rate over smooth landscapes. We also show that using the adaptive step-sizes allows convergence at a slightly slower rate but without the need to know the smoothness constant. Several numerical experiments on machine learning problems verify the above theoretical results and suggest that the adaptive SES methods show much promise.

Alternative structured spectral gradient algorithms for solving nonlinear least-squares problems

The study of efficient iterative algorithms for addressing nonlinear least-squares (NLS) problems is of great importance. The NLS problems, which belong to a special class of unconstrained optimization problems, are of particular interest because of the special structure of their gradients and Hessians. In this paper, based on the spectral parameters of Barzillai and Borwein (1998), we propose three structured spectral gradient algorithms for solving NLS problems. Each spectral parameter in the respective algorithms incorporates the structured gradient and the information gained from the structured Hessian approximation. Moreover, we develop a safeguarding technique for the first two structured spectral parameters to avoid negative curvature directions. Moreso, using a nonmonotone line-search strategy, we show that the proposed algorithms are globally convergent under some standard conditions. The comparative computational results on some standard test problems show that the proposed algorithms are efficient.

On unconstrained optimization problems solved using the canonical duality and triality theories

DY Gao solely or together with some of his collaborators applied his Canonical duality theory (CDT) for solving a class of unconstrained optimization problems, getting the so-called triality theorems. Unfortunately, the ‘double-min duality’ from these results published before 2010 revealed to be false, even if in 2003 DY Gao announced that ‘certain additional conditions’ are needed for getting it. After 2010 DY Gao together with some of his collaborators published several papers in which they added additional conditions for getting ‘double-min’ and ‘double-max’ dualities in the triality theorems. The aim of this paper is to treat rigorosly this kind of problems and to discuss several results concerning the ‘triality theory’ obtained up to now.

A Selection Process for Genetic Algorithm Using Clustering Analysis

This article presents a newly proposed selection process for genetic algorithms on a class of unconstrained optimization problems. The k-means genetic algorithm selection process (KGA) is composed of four essential stages: clustering, membership phase, fitness scaling and selection. Inspired from the hypothesis that clustering the population helps to preserve a selection pressure throughout the evolution of the population, a membership probability index is assigned to each individual following the clustering phase. Fitness scaling converts the membership scores in a range suitable for the selection function which selects the parents of the next generation. Two versions of the KGA process are presented: using a fixed number of clusters K (KGAf) and via an optimal partitioning Kopt (KGAo) determined by two different internal validity indices. The performance of each method is tested on seven benchmark problems.