Multi-search optimization techniques

Abstract

Two multi-search optimization techniques are developed. The motive is to show that increasing the search directions results in a better rate of convergence. This idea was first investigated by Miele and Cantrell [1]. Their technique is computationally attractive, but exact line-search is needed in two directions. Therefore the problem of line-search has been more complicated since at the end of each iteration two stopping functions have to satisfied. Also, built-in safeguards were used to ensure the stability of the descent process. For nonquadratic functions Miele's method (as well as many other optimization techniques) needs restarting every n + 1 iterations in order to improve the convergence rate. Extending this two-direction search of Miele to an n- direction search in order to further improve the rate of convergence would highly complicate the search process, since exactly n line-searches are needed together with n simultaneous stopping functions to be satisfied besides the resulting increase in the built-in safeguards to impractical limits. In the techniques presented the optimum step-size in each direction is obtained by simply minimizing a quadratic function of a linearized gradient, and only sufficient decrease is needed along the search directions without the need of exact line-searches. Instead, simple successive halving is used till the function starts to increase; then a new iteration is started without the use of any simultaneous stopping functions at the end of each iteration. Also, the n- direction case has been developed in exactly the same fashion as the two-direction case without quiring the use of n simultaneous stopping functions a property which is not possible if the method of Miele is to be extended to the n- direction case. The highly effective rate of convergence obtained by using the n- direction search really justifies the would-be increase in the required computational effort. Furthermore, no restarting has been needed to improve the convergence rate. The results of two tried examples show that the multi-search optimization techniques that have been developed have excellent convergence rates without endangering the stability requirement.