Multi‐directional search to optimize neural network error function

Abstract

PurposeThe proposed algorithm successfully optimizes complex error functions, which are difficult to differentiate, ill conditioned or discontinuous. It is a benchmark to identify initial solutions in artificial neural network (ANN) training.Design/methodology/approachA multi‐directional ANN training algorithm that needs no derivative information is introduced as constrained one‐dimensional problem. A directional search vector examines the ANN error function in weight parameter space. The search vector moves in all possible directions to find minimum function value. The network weights are increased or decreased depending on the shape of the error function hyper surface such that the search vector finds descent directions. The minimum function value is thus determined. To accelerate the convergence of the algorithm a momentum search is designed. It avoids overshooting the local minimum.FindingsThe training algorithm is insensitive to the initial starting weights in comparison with the gradient‐based methods. Therefore, it can locate a relative local minimum from anywhere of the error surface. It is an important property of this training method. The algorithm is suitable for error functions that are discontinuous, ill conditioned or the derivative of the error function is not readily available. It improves over the standard back propagation method in convergence and avoids premature termination near pseudo local minimum.Research limitations/implicationsClassifications problems are efficiently classified when using this method but the complex time series in some instances slows convergence due to complexity of the error surface. Different ANN network structure can further be investigated to find the performance of the algorithm.Practical implicationsThe search scheme moves along the valleys and ridges of the error function to trace minimum neighborhood. The algorithm only evaluates the error function. As soon as the algorithm detects flat surface of the error function, care is taken to avoid slow convergence.Originality/valueThe algorithm is efficient due to incorporation of three important methodologies. The first mechanism is the momentum search. The second methodology is the implementation of directional search vector in coordinate directions. The third procedure is the one‐dimensional search in constrained region to identify the self‐adaptive learning rates, to improve convergence.