Neural Network Optimization Method Research Articles

Ray-guided global optimization method for training neural networks

A novel method for globally searching to find the good minima is proposed in this paper. Starting from a local minimum, the weight space around it is scanned with the process being guided by terrain-independent emanating rays. During the search, starting points for further exploration are identified and used to find corresponding local minima. Based on the correct classification rate (CCR) on the validation data, the best minimum is found.

Quadratic optimization method for multilayer neural networks with local error-backpropagation

A fast new local error-backpropagation (LBP) algorithm is presented for the training of multilayer neural networks. This algorithm is based on the definition of a new local mean-squared error function. If the local desired outputs have been estimated, the multilayer neural networks can be decomposed into a set of adaptive linear elements (Adaline) that can be trained by quadratic optimization methods. Among a lot of quadratic optimization methods, the conjugate gradient (CG) method is one of the most famous methods that can find the global optimal solution of quadratic problems within finite steps. The iteration number and the computation time are significantly reduced because the stepsize is computed without line-search. Experimental results on the pattern recognition and memorizing of spatiotemporal patterns are provided.

Multiscale optimization in neural nets

One way to speed up convergence in a large optimization problem is to introduce a smaller, approximate version of the problem at a coarser scale and to alternate between relaxation steps for the fine-scale and coarse-scale problems. Such an optimization method for neural networks governed by quite general objective functions is presented. At the coarse scale, there is a smaller approximating neural net which, like the original net, is nonlinear and has a nonquadratic objective function. The transitions and information flow from fine to coarse scale and back do not disrupt the optimization, and the user need only specify a partition of the original fine-scale variables. Thus, the method can be applied easily to many problems and networks. There is generally about a fivefold improvement in estimated cost under the multiscale method. In the networks to which it was applied, a nontrivial speedup by a constant factor of between two and five was observed, independent of problem size. Further improvements in computational cost are very likely to be available, especially for problem-specific multiscale neural net methods.