Multiscale optimization in neural nets: preliminary report

Abstract

A multiscale optimization method for neural networks governed by quite general objective functions is presented. At the coarse scale, there is a smaller, approximating neural net. Like the original net, it is nonlinear and has a nonquadratic objective function, so the coarse-scale net is a more accurate approximation than a quadratic objective would be. The transitions and information flow form fine to coarse scale and back do not disrupt the optimization. The problem need not involve any geometric domain; all that is required is a partition of the original fine-scale variables. Given this partition, the rest of the multiscale optimization method requires no problem-specific design effort on the part of the user, since the mapping between coarse and fine scales is determined. Thus, the method can be applied easily to many problems and networks. Positive experimental results including cost comparisons are shown