Finding the Optimal Topology of an Approximating Neural Network

Abstract

A large number of researchers spend a lot of time searching for the most efficient neural network to solve a given problem. The procedure of configuration, training, testing, and comparison for expected performance is applied to each experimental neural network. The configuration parameters—training methods, transfer functions, number of hidden layers, number of neurons, number of epochs, and tolerable error—have multiple possible values. Setting guidelines for appropriate parameter values would shorten the time required to create an efficient neural network, facilitate researchers, and provide a tool to improve the performance of automated neural network search methods. The task considered in this paper is related to the determination of upper bounds for the number of hidden layers and the number of neurons in them for approximating artificial neural networks trained with algorithms using the Jacobi matrix in the error function. The derived formulas for the upper limits of the number of hidden layers and the number of neurons in them are proved theoretically, and the presented experiments confirm their validity. They show that the search for an efficient neural network can focus below certain upper bounds, and above them, it becomes pointless. The formulas provide researchers with a useful auxiliary tool in the search for efficient neural networks with optimal topology. They are applicable to neural networks trained with methods such as Levenberg–Marquardt, Gauss–Newton, Bayesian regularization, scaled conjugate gradient, BFGS quasi-Newton, etc., which use the Jacobi matrix.