Direct Differentiation Based Hessian Formulation for Training Multilayer Feed forward Neural Networks using the LM Algorithm-Performance Comparison with Conventional Jacobian-Based Learning

Abstract

The Levenberg-Marquardt (LM) algorithm is the most commonly used training algorithm for moderate-sized feed forward artificial neural networks (ANNs) due to its high convergence rate and reasonably good accuracy. It conventionally employs a Jacobian-based approximation to the Hessian matrix, since exact evaluation of the Hessian matrix is generally considered computationally prohibitive. However, the storage of Jacobian matrix in computer memory is itself prone towards memory constraints, especially if the number of patterns in the training data exceeds a critical threshold. This paper presents a first attempt of evaluating the exact Hessian matrix using the direct differentiation approach for training a multilayer feed forward neural network using the LM algorithm. The weights employed for network training are initialized using a random number generator in MATLAB (R2010a). The efficiency of the proposed algorithm has been demonstrated using the well-known 2-spiral and the parity-N datasets, and the training performance has been compared with the Neural Network Toolbox in MATLAB (R2010a) which employs the conventional Jacobian-based learning methodology.