Variable Projection Method and Levenberg-Marquardt Algorithm for Neural Network Training

Abstract

An optimal solution for single hidden layered feedforward neural network (SLFN) is proposed. SLFN can be considered as separable nonlinear least squares problem and variable projection (VP) method gives the approximation of the Jacobian matrix of the problem. The Jacobian calculation of VP-SLFN is suggested with simplified form. Based on this simplification, Levenberg-Marquardt (LM) algorithm for VP-SLFN is suggested and has faster convergence rate than LM algorithm without VP. Two numerical examples show the superiority than extreme learning machine and LM without variable projection