A new training and pruning algorithm based on node dependence and Jacobian rank deficiency

Abstract

In this paper, a new subset-based training and pruning (SBTP) algorithm is proposed based on the relationship between node dependence and Jacobian rank deficiency. At each training iteration, the orthogonal factorization with column permutation is applied to the output of the nodes in the same hidden layer to identify the dependent nodes. The output weights of the dependent nodes will be set as zeros, and the output weights of the independent nodes will be recalculated to maintain the original input–output behavior. Then, only the weights of the independent nodes will be trained using the Levenberg–Marquardt (LM) algorithm at this iteration, while keeping the weights of the dependent nodes unchanged. In this way, the computational cost of the LM algorithm will be reduced significantly. After the training process, a unit-based optimal brain surgeon (UB-OBS) pruning method is used to prune the insensitive hidden units to further reduce the size of the neural network, and no retraining is needed. Simulations are presented to demonstrate the effectiveness of the proposed approach.