Learning Algorithm For Multilayer Networks Research Articles

An analytic layer-wise deep learning framework with applications to robotics

Deep learning (DL) has achieved great success in many applications, but it has been less well analyzed from the theoretical perspective. The unexplainable success of black-box DL models has raised questions among scientists and promoted the emergence of the field of explainable artificial intelligence (XAI). In robotics, it is particularly important to deploy DL algorithms in a predictable and stable manner as robots are active agents that need to interact safely with the physical world. This paper presents an analytic deep learning framework for fully connected neural networks, which can be applied for both regression problems and classification problems. Examples for regression and classification problems include online robot control and robot vision. We present two layer-wise learning algorithms such that the convergence of the learning systems can be analyzed. Firstly, an inverse layer-wise learning algorithm for multilayer networks with convergence analysis for each layer is presented to understand the problems of layer-wise deep learning. Secondly, a forward progressive learning algorithm where the deep networks are built progressively by using single hidden layer networks is developed to achieve better accuracy. It is shown that the progressive learning method can be used for fine-tuning of weights from convergence point of view. The effectiveness of the proposed framework is illustrated based on classical benchmark recognition tasks using the MNIST and CIFAR-10 datasets and the results show a good balance between performance and explainability. The proposed method is subsequently applied for online learning of robot kinematics and experimental results on kinematic control of UR5e robot with unknown model are presented.

A second-order learning algorithm for multilayer networks based on block Hessian matrix

This article proposes a new second-order learning algorithm for training the multilayer perceptron (MLP) networks. The proposed algorithm is a revised Newton's method. A forward–backward propagation scheme is first proposed for network computation of the Hessian matrix, H, of the output error function of the MLP. A block Hessian matrix, H b, is then defined to approximate and simplify H. Several lemmas and theorems are proved to uncover the important properties of H and H b, and verify the good approximation of H b to H; H b preserves the major properties of H. The theoretic analysis leads to the development of an efficient way for computing the inverse of H b recursively. In the proposed second-order learning algorithm, the least squares estimation technique is adopted to further lessen the local minimum problems. The proposed algorithm overcomes not only the drawbacks of the standard backpropagation algorithm (i.e. slow asymptotic convergence rate, bad controllability of convergence accuracy, local minimum problems, and high sensitivity to learning constant), but also the shortcomings of normal Newton's method used on the MLP, such as the lack of network implementation of H, ill representability of the diagonal terms of H, the heavy computation load of the inverse of H, and the requirement of a good initial estimate of the solution (weights). Several example problems are used to demonstrate the efficiency of the proposed learning algorithm. Extensive performance (convergence rate and accuracy) comparisons of the proposed algorithm with other learning schemes (including the standard backpropagation algorithm) are also made.

Bounds on the learning capacity of some multi-layer networks

We obtain bounds for the capacity of some multi-layer networks of linear threshold units. In the case of a network having n inputs, a single layer of h hidden units and an output layer of s units, where all the weights in the network are variable and s?h?n, the capacity m satisfies 2n?m?nt logt, where t=1=h/s. We consider in more detail the case where there is a single output that is a fixed boolean function of the hidden units. In this case our upper bound is of order nh logh but the argument which provided the lower bound of 2n no longer applies. However, by explicit computation in low dimensional cases we show that the capacity exceeds 2n but is substantially less than the upper bound. Finally, we describe a learning algorithm for multi-layer networks with a single output unit. This greatly outperforms back propagation at the task of learning random vectors and provides further empirical evidence that the lower bound of 2n can be exceeded.