Generalization Properties Of Neural Networks Research Articles

Do Kernel and Neural Embeddings Help in Training and Generalization?

Recent results on optimization and generalization properties of neural networks showed that in a simple two-layer network, the alignment of the labels to the eigenvectors of the corresponding Gram matrix determines the convergence of the optimization during training. Such analyses also provide upper bounds on the generalization error. We experimentally investigate the implications of these results to deeper networks via embeddings. We regard the layers preceding the final hidden layer as producing different representations of the input data which are then fed to the two-layer model. We show that these representations improve both optimization and generalization. In particular, we investigate three kernel representations when fed to the final hidden layer: the Gaussian kernel and its approximation by random Fourier features, kernels designed to imitate representations produced by neural networks and finally an optimal kernel designed to align the data with target labels. The approximated representations induced by these kernels are fed to the neural network and the optimization and generalization properties of the final model are evaluated and compared.

Influence of the optimization methods on neural state estimation quality of the drive system with elasticity

The paper deals with the implementation of optimized neural networks (NNs) for state variable estimation of the drive system with an elastic joint. The signals estimated by NNs are used in the control structure with a state-space controller and additional feedbacks from the shaft torque and the load speed. High estimation quality is very important for the correct operation of a closed-loop system. The precision of state variables estimation depends on the generalization properties of NNs. A short review of optimization methods of the NN is presented. Two techniques typical for regularization and pruning methods are described and tested in detail: the Bayesian regularization and the Optimal Brain Damage methods. Simulation results show good precision of both optimized neural estimators for a wide range of changes of the load speed and the load torque, not only for nominal but also changed parameters of the drive system. The simulation results are verified in a laboratory setup.

Self-organizing Neural Network for Trajectory Control and Task Coordination of a Mobile Robot

A multi-layer neural network is used to control the navigation of a mobile robot in an environment containing obstacles in a temperature field. The mobile robot must avoid obstacles and hill climb towards the maximum of the temperature field. The strategy for each of these two tasks is acquired by learning. First by exploring the environment, the mobile robot extracts the relevant sensory situations by building up an internal map of the environment. The associations between these situations and the appropriate actions are then formed in an unsupervised manner, i.e. with no ‘teacher’required. The proposed structure of the system permits the coordination of the two tasks. Simulation results display not only the ability of the robot to achieve collision-free navigation towards its target in the explored environment, but also in new unvisited environments, illustrating the generalization property of neural networks.