Dynamics of learning near singularities in radial basis function networks

Abstract

The radial basis function (RBF) networks are one of the most widely used models for function approximation in the regression problem. In the learning paradigm, the best approximation is recursively or iteratively searched for based on observed data (teacher signals). One encounters difficulties in such a process when two component basis functions become identical, or when the magnitude of one component becomes null. In this case, the number of the components reduces by one, and then the reduced component recovers as the learning process proceeds further, provided such a component is necessary for the best approximation. Strange behaviors, especially the plateau phenomena, have been observed in dynamics of learning when such reduction occurs. There exist singularities in the space of parameters, and the above reduction takes place at the singular regions. This paper focuses on a detailed analysis of the dynamical behaviors of learning near the overlap and elimination singularities in RBF networks, based on the averaged learning equation that is applicable to both on-line and batch mode learning. We analyze the stability on the overlap singularity by solving the eigenvalues of the Hessian explicitly. Based on the stability analysis, we plot the analytical dynamic vector fields near the singularity, which are then compared to those real trajectories obtained by a numeric method. We also confirm the existence of the plateaus in both batch and on-line learning by simulation.