Stochastic convergence analysis of a partially adaptive two-layer perceptron using a system identification model

Abstract

This paper studies the stationary points of the output layer of a two-layer perceptron which attempts to identify the parameters of a specific nonlinear system. The training sequence is modeled as the binary output of the nonlinear system when the input is comprised of an independent sequence of zero-mean Gaussian vectors with independent components. The training rule for the output layer weights is a modified version of Rosenblatt's algorithm. Equations are derived which define the stationary points of the algorithm for an arbitrary output nonlinearity g(x). For the subsequent analysis, the output nonlinearity is specialized to g( x) = sgn( x). The solutions to these equations show that the only stationary points occur when the hidden weights of the perceptron are constrained to lie on the plane spanned by the nonlinear system model. In this plane, the angles of the perceptron weights and of the nonlinear system model weights satisfy a pair of homogeneous linear equations with an infinity of solutions. However, there is a unique solution for algorithm convergence (i.e., zero error) such that the parameters of the two-layer perceptron must exactly match that of the nonlinear system.