Recurrent neural networks can be trained to be maximum a posteriori probability classifiers

Abstract

This paper proves that supervised learning algorithms used to train recurrent neural networks have an equilibrium point when the network implements a maximum a posteriori probability (MAP) classifier. The result holds as a limit when the size of the training set goes to infinity. The result is general, because it stems as a property of cost minimizing algorithms, but to prove it we implicitly assume that the network we are training has enough computing power to actually implement the MAP classifier. This assumption can be satisfied using a universal dynamic system approximator We refer our discussion to Block Feedback Neural Networks (B FNs) and show that they actually have the universal approximation property.