Singularities in neural networks make Bayes generalization errors smaller even if they do not contain the true

Abstract

Hierarchical learning machines used in information science have singularities in their parameter spaces. At singularities, the Fisher information matrix becomes degenerate, resulting that the learning theory of regular statistical models can not be applied. Recently, it was proven that, if the true parameter is contained in the singularities, then the generalization error in Bayes estimation is far smaller than those of regular statistical models. In this paper, under the condition that the true parameter is not contained in singularities, we show two results: (1) if the dimension of the parameter from inputs to hidden units is not larger than three, then there exits a region of true parameters such that the generalization error is larger than those of regular models; and (2) if the dimension is larger than three, then, for an arbitrary true parameter, the generalization error is smaller than those of regular models.