Weight space structure and generalization in the reversed-wedge perceptron

Abstract

The generalization ability of the reversed-wedge perceptron serving as a toy model for multilayer neural networks is investigated. We analyse the decomposition of the version space into disjoint cells belonging to different internal representations defined by the signs of the aligning fields. The version space is characterized by the number and size of these cells and their typical overlap with the teacher network. For a small training set the system is unable to detect the structure of the patterns induced by the teacher. Accordingly it performs as if storing random input - output patterns with very low generalization ability and a large misfit in the internal representation. With increasing training set size, cells with large misfit are eliminated at a much higher rate than those with internal representation similar to that of the teacher. This results eventually in the discontinuous phase transition to good generalization typical for multilayer neural networks.