Constrained Information Maximization to Control Internal Representatio

Abstract

In the present paper, we propose a constrained information maximization method to control internal representations obtained in a course of learning. We focus upon hidden units and define information in hidden units acquired by learning. Internal representations are transformed by controlling this information. To control internal representations, a constraint is introduced in information maximization that total output from all the hidden units is a constant. By changing values of the constant, it is possible to generate many kinds of different internal representations, corresponding to the information content in hidden units. For example, we can obtain compact output patterns and specialized patterns of hidden units by changing the constant. We applied the constrained information maximization method to alphabet character recognition problems and a rule acquisition problem of an artificial language close to English. In the experiments, we were especially concerned with the generation of specialized hidden units, one of the typical example of the control of internal representations. Experimental results confirmed that we can control internal representations to produce specialized hidden units and to detect and extract main features of input patterns