Constrained information maximization by free energy minimization

Abstract

In this paper we introduce free energy-based methods to constrain mutual information maximization, developed to realize competitive learning. The new method is introduced to simplify the computational procedures of mutual information and to improve the fidelity of representation and to stabilize learning. First, the free energy is effective in simplifying the computation procedures of mutual information because we need not directly compute mutual information, which needs heavy computation, but only deals with partition functions. With partition functions, computational complexity is significantly reduced. Second, fidelity to input patterns can be improved because training errors between input patterns and connection weights are implicitly incorporated. This means that mutual information is maximized under the constraint of the errors between input patterns and connection weights. Finally, learning can be stabilized in our approach. One of the problems of the free energy approach is that learning processes should be carefully controlled to keep its stability. The present paper shows that the conventional computational techniques in the field of self-organizing maps are really effective in controlling the processes. In particular, minimum information production learning can be used further to stabilize learning by decreasing information obtained at each learning step as much as possible. Thus, we can expect that our new method can be used to increase mutual information between competitive units and input patterns without decreasing errors between input patterns and connection weights and with stabilized learning processes. We applied the free energy-based models to the well-known Iris problem and a student survey, and succeeded in improving the performance in terms of classification rates. In addition, the minimum information production learning turned out to be effective in stabilizing learning.