Information maximization and cost minimization in information-theoretic competitive learning

Abstract

In this paper, we introduce costs in the framework of information maximization and try to maximize the ratio of information to its associated cost. We have shown that competitive learning is realized by maximizing mutual information between input patterns and competitive units. One shortcoming of the method is that maximizing information does not necessarily produce representations faithful to input patterns. Information maximizing primarily focuses on some parts of input patterns that are used to distinguish between patterns. Thus, we introduce the cost that represents average distance between input patterns and connection weights. By minimizing the cost, final connection weights by information maximization reflect well input patterns. We applied the method to a political data analysis and a Wisconsin cancer problem. Experimental results confirmed that by introducing the cost, representations faithful to input patterns were obtained. In addition, generalization performance was significantly improved.