Competitive learning by mutual information maximization

Abstract

We propose a new information maximization method for feature discovery and demonstrate that it can discover linguistic rules in unsupervised ways. The new method can directly control competitive unit activation patterns to which input-competitive connections are adjusted. This direct control of the activation patterns permits considerable flexibility for connections and shows the ability to discover salient features not captured by traditional methods. We applied the new method to a linguistic rule acquisition problem. In this problem, unsupervised methods are needed because children acquire rules even without any explicit instruction. Our results confirmed that only by maximizing information content in competitive units linguistic rules can be extracted. These results suggest that linguistic rule acquisition is induced by the processes of information maximization in living systems.