Teacher-forced information theoretic competitive learning

Abstract

In this paper, we propose a new type of competitive learning. Competition is realized by maximizing mutual information on training patterns as well as target patterns. After teaming, networks can generate correct outputs without teacher information. Thus, teaming is regulated by correlated teachers in an input layer. By this teacher-forced method, we need not back-propagate errors between targets and outputs into networks. Information flows always from an input layer to an output layer. Thus, the method is expected to be computationally effective. In addition, because information is maximized, information is compressed into networks in explicit ways, which enables us to discover salient features in input patterns. We applied this method to two problems: the vertical and horizontal lines detection problem and a fairly complex syntactic analysis system. Experimental results confirmed that teacher information in an input layer forces networks to produce correct answers. In addition, because of maximized information in competitive units, easily interpretable internal representations can be obtained. This method certainly opens up a new perspective in neural computing.