Minimum α-information strategy for the interpretation of the network behaviors and the improved generalization

Abstract

Proposes a minimum /spl alpha/-information strategy for the explicit interpretation of the network behaviors and for the improved generalization performance. The /spl alpha/-information is defined by the difference between Shannon and Renyi entropy. The /spl alpha/-information minimization can be translated into entropy maximization and entropy minimization for hidden units in term of Shannon entropy. Thus, the /spl alpha/-information minimization forces hidden units to have a maximum information content or a minimum information content, depending on the importance of the hidden units. For the interpretation of the network behaviors, we have only to see a small number of maximum information hidden units, ignoring the minimum information hidden units. In addition, by minimizing the /spl alpha/-information, the unnecessary information can be eliminated, leading to the better generalization. The authors applied the /spl alpha/-information minimization to the inference of the sonority of the artificial language. Experimental results explicitly confirmed that the explicit internal representation could be obtained and the generalization could significantly be improved.