Cost-conscious mutual information maximization for improving collective interpretation of multi-layered neural networks

Abstract

The present paper aims to improve the collective interpretation realized by compressing multi-layered neural networks and to make the interpretation as natural and stable as possible. We collectively interpret the final representations by maximizing mutual information between inputs and neurons, expecting that mutual information maximization can disentangle complex features into simpler ones. However, we have had difficulty in increasing mutual information and in obtaining interpretable features for several data sets. By examining closely the processes of information maximization, we found that, in addition to the information maximization, we need to consider the cost associated with this information maximization. Thus, we try to maximize not simply mutual information but the ratio of mutual information to the cost, and this method can be called “cost-conscious mutual information maximization.” The cost-conscious method aims to extend Linsker’s maximum information preservation principle to a variety of data sets by more directly taking into account the cost associated with the process of information maximization. The method was applied to two data sets: the artificial and symmetric data set and the credit default data set. First, by using the symmetric data set injected with random noises, the cost-conscious information maximization method could extract the symmetric property almost perfectly against the random noises. In the experimental results on the credit default data set, the present method could make it possible to interpret the final results the most naturally, showing why and how the credit default could occur very naturally. The experimental results show that the neural networks can be used to interpret data sets more naturally than the conventional methods such as the logistic regression analysis.