Information Compression for Intelligible Multi-Layered Neural Networks

Abstract

The paper aims to propose a new type of information-theoretic method to interpret the final results of multi-layered neural networks by compressing multi-layered neural networks into ones without hidden layers, keeping good generalization performance. First, information on input patterns is augmented as much as possible to cope with the natural tendency of losing information in the process of compression. Then, multi-layered neural networks with much information content are compressed into simple networks without hidden layers by compressing connection weights or collectively treating all connection weights. With those collective weights, simple networks without hidden layers can be trained, incorporating information acquired by multi-layered neural networks with better generalization performance. The method was applied to two business data sets, where the interpretation of relations between inputs and outputs is much more important. In both cases, improved generalization performance was obtained by simple networks without hidden layers, trained with collective weights of multi-layered networks. The final connection weights tended to choose simpler and clearer linear correlations between inputs and outputs, which were easily and naturally interpreted.