An information-theoretic approach to feature extraction in competitive learning

Abstract

In this paper, we propose a new information-theoretic approach to competitive learning and self-organizing maps. We use several information-theoretic measures, such as conditional information and information losses, to extract main features in input patterns. For each competitive unit, conditional information content is used to show how much information on input patterns is contained. In addition, for detecting the importance of each variable, information losses are introduced. The information loss is defined as the difference between information with all input units and information without an input unit. We applied the information loss to conventional competitive learning to show that distinctive features could be extracted by the information loss. Then, we analyzed the self-organizing maps by the conditional information and the information loss. Experimental results showed that main features in input patterns were more clearly detected.