Conditional information and information loss for flexible feature extraction

Abstract

In this paper, we propose a new information-theoretic approach to competitive learning and self-organizing maps. We use several information-theoretic measures such as conditional information and information losses to extract main features in input patterns. First, conditional information content is used to show how much information is contained in a competitive unit or an input pattern. Then, information content in each variable is detected by information losses. The information loss is defined by difference between information with all input units and information without an input unit. We applied the method to an artificial data, the Iris problem, a student survey, a CPU classification problem and a company survey. In all cases, experimental results showed that main features in input patterns were clearly detected.