Decision tree design using information theory

Abstract

The theoretical models based on rate-distortion theory and prefix-coding analogies explain previously observed experimental phenomena reported in the literature. An application to edge detection is described where we primarily emphasize the inductive methodology rather than the domain application (image processing) per se. We conclude that inductive learning paradigms based on information-theoretic models are both theoretically well-behaved and useful in practical problems. i Rule induction from large data sets is currently receiving attention in the areas of machine learning and expert systems. Classifier design from labelled training l' samples is a problem which shares many characteristics with the rule induction problem. A recent paper by Bundy, Silver & Plummer (1985) provides a useful discussion of how the two problems relate to each other. The basic premise of many rule induction mechanisms, when the data is probabilistic rather than deterministic, is to induce a hierarchy or decision tree as a representation of the relationships between the attributes (evidence) and the classes (hypotheses). Hence general relationships between classes and attributes are induced or learned by the induction mechanism. Note that the terms attributes and classes occur more often in pattern recognition literature than the terms evidence and hypotheses which tend to be used in the artificial intelligence domain. For the purposes of this paper we adopt the former. When the attribute-class relationships are probabilistic rather than deterministic, the induction mechanisms which work best appear to be those based on statistical