Discriminating features and information theory

Abstract

We assume that visual recognition is accomplished by describing an object using visual features and that description is matched to some stored data base. These features must be chosen from the set of all computable properties of an image, be they direct image properties or properties of the actual object inferred from the image. What criteria should be used in selecting the features? Previous work1,2 has investigated the use of information theoretic constraints on feature selection, especially with respect to designing hierarchical classifiers. These methods attempt to maximize the entropy reduction at each decision step, where the entropy is based on the class membership probability. No consideration, however, is given to the description of the world generated by the particular features chosen. We propose that objects in the world are clustered, exhibiting high degrees of redundancy; as such, their description should reflect this structure. Therefore, the description of objects should be used as criteria for evaluating the chosen features. Specifically, the redundancy and discriminability of objects as described by the features should satisfy certain constraints derived from information theory. These constraints, expressed in terms of entropy and mutual-information measures, are motivated theoretically from the goals of a visual recognition system and empirically from cognitive science.