Tree-Structured Clustering via the Minimum Cross Entropy Principle

Abstract

We propose a new interdisciplinary approach to the tree-structured clustering problem, wherein structural constraints are imposed in order to reduce the classification search complexity of the resulting statistical classifier. Most known methods are greedy and optimize nodes of the tree one at a time to minimize a local cost. By constrast, we develop a joint optimization method, derived based on information-theoretic principles and closely related to known methods in statistical physics. The approach is inspired by the deterministic annealing method for unstructured clustering, which was based on maximum entropy inference. The new approach is based on the principle of minimum cross entropy, using informative priors to approximate the unstructured clustering solution while imposing the structural constraint. As in the original deterministic annealing method, the number of distinct representatives (and hence the tree) grows in a non-heuristic fashion by a sequence of phase transitions which occur so as to optimize the effective free energy cost. Examples demonstrate considerable improvement over known methods.