Tree pruning with subadditive penalties

Abstract

In this paper we study the problem of pruning a binary tree by minimizing, over all pruned subtrees of the given tree, an objective function that combines an additive cost term with a penalty term that depends only on tree size. We present algorithms for general size-based penalties, although our focus is on subadditive penalties (roughly, penalties that grow more slowly than linear penalties with increasing tree size). Such penalties are motivated by recent results in statistical learning theory for decision trees, but may have wider application as well. We show that the family of pruned subtrees induced by a subadditive penalty is a subset of the family induced by an additive penalty. This implies (by known results about additive penalties) that the family induced by a subadditive penalty 1) is nested; 2) is unique; and 3) can be computed efficiently. It also implies that, when a single tree is to be selected by cross-validation from the family of prunings, subadditive penalties will never present a richer set of options than an additive penalty.