Maximum Entropy Distribution Estimation with Generalized Regularization

Abstract

AbstractWe present a unified and complete account of maximum entropy distribution estimation subject to constraints represented by convex potential functions or, alternatively, by convex regularization. We provide fully general performance guarantees and an algorithm with a complete convergence proof. As special cases, we can easily derive performance guarantees for many known regularization types, including ℓ1, ℓ2, \(\ell_{\rm 2}^{\rm 2}\) and ℓ1 + \(\ell_{\rm 2}^{\rm 2}\) style regularization. Furthermore, our general approach enables us to use information about the structure of the feature space or about sample selection bias to derive entirely new regularization functions with superior guarantees. We propose an algorithm solving a large and general subclass of generalized maxent problems, including all discussed in the paper, and prove its convergence. Our approach generalizes techniques based on information geometry and Bregman divergences as well as those based more directly on compactness.KeywordsRelative EntropyPerformance GuaranteeGeneralize RegularizationGibbs DistributionDual ObjectiveThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.