Stochastic Chaining and Strengthened Information-Theoretic Generalization Bounds

Abstract

We propose a new approach to apply the chaining technique in conjunction with information-theoretic measures to bound the generalization error of machine learning algorithms. Different from the deterministic approach previously proposed by Asadi et al., which is based on hierarchical partitions of a bounded metric space, we propose a stochastic approach that replaces the hierarchical partitions with an abstract Markov model inspired by successive refinement source coding in information theory. Our approach has three main benefits over the deterministic approach: 1) applicability to unbounded metric space, 2) feasibility of subsequent analysis to yield explicit bounds, and 3) increased flexibility for optimization. We illustrate these benefits through the problems of estimating Gaussian mean and phase retrieval. For the problem of estimating Gaussian mean, we derive a chaining bound that provides an order-wise improvement over previous results; for the problem of phase retrieval, we construct a stochastic chain that allows optimization over the chaining parameter.