Exploring the Trade-Off in the Variational Information Bottleneck for Regression with a Single Training Run

Abstract

An information bottleneck (IB) enables the acquisition of useful representations from data by retaining necessary information while reducing unnecessary information. In its objective function, the Lagrange multiplier β controls the trade-off between retention and reduction. This study analyzes the Variational Information Bottleneck (VIB), a standard IB method in deep learning, in the settings of regression problems and derives its optimal solution. Based on this analysis, we propose a framework for regression problems that can obtain the optimal solution of the VIB for all β values with a single training run. This is in contrast to conventional methods that require one training run for each β. The optimization performance of this framework is theoretically discussed and experimentally demonstrated. Our approach not only enhances the efficiency of exploring β in regression problems but also deepens the understanding of the IB’s behavior and its effects in this setting.