A Measure of Synergy, Redundancy, and Unique Information using Information Geometry

Abstract

It is well known that joint interactions between agents can be described qualitatively as having synergistic, unique, and redundant components. In recent years, there have been renewed efforts to decompose mutual information, a general, non-parametric measure of joint interactions, into constituent parts. We propose a novel, non-negative decomposition of mutual information between two sources and a target variable. The decomposition is for the exponential family, and thus can be applied to a broad range of distributions. We also show that values from our decomposition arise naturally from testing hypotheses of conditional dependence. We demonstrate the method numerically using standard binary logic gates and Gaussian channels, as well as apply the method to investigate redundancy between brain regions using an fMRI-based image classification data-set.