Redundant Information Neural Estimation.

Michael Kleinman,Stefano Soatto,Alessandro Achille,Jonathan C Kao

doi:10.3390/e23070922

Michael Kleinman, Stefano Soatto + Show 2 more

Open Access

https://doi.org/10.3390/e23070922

Copy DOI

Abstract

We introduce the Redundant Information Neural Estimator (RINE), a method that allows efficient estimation for the component of information about a target variable that is common to a set of sources, known as the “redundant information”. We show that existing definitions of the redundant information can be recast in terms of an optimization over a family of functions. In contrast to previous information decompositions, which can only be evaluated for discrete variables over small alphabets, we show that optimizing over functions enables the approximation of the redundant information for high-dimensional and continuous predictors. We demonstrate this on high-dimensional image classification and motor-neuroscience tasks.

Highlights

Given a set of sources X1, . . . , Xn and a target variable Y, we study how information about the target Y is distributed among the sources: different sources may contain information that no other source has (“unique information”), contain information that is common to other sources (“redundant information”), or contain complementary information that is only accessible when considered jointly with other sources (“synergistic information”)
We introduce the Redundant Information Neural Estimator (RINE), a method that enables the approximation of the redundant information that high-dimensional sources contain about a target variable
We apply our method to estimate the redundant information on canonical examples that were previously used to study the Partial Information Decomposition (PID), and demonstrate the ability to compute the redundant information for problems where the predictors are high dimensional

Summary

Introduction

Xn and a target variable Y, we study how information about the target Y is distributed among the sources: different sources may contain information that no other source has (“unique information”), contain information that is common to other sources (“redundant information”), or contain complementary information that is only accessible when considered jointly with other sources (“synergistic information”). Such a decomposition of the information across the sources can inform the design of multi-sensor systems (e.g., to reduce redundancy between sensors), or support research in neuroscience, where neural activity is recorded from two areas during a behavior.

Objectives

Results

Conclusion