Equivalence of Additive-Combinatorial Linear Inequalities for Shannon Entropy and Differential Entropy

Abstract

This paper addresses the correspondence between linear inequalities for Shannon entropy and differential entropy for sums of independent group-valued random variables. We show that any balanced (with the sum of coefficients being zero) linear inequality for Shannon entropy holds if and only if its differential entropy counterpart also holds; moreover, any linear inequality for differential entropy must be balanced. In particular, our result shows that recently proved differential entropy inequalities by Kontoyiannis and Madiman can be deduced from their discrete counterparts due to Tao in a unified manner. Generalizations to certain abelian groups are also obtained. Our proof of extending inequalities for Shannon entropy to differential entropy relies on a result of Renyi which relates the Shannon entropy of a finely discretized random variable to its differential entropy and also helps in establishing that the entropy of the sum of quantized random variables is asymptotically equal to that of the quantized sum; the converse uses the asymptotics of the differential entropy of convolutions with weak additive noise.