Some Useful Integral Representations for Information-Theoretic Analyses.

Neri Merhav,Igal Sason

doi:10.3390/e22060707

Abstract

This work is an extension of our earlier article, where a well-known integral representation of the logarithmic function was explored and was accompanied with demonstrations of its usefulness in obtaining compact, easily-calculable, exact formulas for quantities that involve expectations of the logarithm of a positive random variable. Here, in the same spirit, we derive an exact integral representation (in one or two dimensions) of the moment of a nonnegative random variable, or the sum of such independent random variables, where the moment order is a general positive non-integer real (also known as fractional moments). The proposed formula is applied to a variety of examples with an information-theoretic motivation, and it is shown how it facilitates their numerical evaluations. In particular, when applied to the calculation of a moment of the sum of a large number, n, of nonnegative random variables, it is clear that integration over one or two dimensions, as suggested by our proposed integral representation, is significantly easier than the alternative of integrating over n dimensions, as needed in the direct calculation of the desired moment.

Highlights

In mathematical analyses associated with many problems in information theory and related fields, one is often faced with the need to compute expectations of logarithmic functions of composite random variables, or moments of such random variables, whose order may be a general positive real, not even necessarily an integer
We proceed in the same spirit as in [6], and we extend the scope to propose an integral representation of a general moment of a nonnegative random variable, X, namely the expectation, E{ X ρ } for a given real ρ > 0
We refer to this representation as an extension of (2), as the latter can be obtained as a special case of the formula for E{ X ρ }, by invoking one of the equivalent identities

Summary

Introduction

In mathematical analyses associated with many problems in information theory and related fields, one is often faced with the need to compute expectations of logarithmic functions of composite random variables (see, e.g., [1,2,3,4,5,6,7,8]), or moments of such random variables, whose order may be a general positive real, not even necessarily an integer (see, e.g., [9,10,11,12,13,14,15,16,17,18,19,20,21,22]). The integral representation we propose, in this work, applies to any non-integer, positive ρ, and here too, it replaces the direct calculation of E{ X ρ } by integration of an expression that involves the MGF of X We refer to this representation as an extension of (2), as the latter can be obtained as a special case of the formula for E{ X ρ }, by invoking one of the equivalent identities. Each one of these examples occupies one subsection of Section 3. The integral representations in this paper are not limited to the examples in Section 3, and such representations can be proven useful in other information-theoretic problems (see, e.g., [6] and the references therein for the integral representation of the logarithmic expectation and some of its information-theoretic applications)

Statistical Moments of Arbitrary Positive Orders

Applications

Moments of Guesswork

Moments of Estimation Errors

Rényi Entropy of Extended Multivariate Cauchy Distributions

Mutual Information Calculations for Communication Channels with Jamming