Information transmission with time-continuous random processes

Abstract

Shannon's definition for the information content of a Gaussian, time-continuous process in Gaussian noise is extended to the case where the observation interval is finite, and where the processes may be nonstationary, in a straightforward way. The extension is based on a generalization of the Karhunen-Loeve Expansion, which allows both the signal and noise processes to be expanded in the same set of functions, with uncorrelated coefficients. The resultant definition is consistent with that of Gel'fand and Yaglom, and avoids the difficulties posed by Good and Doog to Shannon's original definition. This definition is shown to be useful by applying it to the calculation of the information content of some cases of stationary signals in stationary noise, with different spectra, and to one case where both are nonstationary. Limiting relations are derived, to show that this reduces to previously established results in some cases, and to enable one to obtain rule-of-thumb estimates in others. In addition, both the matched filter and the Wiener filter are related to the information; the matched filter in a very direct way, in that it converts a time-continuous process to a set of random variables while conserving the information.