Statistical thermodynamic formalism in the solution of information theory problems

Abstract

This is the first of several papers dealing with the application of statistical thermodynamic methodology to the solution of coding and communication theory problems. Emphasis is placed on the various “ensemble techniques” of statistical mechanics, the words or “samples” of a message taking the place of molecules in the prototype physical system. Analogs of temperature, internal energy, pressure, chemical potential, volume, entropy, etc., are developed. The isomorphism with thermodynamics is complete and these quantities transform (for example, by partial differentiation) in exactly the same way as the prototype physical quantities. The methods are nicely applicable to coding cases involving sources with memory, in which case, correlation can be discussed in terms of analog “coupling energies” between signals or words so that the store of “many-body-problem” techniques can be used. In addition, the manipulative freedom stemming from the possibility of choosing from a multiplicity of ensembles constrained by “intensive” parameters proves a distinct advantage. A concrete example dealing with the choice of a compact code for a nonextended source with memory is presented. The compact code is derived, and some discussion is given concerning the breadth of its power spectrum. In a following paper, its autocorrelation function within the framework of “pulse code modulation” is derived and transformed by Wiener theory so that the power spectrum is directly exhibited (along with the spectra for several other cases).