A Critical Review of Shannon Information Theory

Abstract

In the preceding chapter, we summarized the basic elements of information theory, and we now proceed to examine and analyze the main characteristics of this theory. The term “critical” in the title of the chapter implies simply that we shall present a review of the main features for and against the theory. To support the theory in its present form, one can mention Shannon results on the capacity of a channel, the Boltzmann equation, and the fact that one can prove the central limit theorem in probability by using the properties of entropy only. Against the present form of the theory we have the apparent discrepancy between discrete entropy and continuous entropy, the absence of a concept of negative information to describe information lost, and the fact that the model does not take explicitly into account syntax and semantics. In the present chapter, we shall review these features and one of our conclusions will be as follows: Contrary to what some scientists are inclined to believe, we maintain that the continuous entropy is soundly defined, and that it merely remains to exhibit the differences in physical nature between discrete entropy and continuous entropy.