Chapter 2 - Complexity Measures

Abstract

This chapter introduces an alternative measure of information, or string complexity. First, it outlines a few issues with "Shannon Information Theory," all of which are addressed in a satisfactory way by "Algorithmic Information Theory," and then it provides basic definitions and properties of the "Kolmogorov complexity." The basic idea of the Kolmogorov complexity is to associate information content with the difficulty of describing data and whether the data could be modeled as coming from a probabilistic source as in Shannon Information Theory. Later, the chapter explains why this theory solves some of the problems inherent in traditional information theory. The chapter explores the fundamental properties of this complexity measure, including the computational complexity to determine the Kolmogorov complexity, which shows that the Kolmogorov complexity is uncomputable. The chapter further explains that the Kolmogorov complexity and the traditional information theory strongly agree in some situations where both may be applied.