Information Theoretic Signal Processing and Its Applications [Bookshelf

Abstract

The roots of information theory are almost 100 years old and include early works by Fisher <xref ref-type="bibr" rid="ref1" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">[1]</xref> , Hartley <xref ref-type="bibr" rid="ref2" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">[2]</xref> , and others. According to a history of information theory <xref ref-type="bibr" rid="ref3" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">[3]</xref> , motivated to understand how to draw information from experiments, Fisher <xref ref-type="bibr" rid="ref1" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">[1]</xref> stated “the nature and degree of the uncertainty [must] be capable of rigorous expression.” Subsequently, he defined statistical information as the reciprocal of the variance of a statistical sample. However, it was Shannon’s work <xref ref-type="bibr" rid="ref4" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">[4]</xref> that laid the mathematical foundations of information theory and revolutionized communications. Shannon developed two fundamental bounds, one on data compression and the other on transmission rate. He proved that even in the presence of noise, an arbitrarily small probability of error may be achieved as long as the transmission rate is below a quantity he defined as channel capacity.