Entropy Estimation and Lossless Compression

Abstract

A frequently asked question is how much lossless compression can be achieved for a given image. In light of the noiseless source coding theorem, we know that the bit rate can be made arbitrarily close to the entropy of the source that generated the image. However, a fundamental problem is determining that entropy. An obvious approach to estimating the entropy is to characterize the source using a certain model and then find the entropy with respect to that model. Accurate source modeling is essential to any compression scheme since the performance bounds are established by the entropy with respect to that model. The effectiveness of a model is determined by how accurately it predicts the symbol probabilities. With natural information-generating sources such as speech and images, the more complex models, which are capable of accounting for the structure present in such sources, result in lower entropies and higher compression. The real challenge with this approach lies in approximating the source structure as close as possible while keeping the complexity of the model (the number of parameters) to a minimum. Another approach to estimating the entropy is to segment the image into blocks of size N and use the frequency of occurrence of each block as a measure of its probability. The entropy per original source symbol of the adjoint source formed in this way would approach the entropy of the original source as the block size goes to infinity. Unfortunately, the convergence to the true entropy is slow and one needs to consider large values of N. Since there are 256 N possible values for each N-pixel block with an 8-bit image, the required computational resources would run scarce even for small values of N.