Natural type selection in adaptive lossy compression

Abstract

Consider approximate (lossy) matching of a source string /spl sim/P, with a random codebook generated from reproduction distribution Q, at a specified distortion d. Previous work determined the minimum coding rate R/sub 1/=R(P, Q, d) for this setting. We observe that for a large word length and with high probability, the matching codeword is typical with a distribution Q/sub 1/ which is different from Q. If a new random codebook is generated /spl sim/Q/sub 1/, then the source string will favor codewords which are typical with a new distribution Q/sub 2/, resulting in a minimum coding rate R/sub 2/=R(P, Q/sub 1/, d), and so on. We show that the sequences of distributions Q/sub 1/, Q/sub 2/,... and rates R/sub 1/, R/sub 2/,..., generated by this procedure, converge to an optimum reproduction distribution Q*, and the rate-distortion function R(P, d), respectively. We also derive a fixed rate-distortion slope version of this natural type selection process. In the latter case, an iteration of the process stochastically simulates an iteration of the Blahut-Arimoto (1972) algorithm for rate-distortion function computation (without recourse to prior knowledge of the underlying source distribution). To strengthen these limit statements, we also characterize the steady-state error of these procedures when iterating at a finite string length. Implications of the main results provide fresh insights into the workings of lossy variants of the Lempel-Ziv algorithm for adaptive compression.