Source Entropy Research Articles

Non-sequential Recursive Pair Substitutions and Numerical Entropy Estimates in Symbolic Dynamical Systems

We numerically test the method of non-sequential recursive pair substitutions to estimate the entropy of an ergodic source. We compare its performance with other classical methods to estimate the entropy (empirical frequencies, return times, and Lyapunov exponent). We have considered as a benchmark for the methods several systems with different statistical properties: renewal processes, dynamical systems provided and not provided with a Markov partition, and slow or fast decay of correlations. Most experiments are supported by rigorous mathematical results, which are explained in the paper.

Universal Decremental Redundancy Compression with Fountain Codes

A new universal noise-robust lossless compression algorithm based on a decremental redundancy approach with Fountain codes is proposed. The binary entropy code is harnessed to compress complex sources with the addition of a preprocessing system in this paper. Both the whole binary entropy range compression performance and the noise-robustness of an existing incremental redundancy Fountain code compression technique are exceeded. A new autocorrelation-based symbol length estimator, the Burrows-Wheeler block sorting transform (BWT) and Move-to-Front transformation (MTF) with a new entropy ordered MTF indices transformation reduces the binary entropy of a universal data source. The preprocessed input source is coded with a new modified incremental degree LT-code (Luby Transform) and a low-complexity decremental redundancy algorithm is used to compress the Fountain-coded source. The improved compression and robustness against transmission errors with our novel incremental degree puncturing decremental redundancy algorithm is shown. The universal (complex memory source) compression performance of the proposed system is shown to achieve appreciable compression.

Channel capacity and non-uniform signalling for free-space optical intensity channels

This work considers the design of capacity-approaching, non-uniform optical intensity signalling in the presence of average and peak amplitude constraints. Although it is known that the capacity-achieving input distribution is discrete with a finite number of mass points, finding it requires complex non-linear optimization at every SNR. In this work, a simple expression for a capacity-approaching distribution is derived via source entropy maximization. The resulting mutual information using the derived discrete non-uniform input distribution is negligibly far away from the channel capacity. The computation of this distribution is substantially less complex than previous optimization approaches and can be easily computed at different SNRs. A practical algorithm for non-uniform optical intensity signalling is presented using multi-level coding followed by a mapper and multi-stage decoding at the receiver. The proposed signalling is simulated on free-space optical channels and outage capacity is analyzed. A significant gain in both rate and probability of outage is achieved compared to uniform signalling, especially in the case of channels corrupted by fog.

Entropy estimation using pattern matching in bioacoustic signals.

Many bioacoustic signals consist of a sequence of discrete stereotyped sounds occurring in repeated patterns. A natural question to ask is how best to characterize the underlying structure of the source producing the sequence of sounds. The structure of the source manifests itself as constraints on the patterns observed in the sequence of sounds. These constraints determine how predictable the order of the sounds is. The information entropy of a discrete symbol sequence is a quantitative measure of how unpredictable the sequence is. A straightforward but biased technique for estimating the entropy of an unknown source is to substitute observed symbol frequencies into parametric models such as Markov models. More general nonparametric entropy estimators exploit the relationship between the entropy and the average length of matching patterns within the sequence. This nonparametric entropy estimate forms an upper bound on the amount of information conveyed by the sequence of sounds. Additionally, comparing entropy estimates from the parametric and nonparametric models provides a hypothesis test determining whether the parametric model sufficiently captures the constraints of the source. These techniques are illustrated in analyses of humpback whale songs and leopard seal calling bouts.

On the Expected Codeword Length Per Symbol of Optimal Prefix Codes for Extended Sources

Given a discrete memoryless source <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">X</i> , it is well known that the expected codeword length per symbol <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">L</i> <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">n</sub> ( <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">X</i> ) of an optimal prefix code for the extended source <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">X</i> <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">n</sup> converges to the source entropy as <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">n</i> approaches infinity. However, the sequence <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">L</i> <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">n</sub> ( <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">X</i> ) need not be monotonic in <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">n</i> , which implies that the coding efficiency cannot be increased by simply encoding a larger block of source symbols (unless the block length is appropriately chosen). As the encoding and decoding complexity increases exponentially with the block length, from a practical perspective it is useful to know when an increase in the block length guarantees a decrease in the expected codeword length per symbol. While this paper does not provide a complete answer to that question, we give some properties of <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">L</i> <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">n</sub> ( <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">X</i> ) and obtain for each <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">n</i> ges1 and nondyadic <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">p</i> <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sub> <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">n</i> ( <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">p</i> <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sub> is the probability of the most likely source symbol) an integer <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">k</i> * for which <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">L</i> <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">kn</sub> ( <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">X</i> )< <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">L</i> <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">n</sub> ( <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">X</i> ) for all <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">k</i> ges <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">k*</i> , implying that the coding efficiency of encoding blocks of length <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">kn</i> is higher than that of encoding blocks of length <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">n</i> for all <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">k</i> ges <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">k</i> *. This question is simpler in part because <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">L</i> <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">kn</sub> ( <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">X</i> )les <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">L</i> <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">n</sub> ( <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">X</i> ) is guaranteed for all <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">n</i> ges1 and <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">k</i> ges1, but our results distinguish scenarios where increasing the multiplicative factor guarantees strict improvement. These results extend and generalize those by Montgomery and Kumar.

A One-to-One Code and Its Anti-Redundancy

One-to-one codes are ldquoone-shotrdquo codes that assign a distinct codeword to source symbols and are not necessarily prefix codes (more generally, uniquely decodable). Interestingly, as Wyner proved in 1972, for such codes the average code length can be smaller than the source entropy. By how much? We call this difference the anti-redundancy. Various authors over the years have shown that the anti-redundancy can be as big as minus the logarithm of the source entropy. However, to the best of our knowledge precise estimates do not exist. In this note, we consider a block code of length n generated for a binary memoryless source, and prove that the average anti-redundancy is -1/2 log2 n +C +F (n)+o (1) where C is a constant and either F (n) = 0 if log2(1-p)/p is irrational (where p is the probability of generating a ldquo0rdquo) or F(n) is a fluctuating function as the code length increases. This relatively simple finding requires a combination of analytic tools such as precise evaluation of Bernoulli sums, the saddle point method, and theory of distribution of sequences modulo 1.

Entropy of Patterns of i.i.d. Sequences—Part I: General Bounds

Tight bounds on the block entropy of patterns of sequences generated by independent and identically distributed (i.i.d.) sources are derived. A pattern of a sequence is a sequence of integer indices with each index representing the order of first occurrence of the respective symbol in the original sequence. Since a pattern is the result of data processing on the original sequence, its entropy cannot be larger. Bounds derived here describe the pattern entropy as function of the original i.i.d. source entropy, the alphabet size, the symbol probabilities, and their arrangement in the probability space. Matching upper and lower bounds derived provide a useful tool for very accurate approximations of pattern block entropies for various distributions, and for assessing the decrease of the pattern entropy from that of the original i.i.d. sequence.

Two Pass Improved Encoding and its Parallel Processing for Fractal Image Compression

An improvement scheme, so named the Two-Pass Improved Encoding Scheme (TIES), for the application to image compression through the extension of the existing concept of fractal image compression (FIC), which capitalizes on the self-similarity within a given image which is to be compressed, is proposed in this paper. This paper first briefly explores the existing image compression technology based on FIC, before exploring the areas which can be improved and hence establishing the concept behind the TIES algorithm. An effective encoding and decoding algorithm for the implementation of TIES is developed, through the consideration of the domain pool, block scaling and transformation, range block approximation using linear combinations and arithmetic encoding for storing data as close to source entropy as possible. The performance of TIES is then explicitly compared against that of FIC under the same conditions. Finally, due to the long encoding time required by TIES, this paper then proceeds to propose parallelized versions of the two TIES algorithms, before finally concluding with an empirical analysis of the speedup and scalability of the parallelized TIES algorithms, as well as compare the effect of parallelization between the two.

Inferring Markov chains: Bayesian estimation, model comparison, entropy rate, and out-of-class modeling

Markov chains are a natural and well understood tool for describing one-dimensional patterns in time or space. We show how to infer kth order Markov chains, for arbitrary k , from finite data by applying Bayesian methods to both parameter estimation and model-order selection. Extending existing results for multinomial models of discrete data, we connect inference to statistical mechanics through information-theoretic (type theory) techniques. We establish a direct relationship between Bayesian evidence and the partition function which allows for straightforward calculation of the expectation and variance of the conditional relative entropy and the source entropy rate. Finally, we introduce a method that uses finite data-size scaling with model-order comparison to infer the structure of out-of-class processes.

New Bounds on the Expected Length of Optimal One-to-One Codes

In this correspondence, we consider one-to-one encodings for a discrete memoryless source, which are "one-shot" encodings associating a distinct codeword with each source symbol. Such encodings could be employed when only a single source symbol rather than a sequence of source symbols needs to be transmitted. For example, such a situation can arise when the last message must be acknowledged before the next message can be transmitted. We consider two slightly different types of one-to-one encodings (depending on whether the empty codeword is used or not) and obtain lower and upper bounds on the expected length of optimal one-to-one codes. We first give an extension of a known tight lower bound on the expected length of optimal one-to-one codes for the case that the the size of the source alphabet is finite and partial information about the source symbol probabilities is available. As expected, our lower bound is no less than the previously known lower bound obtained without side information about the source symbol probabilities. We then consider the case that the source entropy is available and derive arbitrarily tight lower bounds on the expected length of optimal one-to-one codes. We also derive arbitrarily tight lower bounds for the case that the source entropy and the probability of the most likely source symbol are available. Finally, given that the probability of the most likely source symbol is available, we obtain an upper bound on the expected length of optimal one-to-one codes. Our upper bound is tighter than the best upper bound known in the literature

Error Resilient LZ'77 Data Compression: Algorithms, Analysis, and Experiments

We propose a joint source-channel coding algorithm capable of correcting some errors in the popular Lempel-Ziv'77 (LZ'77) scheme without introducing any measurable degradation in the compression performance. This can be achieved because the LZ'77 encoder does not completely eliminate the redundancy present in the input sequence. One source of redundancy can be observed when an LZ'77 phrase has multiple matches. In this case, LZ'77 can issue a pointer to any of those matches, and a particular choice carries some additional bits of information. We call a scheme with embedded redundant information the LZS'77 algorithm. We analyze the number of longest matches in such a scheme and prove that it follows the logarithmic series distribution with mean 1/h (plus some fluctuations), where h is the source entropy. Thus, the distribution associated with the number of redundant bits is well concentrated around its mean, a highly desirable property for error correction. These analytic results are proved by a combination of combinatorial, probabilistic, and analytic methods (e.g., Mellin transform, depoissonization, combinatorics on words). In fact, we analyze LZS'77 by studying the multiplicity matching parameter in a suffix tree, which in turn is analyzed via comparison to its independent version, called trie. Finally, we present an algorithm in which a channel coder (e.g., Reed-Solomon (RS) coder) succinctly uses the inherent additional redundancy left by the LZS'77 encoder to detect and correct a limited number of errors. We call such a scheme the LZRS'77 algorithm. LZRS'77 is perfectly backward-compatible with LZ'77, that is, a file compressed with our error-resistant LZRS'77 can still be decompressed by a generic LZ'77 decoder

Synthesizing folded band chaos

A randomly driven linear filter that synthesizes Lorenz-like, reverse-time chaos is shown also to produce Rössler-like folded band wave forms when driven using a different encoding of the random source. The relationship between the topological entropy of the random source, dissipation in the linear filter, and the positive Lyapunov exponent for the reverse-time wave form is exposed. The two drive encodings are viewed as grammar restrictions on a more general encoding that produces a chaotic superset encompassing both the Lorenz butterfly and Rössler folded band paradigms of nonlinear dynamics.

Design of lossless turbo source encoders

Lossless turbo source coding with decremental redundancy is an effective approach for compressing binary sources. A large block length lossless turbo source encoder offers compression rates close to the source entropy but with large latency. In this letter, we propose a lossless compression technique for binary memoryless sources using short block length turbo codes. To achieve compression rates close to the source entropy, we modify different components of the encoder. We focus on the design of the parity interleaver for different compression rates. Also, we replace the square shape puncturing array with a rectangular shape array that allows finer puncturing and hence improved compression rates. Finally, instead of a single code, we employ many codes operating in parallel. Given these modifications, we evaluate the encoding complexity of the proposed code

Code optimisation for lossless compression of binary memoryless sources based on FEC codes

AbstractA novel source coding scheme based on turbo codes was recently presented. Lossless data compression is thereby achieved by puncturing data encoded with a turbo code while checking the integrity of the reconstructed information during compression. This paper addresses the generalisation of this compression scheme to iteratively decodable forward error correcting (FEC) codes. Furthermore, an alternative puncturing algorithm is presented which fine‐tunes the compression rate to any desired accuracy. Motivated by an area property which states that the area of the tunnel in the modified extrinsic information transfer (EXIT) chart is proportional to the gap between source coding rate and entropy we apply code optimisation tools to serially concatenated codes in order to minimise the tunnel. Numerical results show that compression rates close to the Shannon limit can be obtained by optimised irregular repeat accumulate codes. Copyright © 2006 AEIT.

On the average depth of asymmetric LC-tries

Andersson and Nilsson have already shown that the average depth D n of random LC-tries is only Θ ( log ∗ n ) when the keys are produced by a symmetric memoryless process, and that D n = O ( log log n ) when the process is asymmetric. In this paper we refine the second estimate by showing that asymptotically (with n → ∞ ): D n ∼ 1 η log log n , where n is the number of keys inserted in a trie, η = − log ( 1 − h / h − ∞ ) , h = − p log p − q log q is the entropy of a binary memoryless source with probabilities p, q = 1 − p ( p ≠ q ), and h − ∞ = − log min ( p , q ) .

Exponent parameter estimation for generalized Gaussian probability density functions with application to speech modeling

In this contribution, we exploit entropy matching to estimate the exponent parameter of a generalized Gaussian density. Based on this premise, we derive a new entropic expression with respect to higher-order moments of the modeled data, which yields a novel generalized source entropy matching estimator (G-EME). A number of other popular statistical methods are also reviewed, described and compared against the proposed technique. Extensive comparative experimental results illustrate the high accuracy of the proposed estimator, for both light- and heavy-tailed distributions, as well as speech data.

Information Theory and its Two basic Applications

Entropy, relative entropy and mutual information which are three fundamental quantities in information theory are discussed. The difference between entropy of an information source and thermodynamic entropy is given. The meaning of the term information source is discussed in detail. Two applications of information theory: data compression and data transmission are given. Entropy also gives the lower limit up to which lossless compression can be achieved. When rate of transmission is less than or equal to the capacity of the channel, reliable communication can be obtained on a noisy channel. The theoretical limits given by information theory motivate researchers to find new practical systems to reach closer to these limits.

SYNCHRONIZING TO PERIODICITY: THE TRANSIENT INFORMATION AND SYNCHRONIZATION TIME OF PERIODIC SEQUENCES

We analyze how difficult it is to synchronize to a periodic sequence whose structure is known, when an observer is initially unaware of the sequence's phase. We examine the transient information T, a recently introduced information-theoretic quantity that measures the uncertainty an observer experiences while synchronizing to a sequence. We also consider the synchronization time τ, which is the average number of measurements required to infer the phase of a periodic signal. We calculate T and τ for all periodic sequences up to and including period 23. We show which sequences of a given period have the maximum and minimum possible T and τ values, develop analytic expressions for the extreme values, and show that in these cases the transient information is the product of the total phase information and the synchronization time. Despite the latter result, our analyses demonstrate that the transient information and synchronization time capture different and complementary structural properties of individual periodic sequences — properties, moreover, that are distinct from source entropy rate and mutual information measures, such as the excess entropy.

An entropy estimator improving mean squared error

AbstractEntropy estimation for a memory‐less information source has mainly been performed by the following process under the condition that the occurrence probability of each source symbol is unknown: First, the occurrence probabilities of source symbols are estimated; then the entropy is estimated by substituting the estimated occurrence probabilities into the entropy function. Such an entropy estimation is not optimum in the sense of least squares error; it causes a large error, particularly for small sample size. In this paper, we propose an entropy estimator that strictly minimizes the mean squared error for a two‐ary memory‐less information source; it is formulated as a function of the number of one type of source symbols contained in the sample set. Furthermore, we propose a technique for estimating entropy of an arbitrary M‐ary memory‐less information source by recursively applying the proposed entropy estimator for a two‐ary memory‐less information source. Numerical experiments show that this technique improves mean squared error compared to the conventional method even though the guarantee that it is the least squares error estimator has been lost. © 2004 Wiley Periodicals, Inc. Electron Comm Jpn Pt 3, 87(9): 1–10, 2004; Published online in Wiley InterScience (www.interscience.wiley.com). DOI 10.1002/ecjc.10163

Improved bounds for the rate loss of multiresolution source codes

We present new bounds for the rate loss of multiresolution source codes (MRSCs). Considering an M-resolution code, the rate loss at the ith resolution with distortion D/sub i/ is defined as L/sub i/=R/sub i/-R(D/sub i/), where R/sub i/ is the rate achievable by the MRSC at stage i. This rate loss describes the performance degradation of the MRSC compared to the best single-resolution code with the same distortion. For two-resolution source codes, there are three scenarios of particular interest: (i) when both resolutions are equally important; (ii) when the rate loss at the first resolution is 0 (L/sub 1/=0); (iii) when the rate loss at the second resolution is 0 (L/sub 2/=0). The work of Lastras and Berger (see ibid., vol.47, p.918-26, Mar. 2001) gives constant upper bounds for the rate loss of an arbitrary memoryless source in scenarios (i) and (ii) and an asymptotic bound for scenario (iii) as D/sub 2/ approaches 0. We focus on the squared error distortion measure and (a) prove that for scenario (iii) L/sub 1/<1.1610 for all D/sub 2/<D/sub 1/; (b) tighten the Lastras-Berger bound for scenario (ii) from L/sub 2//spl les/1 to L/sub 2/<0.7250; (c) tighten the Lastras-Berger bound for scenario (i) from L/sub i//spl les/1/2 to L/sub i/<0.3802, i/spl isin/{1,2}; and (d) generalize the bounds for scenarios (ii) and (iii) to M-resolution codes with M/spl ges/2. We also present upper bounds for the rate losses of additive MRSCs (AMRSCs). An AMRSC is a special MRSC where each resolution describes an incremental reproduction and the kth-resolution reconstruction equals the sum of the first k incremental reproductions. We obtain two bounds on the rate loss of AMRSCs: one primarily good for low-rate coding and another which depends on the source entropy.