Order Markov Sources Research Articles

Empirical Models for the Realistic Generation of Cooperative Awareness Messages in Vehicular Networks

Most V2X (Vehicle-to-Everything) applications rely on broadcasting awareness messages known as CAM (Cooperative Awareness Messages) in ETSI or BSM (Basic Safety Message) in SAE standards. A large number of studies have been devoted to guarantee their reliable transmission. However, to date, the studies are generally based on simplified data traffic models that generate awareness messages at periodic intervals or with a constant message size. These models do not accurately represent the real generation of CAM messages that follow specific mobility-based rules. Using simplified and unrealistic traffic models can significantly impact the results and validity of the studies, and hence accurate models for the generation of awareness messages are necessary. This paper proposes the first set of models that can realistically generate CAM messages. The models have been created from real traces collected by two car manufacturers in urban, sub-urban and highway test drives. The models are based on mth order Markov sources, and model the size of CAMs and the time interval between CAMs. The models are openly provided to the community and can be easily integrated into any simulator.

Lossless Data Compression via Substring Enumeration for k-th Order Markov Sources with a Finite Alphabet

Dube and Beaudoin have proposed a technique of lossless data compression called compression via substring enumeration (CSE) for a binary source alphabet. Dube and Yokoo proved that CSE has a linear complexity both in time and in space worst-case performance for the length of string to be encoded. Dube and Yokoo have specified appropriate predictors of the uniform and combinatorial prediction models for CSE, and proved that CSE has the asymptotic optimality for stationary binary ergodic sources. Our previous study evaluated the worst-case maximum redundancy of the modified CSE for an arbitrary binary string from the class of k-th order Markov sources. We propose a generalization of CSE for k-th order Markov sources with a finite alphabet X based on Ota and Morita in this study.

Counting Markov Types, Balanced Matrices, and Eulerian Graphs

The method of types is one of the most popular techniques in information theory and combinatorics. Two sequences of equal length have the same type if they have identical empirical distributions. In this paper, we focus on Markov types, that is, sequences generated by a Markov source (of order one). We note that sequences having the same Markov type share the same so-called balanced frequency matrix that counts the number of distinct pairs of symbols. We enumerate the number of Markov types for sequences of length over an alphabet of size . This turns out to be asymptotically equivalent to estimating the number of the balanced frequency matrices, the number of integer solutions of a system of linear Diophantine equations, and the number of connected Eulerian multigraphs. For fixed , we prove that the number of Markov types is asymptotically equal to d(m) n <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">m</sup> <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">-m</sup> /(m <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> -m)! where we give an integral representation for d(m). For m →∞, we conclude that asymptotically the number of types is equivalent to √2m <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">3m/2</sup> e <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">m</sup> <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> /m <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2m2</sup> 2 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">m</sup> π <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">m/2</sup> n <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">m2</sup> -m provided that m = o(n <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1/4</sup> ). These findings are derived by analytical techniques ranging from analytic combinatorics, to multidimensional generating functions, to the saddle point method.

On Delayed Sequential Coding of Correlated Sources

Motivated by video coding applications, the problem of sequential coding of correlated sources with encoding and/or decoding frame-delays is studied. The fundamental tradeoffs between individual frame rates, individual frame distortions, and encoding/decoding frame-delays are derived in terms of a single-letter information-theoretic characterization of the rate-distortion region for general interframe source correlations and certain types of potentially frame specific and coupled single-letter fidelity criteria. The sum-rate-distortion region is characterized in terms of generalized directed information measures highlighting their role in delayed sequential source coding problems. For video sources which are spatially stationary memoryless and temporally Gauss–Markov, MSE frame distortions, and a sum-rate constraint, our results expose the optimality of idealized differential predictive coding among all causal sequential coders, when the encoder uses a positive rate to describe each frame. Somewhat surprisingly, causal sequential encoding with one-frame-delayed noncausal sequential decoding can exactly match the sum-rate-MSE performance of joint coding for all nontrivial MSE-tuples satisfying certain positive semidefiniteness conditions. Thus, even a single frame-delay holds potential for yielding significant performance improvements. Generalizations to higher order Markov sources are also presented and discussed. A rate-distortion performance equivalence between, causal sequential encoding with delayed noncausal sequential decoding, and delayed noncausal sequential encoding with causal sequential decoding, is also established.

On the sysRatio and its critical point

What is the relationship between the complexity of a learner and the randomness of his mistakes? This question was posed in Ratsaby (2009) [8] who showed that the more complex the learner the higher the possibility that his mistakes deviate from a true random sequence. In the current paper we report on an empirical investigation of this problem. We investigate two characteristics of randomness, the stochastic and algorithmic complexity of the binary sequence of mistakes. A learner with a Markov model of order k is trained on a finite binary sequence produced by a Markov source of order k∗ and is tested on a different random sequence. As a measure of learner’s complexity we define a quantity called the sysRatio, denoted by ρ, which is the ratio between the compressed and uncompressed lengths of the binary string whose ith bit represents the maximum a posteriori decision made at state i of the learner’s model. The quantity ρ is a measure of information density. The main result of the paper shows that this ratio is crucial in answering the above posed question. The result indicates that there is a critical threshold ρ∗ such that when ρ≤ρ∗ the sequence of mistakes possesses the following features: (1) low divergence Δ from a random sequence, (2) low variance in algorithmic complexity. When ρ>ρ∗, the characteristics of the mistake sequence changes sharply towards a high Δ and high variance in algorithmic complexity. It is also shown that the quantity ρ is inversely proportional to k and the value of ρ∗ corresponds to the value k∗. This is the point where the learner’s model becomes too simple and is unable to approximate the Bayes optimal decision. Here the characteristics of the mistake sequence change sharply.

Twice-Universal Simulation of Markov Sources and Individual Sequences

The problem of universal simulation given a training sequence is studied both in a stochastic setting and for individual sequences. In the stochastic setting, the training sequence is assumed to be emitted by a Markov source of unknown order, extending previous work where the order is assumed known and leading to the notion of twice-universal simulation. A simulation scheme, which partitions the set of sequences of a given length into classes, is proposed for this setting and shown to be asymptotically optimal. This partition extends the notion of type classes to the twice-universal setting. In the individual sequence scenario, the same simulation scheme is shown to generate sequences which are statistically similar, in a strong sense, to the training sequence, for statistics of any order, while essentially maximizing the uncertainty on the output.

Counting Markov Types

The method of types is one of the most popular techniques in information theory and combinatorics. Two sequences of equal length have the same type if they have identical empirical distributions. In this paper, we focus on Markov types, that is, sequences generated by a Markov source (of order one). We note that sequences having the same Markov type share the same so called $\textit{balanced frequency matrix}$ that counts the number of distinct pairs of symbols. We enumerate the number of Markov types for sequences of length $n$ over an alphabet of size $m$. This turns out to coincide with the number of the balanced frequency matrices as well as with the number of special $\textit{linear diophantine equations}$, and also balanced directed multigraphs. For fixed $m$ we prove that the number of Markov types is asymptotically equal to $d(m) \frac{n^{m^{2-m}}}{(m^2-m)!}$, where $d(m)$ is a constant for which we give an integral representation. For $m \to \infty$ we conclude that asymptotically the number of types is equivalent to $\frac{\sqrt{2}m^{3m/2} e^{m^2}}{m^{2m^2} 2^m \pi^{m/2}} n^{m^2-m}$ provided that $m=o(n^{1/4})$ (however, our techniques work for $m=o(\sqrt{n})$). These findings are derived by analytical techniques ranging from multidimensional generating functions to the saddle point method.

On the Construction of (Explicit) Khodak's Code and Its Analysis

Variable-to-variable (VV) codes are very attractive yet not well understood data compression schemes. In 1972, Khodak claimed to provide upper and lower bounds for the achievable redundancy rate, however, he did not offer explicit construction of such codes. In this paper, we first present a constructive and transparent proof of Khodak's result showing that for memoryless sources there exists a code with the average redundancy bounded by <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">D</i> <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">-5/3</sup> , where <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">D</i> is the average delay (e.g., the average length of a dictionary entry). We also describe an algorithm that constructs a VV length code with a small redundancy rate for large <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">D</i> . Then, we discuss several generalizations. We prove that the worst case redundancy does not exceed <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">D</i> <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">-4/3</sup> . Furthermore, we provide similar upper bound for Markov sources (of order 1). Finally, we consider bounds that are valid for <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">almost</i> <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">all</i> memoryless and Markov sources for which the set of exceptional source parameters has zero measure. In particular, for all memoryless sources outside this exceptional class, we prove there exists a VV code with the average redundancy rate bounded by <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">D</i> <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">-1-m/3+</sup> <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">epsiv</sup> and the worst case redundancy rate bounded by <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">D</i> <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">-1-m/3+</sup> <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">epsiv</sup> , where <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">m</i> is the cardinality of the alphabet. We complete our analysis with a lower bound showing that for all VV codes the average and the worst case redundancy rates are at least <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">D</i> <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">-2m-1-</sup> <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">epsiv</sup> for almost all memoryless sources in the sense that the set of exceptional source parameters has zero measure. We prove these results using techniques of Diophantine approximations.

Symbol-Based Modeling and Coding of Block Markov Sources

Industry-standard lossless compression algorithms (such as LZW) are usually implemented so that they work on bytes as symbols. Experiments indicate that data for which bytes are not the natural choice of symbols compress poorly using these implementations, while algorithms working on a bit level perform reasonably on byte-based data in addition to having computational advantages resulting from operating on a small alphabet. In this correspondence, we offer an information-theoretic explanation to these experimental results by assessing the redundancy (which is approximated by the divergence rate of two source distributions) of a bit-based model when applied to a byte-based source. More specifically, we study the problem of approximating a block Markov source (our model for byte-based data) with higher order Markov sources (which model bit-based Markov encoders), and show that the divergence rate between a block Markov source and the best matching higher order Markov model for that source converges to zero exponentially fast as the memory of the model increases. This result is applied to obtain bounds on the redundancy of certain symbol-based universal codes when they are used for byte-aligned sources

Redundancy of the Context-Tree Weighting Method on Renewal and Markov Renewal Processes

<para xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> The &lt;emphasis emphasistype="italic"&gt;Context Tree Weighting&lt;/emphasis&gt; method (CTW) is shown to be almost &lt;emphasis emphasistype="italic"&gt;adaptive&lt;/emphasis&gt; on the classes of &lt;emphasis emphasistype="italic"&gt;renewal&lt;/emphasis&gt; and &lt;emphasis emphasistype="italic"&gt;Markov renewal&lt;/emphasis&gt; processes. Up to logarithmic factor, &lt;emphasis emphasistype="smcaps"&gt;ctw&lt;/emphasis&gt; achieves the minimax pointwise redundancy described by I. Csiszár and P. Shields in &lt;emphasis emphasistype="boldital"&gt;IEEE Trans. Inf. Theory&lt;/emphasis&gt;, vol. 42, no. 6, pp. 2065–2072, Nov. 1996. This result not only complements previous results on the adaptivity of the Context-Tree Weighting method on the relatively small class of all finite context-tree sources (which encompasses the class of all finite order Markov sources), it shows that almost minimax redundancy can be achieved on massive classes of sources (classes that cannot be smoothly parameterized by subsets of finite-dimensional spaces). Moreover, it shows that (almost) adaptive compression can be achieved in a computationally efficient way on those massive classes. While previous adaptivity results for CTW could rely on the fact that any Markov source is a finite-context-tree source, this is no longer the case for renewal sources. In order to prove almost adaptivity of CTW over renewal sources, it is necessary to establish that CTW carefully balances estimation error and approximation error. </para>

Universal codes as a basis for nonparametric testing of serial independence for time series

We address the problem of nonparametric testing of serial independence for time series and its generalization. More precisely, we consider a stationary and ergodic source p , which generates symbols x 1 … x t from some finite set A and a null hypothesis H 0 that p is a Markov source of order at most m , ( m ⩾ 0 ) . The alternative hypothesis H 1 is that the sequence is generated by a stationary and ergodic source, which differs from the source under H 0 . In particular, if m = 0 we have the null hypothesis H 0 that the sequence is generated by a Bernoulli source (i.e. the hypothesis that x 1 … x t are independent). In this paper some new tests that are based on so-called universal codes and universal predictors, are suggested.

Markov Types and Minimax Redundancy for Markov Sources

Redundancy of universal codes for a class of sources determines by how much the actual code length exceeds the optimal code length. In the minimax scenario, one designs the best code for the worst source within the class. Such minimax redundancy comes in two flavors: average minimax or worst case minimax. We study the worst case minimax redundancy of universal block codes for Markovian sources of any order. We prove that the maximal minimax redundancy for Markov sources of order r is asymptotically equal to 1/2m/sup r/(m-1)log/sub 2/n+log/sub 2/A/sub m//sup r/-(lnlnm/sup 1/(m-1)/)/lnm+o(1), where n is the length of a source sequence, m is the size of the alphabet, and A/sub m//sup r/ is an explicit constant (e.g., we find that for a binary alphabet m=2 and Markov of order r=1 the constant A/sub 2//sup 1/=16/spl middot/G/spl ap/14.655449504 where G is the Catalan number). Unlike previous attempts, we view the redundancy problem as an asymptotic evaluation of certain sums over a set of matrices representing Markov types. The enumeration of Markov types is accomplished by reducing it to counting Eulerian paths in a multigraph. In particular, we propose exact and asymptotic formulas for the number of strings of a given Markov type. All of these findings are obtained by analytic and combinatorial tools of analysis of algorithms.

The Kullback–Leibler Divergence Rate Between Markov Sources

In this work, we provide a computable expression for the Kullback-Leibler divergence rate lim/sub n/spl rarr//spl infin//1/nD(p/sup (n)//spl par/q/sup (n)/) between two time-invariant finite-alphabet Markov sources of arbitrary order and arbitrary initial distributions described by the probability distributions p/sup (n)/ and q/sup (n)/, respectively. We illustrate it numerically and examine its rate of convergence. The main tools used to obtain the Kullback-Leibler divergence rate and its rate of convergence are the theory of nonnegative matrices and Perron-Frobenius theory. Similarly, we provide a formula for the Shannon entropy rate lim/sub n/spl rarr//spl infin//1/nH(p/sup (n)/) of Markov sources and examine its rate of convergence.

Renyi's divergence and entropy rates for finite alphabet Markov sources

In this work, we examine the existence and the computation of the Renyi divergence rate, lim/sub n/spl rarr//spl infin// 1/n D/sub /spl alpha//(p/sup (n)//spl par/q/sup (n)/), between two time-invariant finite-alphabet Markov sources of arbitrary order and arbitrary initial distributions described by the probability distributions p/sup (n)/ and q/sup (n)/, respectively. This yields a generalization of a result of Nemetz (1974) where he assumed that the initial probabilities under p/sup (n)/ and q/sup (n)/ are strictly positive. The main tools used to obtain the Renyi divergence rate are the theory of nonnegative matrices and Perron-Frobenius theory. We also provide numerical examples and investigate the limits of the Renyi divergence rate as /spl alpha//spl rarr/1 and as /spl alpha//spl darr/0. Similarly, we provide a formula for the Renyi entropy rate lim/sub n/spl rarr//spl infin// 1/n H/sub /spl alpha//(p/sup (n)/) of Markov sources and examine its limits as /spl alpha//spl rarr/1 and as /spl alpha//spl darr/0. Finally, we briefly provide an application to source coding.

Redundancy of the Krichevsky-Trofimov estimator with a finite window for a Markov source

The Krichevsky–Trofimov estimator can be implemented as an arithmetic data compressor based on a finite window. We analyze the redundancy of this estimator for the Markov source of finite order. The per-letter redundancy behaves with near certainty as (the number of effective states) × (alphabet size)/2 (window size) when the window size is large. Our analysis applies the renewal theory. © 2000 Scripta Technica, Electron Comm Jpn Pt 3, 83(11): 56–62, 2000

Significance-linked connected component analysis for very low bit-rate wavelet video coding

A novel hybrid wavelet video coding algorithm termed video significance-linked connected component analysis (VSLCCA) is developed for very low bit-rate applications. In the proposed VSLCCA codec, first, fine-tuned motion estimation based on the H.263 Recommendation is developed to reduce temporal redundancy, and exhaustive overlapped block motion compensation is utilized to ensure coherency in motion compensated error frames. Second, the wavelet transform is applied to each coherent motion compensated error frame to attain global energy compaction. Third, significant fields of wavelet-transformed error frames are organized and represented as significance-linked connected components so that both the within-subband clustering and the cross-scale dependency are exploited. Last, the horizontal and vertical components of motion vectors are encoded separately using adaptive arithmetic coding while significant wavelet coefficients are encoded in bit-plane order by using high order Markov source modeling and adaptive arithmetic coding. Experimental results on eight standard MPEG-4 test sequences show that for intraframe coding, on average the proposed codec exceeds H.263 and ZTE (zero-tree entropy) in peak signal-to-noise ratio by as much as 2.07 and 1.38 dB at 28 kbit/s, respectively. For entire sequence coding, VSLCCA is superior to H.263 and ZTE by 0.35 and 0.71 dB on average, respectively.

Optimal prefetching via data compression

Caching and prefetching are important mechanisms for speeding up access time to data on secondary storage. Recent work in competitive online algorithms has uncovered several promising new algorithms for caching. In this paper, we apply a form of the competitive philosophy for the first time to the problem of prefetching to develop an optimal universal prefetcher in terms of fault rate, with particular applications to large-scale databases and hypertext systems. Our prediction algorithms with particular applications to large-scale databases and hypertext systems. Our prediction algorithms for prefetching are novel in that they are based on data compression techniques that are both theoretically optimal and good in practice. Intuitively, in order to compress data effectively, you have to be able to predict future data well, and thus good data compressors should be able to predict well for purposes of prefetching. We show for powerful models such as Markov sources and m the order Markov sources that the page fault rate incurred by our prefetching algorithms are optimal in the limit for almost all sequences of page requests.

On the Gaarder-Slepian "tracking system" conjecture (source coding)

The authors examine under what conditions, and with what notion of optimality, the coder of an optimal system can be operated with the information which is also accessible to the decoder. After a discussion of the difficulties involved, a theorem is proved for Markov sources which states that the extra memory of the coder can be substituted with independent randomization. The result is not tied to any specific notion of optimality. The main result is shown to hold for Markov sources of order k under weak assumptions on the structure and with respect to any per-letter fidelity criterion. >

Fish/Gear-Space System

This study investigates the following problems. The first problem is to describe a model of a set-net as a fish/gear-space system. The second problem is to develop a method for deriving probable swimming paths. Third problem is to evaluate the probable swimming paths. Research was carried out in a water tank using Yamame onchorhynchus masou. This was a relatively easy means for obtaining data on the ecology of fish and long observation records of fish behavior. Fish behavior can be termed as an m-th order Markov process. Therefore, the fish/gear-space system was described as an m-th order Markov source. The probable swimming paths were derived from the joint probabilities of 3 sequential zones(mesh zones) in this study because fish swimming through 3 or more zones appeared to have a chosen path, or direction, in mind. Those were evaluated by using the entropy as defined by the author. In conclusion, it appears that the above model, the method for deriving probable swimming paths and the evaluating method are applicable for investigating fish behavior in relation to any given fish/gear-space system

トラップ内での魚の行動解析に関する一考察

This study investigates the following problems. The first problem is to describe a model of a set-net as a fish/gear-space system. The second problem is to develop a methodfor derivin gprobable swimming paths, and the third is to evaluate the probable swimming paths. Research was carried out in a water tank using Yamame Onchorhynchus masou ahd Ugui Tribolodon hakonensis. This was a relatively easy means for obtaining data on the ecology of fish and long observation records of fish behavior. Fish behavior can be termed as an m-th order Markov process. Therefore, the fish/gear-space system was described as an m-th order Markov source. The probable swimming paths were derived from the joint probability of 3 sequential zones (mesh zones)in this study because fish swimming through 3 or more zones appeared to have a chosen path, or direction, in mind. Those were evaluated by using the entropy as defined by the author. In conclusion, it appears that the above model, the method for deriving probable swimmingpaths and the evaluating method are applicable for investigating fish behavior in relation to any given fish/gear-space system.