Countable Alphabet Research Articles

Confidence Intervals for Parameters of Unobserved Events

Consider a finite sample from an unknown distribution over a countable alphabet. Unobserved events are alphabet symbols which do not appear in the sample. Estimating the probabilities of unobserved events is a basic problem in statistics and related fields, which was extensively studied in the context of point estimation. In this work we introduce a novel interval estimation scheme for unobserved events. Our proposed framework applies selective inference, as we construct confidence intervals (CIs) for the desired set of parameters. Interestingly, we show that obtained CIs are dimension-free, as they do not grow with the alphabet size. Further, we show that these CIs are (almost) tight, in the sense that they cannot be further improved without violating the prescribed coverage rate. We demonstrate the performance of our proposed scheme in synthetic and real-world experiments, showing a significant improvement over the alternatives. Finally, we apply our proposed scheme to large alphabet modeling. We introduce a novel simultaneous CI scheme for large alphabet distributions which outperforms currently known methods while maintaining the prescribed coverage rate. Supplementary materials for this article are available online including a standardized description of the materials available for reproducing the work.

Countable alphabet stationary processes with at least one memory word and intermittent estimation with universal rates

Countable alphabet stationary processes with at least one memory word and intermittent estimation with universal rates

Invariant measures for substitutions on countable alphabets

AbstractIn this work, we study ergodic and dynamical properties of symbolic dynamical system associated to substitutions on an infinite countable alphabet. Specifically, we consider shift dynamical systems associated to irreducible substitutions which have well-established properties in the case of finite alphabets. Based on dynamical properties of a countable integer matrix related to the substitution, we obtain results on existence and uniqueness of shift invariant measures.

Universal Densities Exist for Every Finite Reference Measure

As it is known, universal codes, which estimate the entropy rate consistently, exist for stationary ergodic sources over finite alphabets but not over countably infinite ones. We generalize universal coding as the problem of universal densities with respect to a fixed reference measure on a countably generated measurable space. We show that universal densities, which estimate the differential entropy rate consistently, exist for finite reference measures. Thus finite alphabets are not necessary in some sense. To exhibit a universal density, we adapt the nonparametric differential (NPD) entropy rate estimator by Feutrill and Roughan. Our modification is analogous to Ryabko’s modification of prediction by partial matching (PPM) by Cleary and Witten. Whereas Ryabko considered a mixture over Markov orders, we consider a mixture over quantization levels. Moreover, we demonstrate that any universal density induces a strongly consistent Cesàro mean estimator of conditional density given an infinite past. This yields a universal predictor with the 0–1 loss for a countable alphabet. Finally, we specialize universal densities to processes over natural numbers and on the real line. We derive sufficient conditions for consistent estimation of the entropy rate with respect to infinite reference measures in these domains.

Asymptotic Normality for Plug-In Estimators of Generalized Shannon's Entropy.

Shannon’s entropy is one of the building blocks of information theory and an essential aspect of Machine Learning (ML) methods (e.g., Random Forests). Yet, it is only finitely defined for distributions with fast decaying tails on a countable alphabet. The unboundedness of Shannon’s entropy over the general class of all distributions on an alphabet prevents its potential utility from being fully realized. To fill the void in the foundation of information theory, Zhang (2020) proposed generalized Shannon’s entropy, which is finitely defined everywhere. The plug-in estimator, adopted in almost all entropy-based ML method packages, is one of the most popular approaches to estimating Shannon’s entropy. The asymptotic distribution for Shannon’s entropy’s plug-in estimator was well studied in the existing literature. This paper studies the asymptotic properties for the plug-in estimator of generalized Shannon’s entropy on countable alphabets. The developed asymptotic properties require no assumptions on the original distribution. The proposed asymptotic properties allow for interval estimation and statistical tests with generalized Shannon’s entropy.

Generalized Good-Turing Improves Missing Mass Estimation

Consider a finite sample from an unknown distribution over a countable alphabet. The missing mass refers to the probability of symbols that do not appear in the sample. Estimating the missing mass is a basic problem in statistics and related fields, which dates back to the early work of Laplace, and the more recent seminal contribution of Good and Turing. In this article, we introduce a generalized Good-Turing (GT) framework for missing mass estimation. We derive an upper-bound for the risk (in terms of mean squared error) and minimize it over the parameters of our framework. Our analysis distinguishes between two setups, depending on the (unknown) alphabet size. When the alphabet size is bounded from above, our risk-bound demonstrates a significant improvement compared to currently known results (which are typically oblivious to the alphabet size). Based on this bound, we introduce a numerically obtained estimator that improves upon GT. When the alphabet size holds no restrictions, we apply our suggested risk-bound and introduce a closed-form estimator that again improves GT performance guarantees. Our suggested framework is easy to apply and does not require additional modeling assumptions. This makes it a favorable choice for practical applications. Supplementary materials for this article are available online.

Potential Well in Poincaré Recurrence

From a physical/dynamical system perspective, the potential well represents the proportional mass of points that escape the neighbourhood of a given point. In the last 20 years, several works have shown the importance of this quantity to obtain precise approximations for several recurrence time distributions in mixing stochastic processes and dynamical systems. Besides providing a review of the different scaling factors used in the literature in recurrence times, the present work contributes two new results: (1) For -mixing and -mixing processes, we give a new exponential approximation for hitting and return times using the potential well as the scaling parameter. The error terms are explicit and sharp. (2) We analyse the uniform positivity of the potential well. Our results apply to processes on countable alphabets and do not assume a complete grammar.

On generalized compensation functions for factor maps between shift spaces on countable alphabets

We show the existence of generalized compensation functions for a particular type of one-block factor maps [Formula: see text] between countable subshifts [Formula: see text] and [Formula: see text]. For factor maps between compact spaces, continuous compensation functions were studied by Walters in relation to the theory of relative pressure. Applying the thermodynamic formalism for sequences on countable subshifts, we generalize some existing results on factor maps between compact spaces to non-compact spaces. For related questions, we also study the existence of a preimage measure on [Formula: see text] of an invariant measure on [Formula: see text], and their relations.

Generalized Mutual Information

Mutual information is one of the essential building blocks of information theory. It is however only finitely defined for distributions in a subclass of the general class of all distributions on a joint alphabet. The unboundedness of mutual information prevents its potential utility from being extended to the general class. This is in fact a void in the foundation of information theory that needs to be filled. This article proposes a family of generalized mutual information whose members are indexed by a positive integer n, with the nth member being the mutual information of nth order. The mutual information of the first order coincides with Shannon’s, which may or may not be finite. It is however established (a) that each mutual information of an order greater than 1 is finitely defined for all distributions of two random elements on a joint countable alphabet, and (b) that each and every member of the family enjoys all the utilities of a finite Shannon’s mutual information.

Borel complexity of sets of normal numbers via generic points in subshifts with specification

We study the Borel complexity of sets of normal numbers in several numeration systems. Taking a dynamical point of view, we offer a unified treatment for continued fraction expansions and base r r expansions, and their various generalisations: generalised Lüroth series expansions and β \beta -expansions. In fact, we consider subshifts over a countable alphabet generated by all possible expansions of numbers in [ 0 , 1 ) [0,1) . Then normal numbers correspond to generic points of shift-invariant measures. It turns out that for these subshifts the set of generic points for a shift-invariant probability measure is precisely at the third level of the Borel hierarchy (it is a Π 3 0 \boldsymbol {\Pi }^0_3 -complete set, meaning that it is a countable intersection of F σ F_\sigma -sets, but it is not possible to write it as a countable union of G δ G_\delta -sets). We also solve a problem of Sharkovsky–Sivak on the Borel complexity of the basin of statistical attraction. The crucial dynamical feature we need is a feeble form of specification. All expansions named above generate subshifts with this property. Hence the sets of normal numbers under consideration are Π 3 0 \boldsymbol {\Pi }^0_3 -complete.

Long hitting times for expanding systems

We prove a new result in the area of hitting time statistics. Currently, there is a lot of papers showing that the first entry times into cylinders or balls are often faster than the Birkhoff’s Ergodic theorem would suggest. We provide an opposite counterpart to these results by proving that the hitting times into shrinking balls are also often much larger than these theorems would suggest, by showing that for many expanding dynamical systemsfor an appropriately large, at least of full measure, set of points y and x.We first do this for all transitive open distance expanding maps and Gibbs/equilibrium states of Hölder continuous potentials; in particular for all irreducible subshifts of finite type with a finite alphabet. Then we prove such result for all finitely irreducible subshifts of finite type with a countable alphabet and Gibbs/equilibrium states for Hölder continuous summable potentials. Next, we show that the limsup result holds for all graph directed Markov systems (far going natural generalizations of iterated function systems) and projections of aforementioned Gibbs states on their limit sets. By utilizing the first return map techniques, we then prove the limsup result for all tame topological Collect–Eckmann multimodal maps of an interval, all tame topological Collect–Eckmann rational functions of the Riemann sphere, and all dynamically semi-regular transcendental meromorphic functions from to .

Hidden Gibbs measures on shift spaces over countable alphabets

We study the thermodynamic formalism for particular types of sub-additive sequences on a class of subshifts over countable alphabets. The subshifts we consider include factors of irreducible countable Markov shifts under certain conditions, which we call irreducible countable sofic shifts. We show the variational principle for topological pressure for some sub-additive sequences with tempered variation on irreducible countable sofic shifts. We also study conditions for the existence and uniqueness of invariant ergodic Gibbs measures and the uniqueness of equilibrium states. Applications are given to some dimension problems and study of factors of (generalized) Gibbs measures on certain subshifts over countable alphabets.

Time change for flows and thermodynamic formalism

This paper is devoted to the study of how thermodynamic formalism properties vary for time changes of suspension flows defined over countable Markov shifts. We prove that in general no property is preserved. We also make a topological description of the space of suspension flows according to certain thermodynamic features. For example, we show that the set of suspension flows defined over the full shift on a countable alphabet having finite entropy is open. Of independent interest might be a set of analytic tools we use to construct examples with prescribed thermodynamic behavior.

Weak specification and Baire Space

In this paper we compute the topological entropy and we consider the question of ω-chaos for shift spaces over countable alphabets. We show that if a shift space has the weak specification property, then it has ω-chaos, and we show that if, in addition, it has a countably infinite alphabet, then it has infinite topological entropy.

Skew product Smale endomorphisms over countable shifts of finite type

We introduce and study skew product Smale endomorphisms over finitely irreducible shifts with countable alphabets. This case is different from the one with finite alphabets and we develop new methods. In the conformal context we prove that almost all conditional measures of equilibrium states of summable Hölder continuous potentials are exact dimensional and their dimension is equal to the ratio of (global) entropy and Lyapunov exponent. We show that the exact dimensionality of conditional measures on fibers implies global exact dimensionality of the original measure. We then study equilibrium states for skew products over expanding Markov–Rényi transformations and settle the question of exact dimensionality of such measures. We apply our results to skew products over the continued fraction transformation. This allows us to extend and improve the Doeblin–Lenstra conjecture on Diophantine approximation coefficients to a larger class of measures and irrational numbers.

The dimension spectrum of conformal graph directed Markov systems

In this paper we study the dimension spectrum of general conformal graph directed Markov systems modeled by countable state symbolic subshifts of finite type. We perform a comprehensive study of the dimension spectrum addressing questions regarding its size and topological structure. As a corollary we prove that the dimension spectrum of infinite conformal iterated function systems is compact and perfect. On the way we revisit the role of the parameter \(\theta \) in graph directed Markov systems and we show that new phenomena arise. We also establish topological pressure estimates for subsystems in the abstract setting of symbolic dynamics with countable alphabets. These estimates play a crucial role in our proofs regarding the dimension spectrum, and they allow us to study Hausdorff dimension asymptotics for subsystems. Finally, we narrow our focus to the dimension spectrum of conformal iterated function systems and we prove, among other things, that the iterated function system resulting from the complex continued fractions algorithm has full dimension spectrum. As a result, we provide a positive answer to the Texan conjecture for complex continued fractions.

Finite sample properties of the mean occupancy counts and probabilities

For a probability distribution $P$ on an at most countable alphabet $\mathcal{A}$, this article gives finite sample bounds for the expected occupancy counts $\mathbb{E}K_{n,r}$ and probabilities $\mathbb{E}M_{n,r}$. Both upper and lower bounds are given in terms of the counting function $\nu$ of $P$. Special attention is given to the case where $\nu$ is bounded by a regularly varying function. In this case, it is shown that our general results lead to an optimal-rate control of the expected occupancy counts and probabilities with explicit constants. Our results are also put in perspective with Turing’s formula and recent concentration bounds to deduce bounds in probability. At the end of the paper, we discuss an extension of the occupancy problem to arbitrary distributions in a metric space.

Asymptotic normality for plug-in estimators of diversity indices on countable alphabets

ABSTRACTThe plug-in estimator is one of the most popular approaches to the estimation of diversity indices. In this paper, we study its asymptotic distribution for a large class of diversity indices on countable alphabets. In particular, we give conditions for the plug-in estimator to be asymptotically normal, and in the case of uniform distributions, where asymptotic normality fails, we give conditions for the asymptotic distribution to be chi-squared. Our results cover some of the most commonly used indices, including Simpson's index, Reńyi's entropy and Shannon's entropy.

Domains of attraction on countable alphabets

For each probability distribution on a countable alphabet, a sequence of positive functionals are developed as tail indices based on Turing's perspective. By and only by the asymptotic behavior of these indices, domains of attraction for all probability distributions on the alphabet are defined. The three main domains of attraction are shown to contain distributions with thick tails, thin tails and no tails respectively, resembling in parallel the three main domains of attraction, Gumbel, Frechet and Weibull families, for continuous random variables on the real line. In addition to the probabilistic merits associated with the domains, the tail indices are partially motivated by the fact that there exists an unbiased estimator for every index in the sequence, which is therefore statistically observable, provided that the sample is sufficiently large.

Entropic Moments and Domains of Attraction on Countable Alphabets

Modern information theory is largely developed in connection with random elements residing in large, complex, and discrete data spaces, or alphabets. Lacking natural metrization and hence moments, the associated probability and statistics theory must rely on information measures in the form of various entropies, for example, Shannon’s entropy, mutual information and Kullback–Leibler divergence, which are functions of an entropic basis in the form of a sequence of entropic moments of varying order. The entropicmoments collectively characterize the underlying probability distribution on the alphabet, and hence provide an opportunity to develop statistical procedures for their estimation. As such statistical development becomes an increasingly important line of research in modern data science, the relationship between the underlying distribution and the asymptotic behavior of the entropic moments, as the order increases, becomes a technical issue of fundamental importance. This paper offers a general methodology to capture the relationship between the rates of divergence of the entropic moments and the types of underlying distributions, for a special class of distributions. As an application of the established results, it is demonstrated that the asymptotic normality of the remarkable Turing’s formula for missing probabilities holds under distributions with much thinner tails than those previously known.