Estimation Of Entropy Rate Research Articles

Robust entropy rate estimation for nonstationary neuronal calcium spike trains based on empirical probabilities

Objective. Temporal patterns in neuronal spiking encode stimulus uncertainty, and convey information about high-level functions such as memory and cognition. Estimating the associated information content and understanding how that evolves with time assume significance in the investigation of neuronal coding mechanisms and abnormal signaling. However, existing estimators of the entropy rate, a measure of information content, either ignore the inherent nonstationarity, or employ dictionary-based Lempel-Ziv (LZ) methods that converge too slowly for one to study temporal variations in sufficient detail. Against this backdrop, we seek estimates that handle nonstationarity, are fast converging, and hence allow meaningful temporal investigations.Approach. We proposed a homogeneous Markov model approximation of spike trains within windows of suitably chosen length and an entropy rate estimator based on empirical probabilities that converges quickly.Main results. We constructed mathematical families of nonstationary Markov processes with certain bi/multi-level properties (inspired by neuronal responses) with known entropy rates, and validated the proposed estimator against those. Further statistical validations were presented on data collected from hippocampal (and primary visual cortex) neuron populations in terms of single neuron behavior as well as population heterogeneity. Our estimator appears to be statistically more accurate and converges faster than existing LZ estimators, and hence well suited for temporal studies.Significance. The entropy rate analysis revealed not only informational and process memory heterogeneity among neurons, but distinct statistical patterns in neuronal populations (from two different brain regions) under basal and post-stimulus conditions. Taking inspiration, we envision future large-scale studies of different brain regions enabled by the proposed tool (estimator), potentially contributing to improved functional modeling of the brain and identification of statistical signatures of neurodegenerative diseases.

Identifying Effective Connectivity between Stochastic Neurons with Variable-Length Memory Using a Transfer Entropy Rate Estimator.

Information theory explains how systems encode and transmit information. This article examines the neuronal system, which processes information via neurons that react to stimuli and transmit electrical signals. Specifically, we focus on transfer entropy to measure the flow of information between sequences and explore its use in determining effective neuronal connectivity. We analyze the causal relationships between two discrete time series, X:=Xt:t∈Z and Y:=Yt:t∈Z, which take values in binary alphabets. When the bivariate process (X,Y) is a jointly stationary ergodic variable-length Markov chain with memory no larger than k, we demonstrate that the null hypothesis of the test-no causal influence-requires a zero transfer entropy rate. The plug-in estimator for this function is identified with the test statistic of the log-likelihood ratios. Since under the null hypothesis, this estimator follows an asymptotic chi-squared distribution, it facilitates the calculation of p-values when applied to empirical data. The efficacy of the hypothesis test is illustrated with data simulated from a neuronal network model, characterized by stochastic neurons with variable-length memory. The test results identify biologically relevant information, validating the underlying theory and highlighting the applicability of the method in understanding effective connectivity between neurons.

Complexity Measures for EEG Microstate Sequences: Concepts and Algorithms

EEG microstate sequence analysis quantifies properties of ongoing brain electrical activity which is known to exhibit complex dynamics across many time scales. In this report we review recent developments in quantifying microstate sequence complexity, we classify these approaches with regard to different complexity concepts, and we evaluate excess entropy as a yet unexplored quantity in microstate research. We determined the quantities entropy rate, excess entropy, Lempel–Ziv complexity (LZC), and Hurst exponents on Potts model data, a discrete statistical mechanics model with a temperature-controlled phase transition. We then applied the same techniques to EEG microstate sequences from wakefulness and non-REM sleep stages and used first-order Markov surrogate data to determine which time scales contributed to the different complexity measures. We demonstrate that entropy rate and LZC measure the Kolmogorov complexity (randomness) of microstate sequences, whereas excess entropy and Hurst exponents describe statistical complexity which attains its maximum at intermediate levels of randomness. We confirmed the equivalence of entropy rate and LZC when the LZ-76 algorithm is used, a result previously reported for neural spike train analysis (Amigó et al., Neural Comput 16:717–736, https://doi.org/10.1162/089976604322860677, 2004). Surrogate data analyses prove that entropy-based quantities and LZC focus on short-range temporal correlations, whereas Hurst exponents include short and long time scales. Sleep data analysis reveals that deeper sleep stages are accompanied by a decrease in Kolmogorov complexity and an increase in statistical complexity. Microstate jump sequences, where duplicate states have been removed, show higher randomness, lower statistical complexity, and no long-range correlations. Regarding the practical use of these methods, we suggest that LZC can be used as an efficient entropy rate estimator that avoids the estimation of joint entropies, whereas entropy rate estimation via joint entropies has the advantage of providing excess entropy as the second parameter of the same linear fit. We conclude that metrics of statistical complexity are a useful addition to microstate analysis and address a complexity concept that is not yet covered by existing microstate algorithms while being actively explored in other areas of brain research.

Universal Densities Exist for Every Finite Reference Measure

As it is known, universal codes, which estimate the entropy rate consistently, exist for stationary ergodic sources over finite alphabets but not over countably infinite ones. We generalize universal coding as the problem of universal densities with respect to a fixed reference measure on a countably generated measurable space. We show that universal densities, which estimate the differential entropy rate consistently, exist for finite reference measures. Thus finite alphabets are not necessary in some sense. To exhibit a universal density, we adapt the nonparametric differential (NPD) entropy rate estimator by Feutrill and Roughan. Our modification is analogous to Ryabko’s modification of prediction by partial matching (PPM) by Cleary and Witten. Whereas Ryabko considered a mixture over Markov orders, we consider a mixture over quantization levels. Moreover, we demonstrate that any universal density induces a strongly consistent Cesàro mean estimator of conditional density given an infinite past. This yields a universal predictor with the 0–1 loss for a countable alphabet. Finally, we specialize universal densities to processes over natural numbers and on the real line. We derive sufficient conditions for consistent estimation of the entropy rate with respect to infinite reference measures in these domains.

Inference, Prediction, & Entropy-Rate Estimation of Continuous-Time, Discrete-Event Processes.

Inferring models, predicting the future, and estimating the entropy rate of discrete-time, discrete-event processes is well-worn ground. However, a much broader class of discrete-event processes operates in continuous-time. Here, we provide new methods for inferring, predicting, and estimating them. The methods rely on an extension of Bayesian structural inference that takes advantage of neural network's universal approximation power. Based on experiments with complex synthetic data, the methods are competitive with the state-of-the-art for prediction and entropy-rate estimation.

Entropy Estimation Using a Linguistic Zipf-Mandelbrot-Li Model for Natural Sequences.

Entropy estimation faces numerous challenges when applied to various real-world problems. Our interest is in divergence and entropy estimation algorithms which are capable of rapid estimation for natural sequence data such as human and synthetic languages. This typically requires a large amount of data; however, we propose a new approach which is based on a new rank-based analytic Zipf–Mandelbrot–Li probabilistic model. Unlike previous approaches, which do not consider the nature of the probability distribution in relation to language; here, we introduce a novel analytic Zipfian model which includes linguistic constraints. This provides more accurate distributions for natural sequences such as natural or synthetic emergent languages. Results are given which indicates the performance of the proposed ZML model. We derive an entropy estimation method which incorporates the linguistic constraint-based Zipf–Mandelbrot–Li into a new non-equiprobable coincidence counting algorithm which is shown to be effective for tasks such as entropy rate estimation with limited data.

A Review of Shannon and Differential Entropy Rate Estimation.

In this paper, we present a review of Shannon and differential entropy rate estimation techniques. Entropy rate, which measures the average information gain from a stochastic process, is a measure of uncertainty and complexity of a stochastic process. We discuss the estimation of entropy rate from empirical data, and review both parametric and non-parametric techniques. We look at many different assumptions on properties of the processes for parametric processes, in particular focussing on Markov and Gaussian assumptions. Non-parametric estimation relies on limit theorems which involve the entropy rate from observations, and to discuss these, we introduce some theory and the practical implementations of estimators of this type.

Specific Differential Entropy Rate Estimation for Continuous-Valued Time Series

We introduce a method for quantifying the inherent unpredictability of a continuous-valued time series via an extension of the differential Shannon entropy rate. Our extension, the specific entropy rate, quantifies the amount of predictive uncertainty associated with a specific state, rather than averaged over all states. We provide a data-driven approach for estimating the specific entropy rate of an observed time series. Finally, we consider three case studies of estimating the specific entropy rate from synthetic and physiological data relevant to the analysis of heart rate variability.

Complex Independent Component Analysis Using Three Types of Diversity: Non-Gaussianity, Nonwhiteness, and Noncircularity

By assuming latent sources are statistically independent, independent component analysis (ICA) separates underlying sources from a given linear mixture. Since in many applications, latent sources are non-Gaussian, noncircular, and have sample dependence, it is desirable to exploit all these properties jointly. Mutual information rate, which leads to the minimization of entropy rate, provides a natural cost for the task. In this paper, we establish the theory for complex-valued ICA giving Cramer-Rao lower bound and identification conditions, and present a new algorithm that takes all these properties into account. We propose an effective estimator of entropy rate and a complex-valued entropy rate bound minimization algorithm based on it. We show that the new method exploits all these properties effectively by comparing the estimation performance with the Cramer-Rao lower bound and by a number of examples.

On the non-randomness of maximum Lempel Ziv complexity sequences of finite size

Random sequences attain the highest entropy rate. The estimation of entropy rate for an ergodic source can be done using the Lempel Ziv complexity measure yet, the exact entropy rate value is only reached in the infinite limit. We prove that typical random sequences of finite length fall short of the maximum Lempel-Ziv complexity, contrary to common belief. We discuss that, for a finite length, maximum Lempel-Ziv sequences can be built from a well defined generating algorithm, which makes them of low Kolmogorov-Chaitin complexity, quite the opposite to randomness. It will be discussed that Lempel-Ziv measure is, in this sense, less general than Kolmogorov-Chaitin complexity, as it can be fooled by an intelligent enough agent. The latter will be shown to be the case for the binary expansion of certain irrational numbers. Maximum Lempel-Ziv sequences induce a normalization that gives good estimates of entropy rate for several sources, while keeping bounded values for all sequence length, making it an alternative to other normalization schemes in use.

Analysis of Bit-Plane Probability for Generalized Gaussian Distribution and its Application in Audio Coding

Bit-plane probability of data is useful information for many applications such as entropy coding and rate estimation, particularly for scalable coding system. Typically, the value of bit-plane probability varies according to the distribution of the data represented, and due to inter-plane correlation, it is hard to get an analytic solution of bit-plane probability for the data with generalized Gaussian distribution. In this paper, bit-plane probability is analyzed when Generalized Gaussian distribution is used to model the input data. Based on the study of bit-plane probability for Laplace distribution and the relationship between different bit-planes, an approximated bit-plane probability for Generalized Gaussian distribution is presented. This closed-form expression is of low computational cost. Furthermore, a much practical format is derived with reduced complexity for implementation in the state-of-the-art MPEG-4 scalable to lossless audio coding system. With the same computational cost, the proposed algorithm presents higher compression efficiency than MPEG-4 scalable to lossless audio coding, which considers Laplace distribution only.

Bias of a nonparametric entropy estimator for Markov measures

We consider the problem of nonparametric entropy (entropy rate) estimation. We study the technique of nonparametric entropy estimation based on the so-called “nearest neighbor distances” and obtain a closed-form expression of the bias for Markov measures. This bias is a discontinuous function of transition probabilities. Bibliography: 20 titles.

Blind Separation of Noncircular Correlated Sources Using Gaussian Entropy Rate

We introduce a new blind source separation (BSS) algorithm for correlated noncircular sources that uses only second-order statistics and fully takes the correlation structure into account. We propose a parametric entropy rate estimator that uses a widely linear autoregressive (AR) model for the sources, and derive the BSS algorithm by minimizing the mutual information of separated time series. We compare the performance of the new algorithm with competing algorithms and demonstrate its superior separation performance as well as its effectiveness in separation of non-Gaussian sources when the identification conditions are met.

The influence of cryogenic cooling on process stability in turning operations

This paper presents results of the influence of cryogenic machining on the process stability. The stability diagrams are obtained experimentally using the coarse-grained entropy rate estimator for chatter detection from measured cutting forces. In comparison with conventional machining, enlarged stability windows are observed for the case of cryogenic machining. Based on the defined specific force models in turning operations, it is shown that a higher machining stability is achievable in cryogenic machining due to the reduction of specific cutting force components, in comparison with dry machining.

Entropy estimators with almost sure convergence and an <I>O(n</I><SUP><I>-1</I></SUP><I>)</I> variance

The problem of the estimation of the entropy rate of a stationary ergodic process $\mu$ is considered. A new nonparametric entropy rate estimator is constructed for a sample of n sequences $(X_1$(1)$,\ldots, (X_m$(1)$),\ldots, (X_1$(n) $,\ldots, (X_m$(n)$)$ independently generated by $\mu$. It is shown that, for $m=O(\log n)$, the estimator converges almost surely and its variance is upper-bounded by $O(n$−1$)$ for a large class of stationary ergodic processes with a finite state space. As the order $O(n$−1$)$ of the variance growth on $n$ is the same as that of the optimal Cramer-Rao lower bound, presented is the first near-optimal estimator in the sense of the variance convergence.

Variance estimators for the Lempel-Ziv entropy rate estimator

Lempel and Ziv's 1976 algorithm provides an easy-to-compute way to automatically estimate the entropy rate for symbolic time series, requiring no free parameters. Here we derive an analytical variance estimate for the Lempel-Ziv entropy rate estimator that is easily computable from observations with negligible extra effort beyond the entropy rate itself, and compare to another procedure, a time-series-based bootstrap method. These provide a justified "error bar" quantifying the size of expected fluctuations on the estimate itself, given by the single time series of symbols.

Entropy rate based speech segmentation

The sequence of estimates of the speech signal’s entropy rate is investigated as a potential basis for speech segmentation. Raising and falling edges of that entropy rate curve and its maxima and minima are considered as candidates for segment boundaries. These prominent points are compared to the phonetic segment boundaries and to acoustic landmarks [S. A. Liu, J. Acoust. Soc. Am. 100, 3417–3430 (1996)]. The comparison is made using the ‘‘Kiel corpus of read speech’’ which is a phonetically manually labeled corpus of German utterances. The evaluation considers the entropy rate labels against the background of the phone segments. In particular the position and the count of the entropy rate labels within each phone are appraised. The average results are reported both as totals and broken down into phone classes and phones. The frame duration of the entropy rate estimation controls the smoothness of the curve and hence the label count per phone. A similar evaluation relates the entropy rate labels to the acoustic landmarks.