Active Information Storage Research Articles

Measuring and understanding information storage and transfer in a simulated human gut microbiome.

Considering biological systems as information processing entities and analyzing their organizational structure via information-theoretic measures has become an established approach in life sciences. We transfer this framework to a field of broad general interest, the human gut microbiome. We use BacArena, a software combining agent-based modelling and flux-balance analysis, to simulate a simplified human intestinal microbiome (SIHUMI). In a first step, we derive information theoretic measures from the simulated abundance data, and, in a second step, relate them to the metabolic processes underlying the abundance data. Our study provides further evidence on the role of active information storage as an indicator of unexpected structural change in the observed system. Besides, we show that information transfer reflects coherent behavior in the microbial community, both as a reaction to environmental changes and as a result of direct effective interaction. In this sense, purely abundance-based information theoretic measures can provide meaningful insight on metabolic interactions within bacterial communities. Furthermore, we shed light on the important however little noticed technical aspect of distinguishing immediate and delayed effects in the interpretation of local information theoretical measures.

Propofol Reversibly Attenuates Short-Range Microstate Ordering and 20Hz Microstate Oscillations.

Microstate sequences summarize the changing voltage patterns measured by electroencephalography, using a clustering approach to reduce the high dimensionality of the underlying data. A common approach is to restrict the pattern matching step to local maxima of the global field power (GFP) and to interpolate the microstate fit in between. In this study, we investigate how the anesthetic propofol affects microstate sequence periodicity and predictability, and how these metrics are changed by interpolation. We performed two frequency analyses on microstate sequences, one based on time-lagged mutual information, the other based on Fourier transform methodology, and quantified the effects of interpolation. Resting-state microstate sequences had a 20Hz frequency peak related to dominant 10Hz (alpha) rhythms, and the Fourier approach demonstrated that all five microstate classes followed this frequency. The 20Hz periodicity was reversibly attenuated under moderate propofol sedation, as shown by mutual information and Fourier analysis. Characteristic microstate frequencies could only be observed in non-interpolated microstate sequences and were masked by smoothing effects of interpolation. Information-theoretic analysis revealed faster microstate dynamics and larger entropy rates under propofol, whereas Shannon entropy did not change significantly. In moderate sedation, active information storage decreased for non-interpolated sequences. Signatures of non-equilibrium dynamics were observed in non-interpolated sequences, but no changes were observed between sedation levels. All changes occurred while subjects were able to perform an auditory perception task. In summary, we show that low dose propofol reversibly increases the randomness of microstate sequences and attenuates microstate oscillations without correlation to cognitive task performance. Microstate dynamics between GFP peaks reflect physiological processes that are not accessible in interpolated sequences.

Information-theoretic analyses of neural data to minimize the effect of researchers' assumptions in predictive coding studies.

Studies investigating neural information processing often implicitly ask both, which processing strategy out of several alternatives is used and how this strategy is implemented in neural dynamics. A prime example are studies on predictive coding. These often ask whether confirmed predictions about inputs or prediction errors between internal predictions and inputs are passed on in a hierarchical neural system-while at the same time looking for the neural correlates of coding for errors and predictions. If we do not know exactly what a neural system predicts at any given moment, this results in a circular analysis-as has been criticized correctly. To circumvent such circular analysis, we propose to express information processing strategies (such as predictive coding) by local information-theoretic quantities, such that they can be estimated directly from neural data. We demonstrate our approach by investigating two opposing accounts of predictive coding-like processing strategies, where we quantify the building blocks of predictive coding, namely predictability of inputs and transfer of information, by local active information storage and local transfer entropy. We define testable hypotheses on the relationship of both quantities, allowing us to identify which of the assumed strategies was used. We demonstrate our approach on spiking data collected from the retinogeniculate synapse of the cat (N = 16). Applying our local information dynamics framework, we are able to show that the synapse codes for predictable rather than surprising input. To support our findings, we estimate quantities applied in the partial information decomposition framework, which allow to differentiate whether the transferred information is primarily bottom-up sensory input or information transferred conditionally on the current state of the synapse. Supporting our local information-theoretic results, we find that the synapse preferentially transfers bottom-up information.

Rotationally invariant estimators on portfolio optimization to unveil financial risk’s states

Rotationally Invariant Estimators (RIE) are a new family of covariance matrix estimators based on random matrix theory and free probability. The family RIE has been proposed to improve the performance of an investment portfolio in the Markowitz model’s framework. Here, we apply state-of-the-art RIE techniques to improve the estimation of financial states via the correlation matrix. The Synthesized Clustering (SYNCLUS) and a dynamic programming algorithm for optimal one-dimensional clustering were employed to that aim. We found that the RIE estimations of the minimum portfolio risk increase the Active Information Storage (AIS) in the American and European markets. AIS’s local dynamic also mimics financial states’ behavior when estimating under the one-dimensional clustering algorithm. Our results suggest that in times of financial turbulence, RIE estimates can be of great advantage in minimizing risk exposure.

Information theoretic evidence for layer- and frequency-specific changes in cortical information processing under anesthesia.

Nature relies on highly distributed computation for the processing of information in nervous systems across the entire animal kingdom. Such distributed computation can be more easily understood if decomposed into the three elementary components of information processing, i.e. storage, transfer and modification, and rigorous information theoretic measures for these components exist. However, the distributed computation is often also linked to neural dynamics exhibiting distinct rhythms. Thus, it would be beneficial to associate the above components of information processing with distinct rhythmic processes where possible. Here we focus on the storage of information in neural dynamics and introduce a novel spectrally-resolved measure of active information storage (AIS). Drawing on intracortical recordings of neural activity in ferrets under anesthesia before and after loss of consciousness (LOC) we show that anesthesia- related modulation of AIS is highly specific to different frequency bands and that these frequency-specific effects differ across cortical layers and brain regions. We found that in the high/low gamma band the effects of anesthesia result in AIS modulation only in the supergranular layers, while in the alpha/beta band the strongest decrease in AIS can be seen at infragranular layers. Finally, we show that the increase of spectral power at multiple frequencies, in particular at alpha and delta bands in frontal areas, that is often observed during LOC ('anteriorization') also impacts local information processing-but in a frequency specific way: Increases in isoflurane concentration induced a decrease in AIS in the alpha frequencies, while they increased AIS in the delta frequency range < 2Hz. Thus, the analysis of spectrally-resolved AIS provides valuable additional insights into changes in cortical information processing under anaesthesia.

Arousal system stimulation and anesthetic state alter visuoparietal connectivity.

Cortical information processing is under the precise control of the ascending arousal system (AAS). Anesthesia suppresses cortical arousal that can be mitigated by exogenous stimulation of the AAS. The question remains to what extent cortical information processing is regained by AAS stimulation. We investigate the effect of electrical stimulation of the nucleus Pontis Oralis (PnO), a distinct source of ascending AAS projections, on cortical functional connectivity (FC) and information storage at mild, moderate, and deep anesthesia. Local field potentials (LFPs) recorded previously in the secondary visual cortex (V2) and the adjacent parietal association cortex (PtA) in chronically instrumented unrestrained rats. We hypothesized that PnO stimulation would induce electrocortical arousal accompanied by enhanced FC and active information storage (AIS) implying improved information processing. In fact, stimulation reduced FC in slow oscillations (0.3-2.5 Hz) at low anesthetic level and increased FC at high anesthetic level. These effects were augmented following stimulation suggesting stimulus-induced plasticity. The observed opposite stimulation-anesthetic impact was less clear in the γ-band activity (30-70 Hz). In addition, FC in slow oscillations was more sensitive to stimulation and anesthetic level than FC in γ-band activity which exhibited a rather constant spatial FC structure that was symmetric between specific, topographically related sites in V2 and PtA. Invariant networks were defined as a set of strongly connected electrode channels, which were invariant to experimental conditions. In invariant networks, stimulation decreased AIS and increasing anesthetic level increased AIS. Conversely, in non-invariant (complement) networks, stimulation did not affect AIS at low anesthetic level but increased it at high anesthetic level. The results suggest that arousal stimulation alters cortical FC and information storage as a function of anesthetic level with a prolonged effect beyond the duration of stimulation. The findings help better understand how the arousal system may influence information processing in cortical networks at different levels of anesthesia.

NoLiTiA: An Open-Source Toolbox for Non-linear Time Series Analysis.

In many scientific fields including neuroscience, climatology or physics, complex relationships can be described most parsimoniously by non-linear mechanics. Despite their relevance, many neuroscientists still apply linear estimates in order to evaluate complex interactions. This is partially due to the lack of a comprehensive compilation of non-linear methods. Available packages mostly specialize in only one aspect of non-linear time-series analysis and most often require some coding proficiency to use. Here, we introduce NoLiTiA, a free open-source MATLAB toolbox for non-linear time series analysis. In comparison to other currently available non-linear packages, NoLiTiA offers (1) an implementation of a broad range of classic and recently developed methods, (2) an implementation of newly proposed spatially and time-resolved recurrence amplitude analysis and (3) an intuitive environment accessible even to users with little coding experience due to a graphical user interface and batch-editor. The core methodology derives from three distinct fields of complex systems theory, including dynamical systems theory, recurrence quantification analysis and information theory. Besides established methodology including estimation of dynamic invariants like Lyapunov exponents and entropy-based measures, such as active information storage, we include recent developments of quantifying time-resolved aperiodic oscillations. In general, the toolbox will make non-linear methods accessible to the broad neuroscientific community engaged in time series processing.

Information dynamics of price and liquidity around the 2017 Bitcoin markets crash.

We study information dynamics between the largest Bitcoin exchange markets during the bubble in 2017-2018. By analyzing high-frequency market microstructure observables with different information-theoretic measures for dynamical systems, we find temporal changes in information sharing across markets. In particular, we study time-varying components of predictability, memory, and (a)synchronous coupling, measured by transfer entropy, active information storage, and multi-information. By comparing these empirical findings with several models, we argue that some results could relate to intra-market and inter-market regime shifts and changes in the direction of information flow between different market observables.

Information dynamics in neuromorphic nanowire networks

Neuromorphic systems comprised of self-assembled nanowires exhibit a range of neural-like dynamics arising from the interplay of their synapse-like electrical junctions and their complex network topology. Additionally, various information processing tasks have been demonstrated with neuromorphic nanowire networks. Here, we investigate the dynamics of how these unique systems process information through information-theoretic metrics. In particular, Transfer Entropy (TE) and Active Information Storage (AIS) are employed to investigate dynamical information flow and short-term memory in nanowire networks. In addition to finding that the topologically central parts of networks contribute the most to the information flow, our results also reveal TE and AIS are maximized when the networks transitions from a quiescent to an active state. The performance of neuromorphic networks in memory and learning tasks is demonstrated to be dependent on their internal dynamical states as well as topological structure. Optimal performance is found when these networks are pre-initialised to the transition state where TE and AIS are maximal. Furthermore, an optimal range of information processing resources (i.e. connectivity density) is identified for performance. Overall, our results demonstrate information dynamics is a valuable tool to study and benchmark neuromorphic systems.

Measuring inter- and intra-individual differences in visual scan patterns in a driving simulator experiment using active information storage.

Scan pattern analysis has been discussed as a promising tool in the context of real-time gaze-based applications. In particular, information-theoretic measures of scan path predictability, such as the gaze transition entropy (GTE), have been proposed for detecting relevant changes in user state or task demand. These measures model scan patterns as first-order Markov chains, assuming that only the location of the previous fixation is predictive of the next fixation in time. However, this assumption may not be sufficient in general, as recent research has shown that scan patterns may also exhibit more long-range temporal correlations. Thus, we here evaluate the active information storage (AIS) as a novel information-theoretic approach to quantifying scan path predictability in a dynamic task. In contrast to the GTE, the AIS provides means to statistically test and account for temporal correlations in scan path data beyond the previous last fixation. We compare AIS to GTE in a driving simulator experiment, in which participants drove in a highway scenario, where trials were defined based on an experimental manipulation that encouraged the driver to start an overtaking maneuver. Two levels of difficulty were realized by varying the time left to complete the task. We found that individual observers indeed showed temporal correlations beyond a single past fixation and that the length of the correlation varied between observers. No effect of task difficulty was observed on scan path predictability for either AIS or GTE, but we found a significant increase in predictability during overtaking. Importantly, for participants for which the first-order Markov chain assumption did not hold, this was only shown using AIS but not GTE. We conclude that accounting for longer time horizons in scan paths in a personalized fashion is beneficial for interpreting gaze pattern in dynamic tasks.

Quantifying the Predictability of Visual Scanpaths Using Active Information Storage.

Entropy-based measures are an important tool for studying human gaze behavior under various conditions. In particular, gaze transition entropy (GTE) is a popular method to quantify the predictability of a visual scanpath as the entropy of transitions between fixations and has been shown to correlate with changes in task demand or changes in observer state. Measuring scanpath predictability is thus a promising approach to identifying viewers’ cognitive states in behavioral experiments or gaze-based applications. However, GTE does not account for temporal dependencies beyond two consecutive fixations and may thus underestimate the actual predictability of the current fixation given past gaze behavior. Instead, we propose to quantify scanpath predictability by estimating the active information storage (AIS), which can account for dependencies spanning multiple fixations. AIS is calculated as the mutual information between a processes’ multivariate past state and its next value. It is thus able to measure how much information a sequence of past fixations provides about the next fixation, hence covering a longer temporal horizon. Applying the proposed approach, we were able to distinguish between induced observer states based on estimated AIS, providing first evidence that AIS may be used in the inference of user states to improve human–machine interaction.

Anatomy of information transfer for Ibis using transfer entropy and active information storage

As birds migrate in a linear formation, the frequent exchange of leader-follower in the Ibises can extend the formation flight time. In this paper, we utilize the information theory to analyze the underlying mechanism of the coordination and cooperation within the Ibises. To this end, we examine and quantify the dynamic characteristics of the information transfer in a group of Ibises at the individual level. Firstly, the local transfer entropy is employed to quantify the local dynamic information transfer during the migration of the Ibises. Therefore, we provide lucid explanations of how Ibis individually influences the group decision through information transfer to generate consistent collective behavior. The local active information storage is used to represent the memory of the Ibis. The relationship between the consistency of the collective behavior and the memory of the Ibis is obtained by calculating the mutual information between the historical state sequences and the future state of the Ibis. Finally, based on the dynamic information transfer, the relationship between the group memory and the consistent collective behavior, we effectively captured the relationship between the excitation and convergence of the Ibis’ collective behavior, and the variance of entropy is obtained.

IDTxl: The Information Dynamics Toolkit xl: a Python package for the efficient analysis of multivariate information dynamics in networks

The Information Dynamics Toolkit xl (IDTxl) is a comprehensive software package for efficient inference of networks and their node dynamics from multivariate time series data using information theory. IDTxl provides functionality to estimate the following measures: 1) For network inference: multivariate transfer entropy (TE)/Granger causality (GC), multivariate mutual information (MI), bivariate TE/GC, bivariate MI 2) For analysis of node dynamics: active information storage (AIS), partial information decomposition (PID) IDTxl implements estimators for discrete and continuous data with parallel computing engines for both GPU and CPU platforms. Written for Python3.4.3+.

Active information storage in Parkinson's disease: a resting state fMRI study over the sensorimotor cortex.

Parkinson's disease (PD), the second most frequent neurodegenerative disease, affects significantly life quality by a combination of motor and cognitive disturbances. Although it is traditionally associated with basal ganglia dysfunction, cortical alterations are also involved in disease symptoms. Our objective is to evaluate the alterations in brain dynamics in de novo and recently treated PD subjects using a nonlinear method known as Active Information Storage. In the current research, Active Information Storage (AIS) was used to study the complex dynamics in motor cortex spontaneous activity captured using resting state functional Magnetic Resonance Imaging (rs-fMRI) at early-stage in non-medicated and recently medicated PD subjects. Supplementary to AIS, the fractional Amplitude of Low Frequency Fluctuation (fALFF), which is a better-established technique of analysis of rs-fMRI signals, was also evaluated. Compared to healthy subjects, the AIS values were significantly reduced in PD patients over the analyzed motor cortex regions; differences were also found at less extent using the fALFF measure. Correlations between AIS and fALFF values showed that the measures seem to capture similar neuronal phenomena in rs-fMRI data. The highest sensitivity when detecting group differences revealed by AIS, and not captured by traditional linear approaches, suggests that this measure is a promising tool for the analysis of rs-fMRI neural data in PD.

Partial Autoinformation to Characterize Symbolic Sequences.

An information-theoretic approach to numerically determine the Markov order of discrete stochastic processes defined over a finite state space is introduced. To measure statistical dependencies between different time points of symbolic time series, two information-theoretic measures are proposed. The first measure is time-lagged mutual information between the random variables Xn and Xn+k, representing the values of the process at time points n and n + k, respectively. The measure will be termed autoinformation, in analogy to the autocorrelation function for metric time series, but using Shannon entropy rather than linear correlation. This measure is complemented by the conditional mutual information between Xn and Xn+k, removing the influence of the intermediate values Xn+k−1, …, Xn+1. The second measure is termed partial autoinformation, in analogy to the partial autocorrelation function (PACF) in metric time series analysis. Mathematical relations with known quantities such as the entropy rate and active information storage are established. Both measures are applied to a number of examples, ranging from theoretical Markov and non-Markov processes with known stochastic properties, to models from statistical physics, and finally, to a discrete transform of an EEG data set. The combination of autoinformation and partial autoinformation yields important insights into the temporal structure of the data in all test cases. For first- and higher-order Markov processes, partial autoinformation correctly identifies the order parameter, but also suggests extended, non-Markovian effects in the examples that lack the Markov property. For three hidden Markov models (HMMs), the underlying Markov order is found. The combination of both quantities may be used as an early step in the analysis of experimental, non-metric time series and can be employed to discover higher-order Markov dependencies, non-Markovianity and periodicities in symbolic time series.

Characterizing information-theoretic storage and transfer in continuous time processes.

The characterization of information processing is an important task in complex systems science. Information dynamics is a quantitative methodology for modeling the intrinsic information processing conducted by a process represented as a time series, but to date has only been formulated in discrete time. Building on previous work which demonstrated how to formulate transfer entropy in continuous time, we give a total account of information processing in this setting, incorporating information storage. We find that a convergent rate of predictive capacity, comprising the transfer entropy and active information storage, does not exist, arising through divergent rates of active information storage. We identify that active information storage can be decomposed into two separate quantities that characterize predictive capacity stored in a process: active memory utilization and instantaneous predictive capacity. The latter involves prediction related to path regularity and so solely inherits the divergent properties of the active information storage, while the former permits definitions of pathwise and rate quantities. We formulate measures of memory utilization for jump and neural spiking processes and illustrate measures of information processing in synthetic neural spiking models and coupled Ornstein-Uhlenbeck models. The application to synthetic neural spiking models demonstrates that active memory utilization for point processes consists of discontinuous jump contributions (at spikes) interrupting a continuously varying contribution (relating to waiting times between spikes), complementing the behavior previously demonstrated for transfer entropy in these processes.

Information-theoretic model selection for optimal prediction of stochastic dynamical systems from data.

In the absence of mechanistic or phenomenological models of real-world systems, data-driven models become necessary. The discovery of various embedding theorems in the 1980s and 1990s motivated a powerful set of tools for analyzing deterministic dynamical systems via delay-coordinate embeddings of observations of their component states. However, in many branches of science, the condition of operational determinism is not satisfied, and stochastic models must be brought to bear. For such stochastic models, the tool set developed for delay-coordinate embedding is no longer appropriate, and a new toolkit must be developed. We present an information-theoretic criterion, the negative log-predictive likelihood, for selecting the embedding dimension for a predictively optimal data-driven model of a stochastic dynamical system. We develop a nonparametric estimator for the negative log-predictive likelihood and compare its performance to a recently proposed criterion based on active information storage. Finally, we show how the output of the model selection procedure can be used to compare candidate predictors for a stochastic system to an information-theoretic lower bound.

Information-Theoretic Evidence for Predictive Coding in the Face-Processing System.

Predictive coding suggests that the brain infers the causes of its sensations by combining sensory evidence with internal predictions based on available prior knowledge. However, the neurophysiological correlates of (pre)activated prior knowledge serving these predictions are still unknown. Based on the idea that such preactivated prior knowledge must be maintained until needed, we measured the amount of maintained information in neural signals via the active information storage (AIS) measure. AIS was calculated on whole-brain beamformer-reconstructed source time courses from MEG recordings of 52 human subjects during the baseline of a Mooney face/house detection task. Preactivation of prior knowledge for faces showed as α-band-related and β-band-related AIS increases in content-specific areas; these AIS increases were behaviorally relevant in the brain's fusiform face area. Further, AIS allowed decoding of the cued category on a trial-by-trial basis. Our results support accounts indicating that activated prior knowledge and the corresponding predictions are signaled in low-frequency activity (<30 Hz).SIGNIFICANCE STATEMENT Our perception is not only determined by the information our eyes/retina and other sensory organs receive from the outside world, but strongly depends also on information already present in our brains, such as prior knowledge about specific situations or objects. A currently popular theory in neuroscience, predictive coding theory, suggests that this prior knowledge is used by the brain to form internal predictions about upcoming sensory information. However, neurophysiological evidence for this hypothesis is rare, mostly because this kind of evidence requires strong a priori assumptions about the specific predictions the brain makes and the brain areas involved. Using a novel, assumption-free approach, we find that face-related prior knowledge and the derived predictions are represented in low-frequency brain activity.

Criticality and Information Dynamics in Epidemiological Models

Understanding epidemic dynamics has always been a challenge. As witnessed from the ongoing Zika or the seasonal Influenza epidemics, we still need to improve our analytical methods to better understand and control epidemics. While the emergence of complex sciences in the turn of the millennium have resulted in their implementation in modelling epidemics, there is still a need for improving our understanding of critical dynamics in epidemics. In this study, using agent-based modelling, we simulate a Susceptible-Infected-Susceptible (SIS) epidemic on a homogeneous network. We use transfer entropy and active information storage from information dynamics framework to characterise the critical transition in epidemiological models. Our study shows that both (bias-corrected) transfer entropy and active information storage maximise after the critical threshold ( R 0 = 1). This is the first step toward an information dynamics approach to epidemics. Understanding the dynamics around the criticality in epidemiological models can provide us insights about emergent diseases and disease control.

Information transfer in a swarm of soldier crabs

Collective behavior is broadly observed in animal groups such as insect swarm, bird flock, and fish school. Both theoretical studies and field observations have investigated possible underlying principles based on local interaction among individuals in a group without global information via conductors or leaders. Information transferred among individuals would play a key role to understand it. In this study, to investigate how individual in a swarm uses information of its own past behavior or swarm mates' behavior, we analyzed behavior of soldier crabs Mictyris guinotae in terms of local active information storage and local transfer entropy.