Active Information Storage Research Articles

Self-organization of information processing in developing neuronal networks

Human brains possess sophisticated information processing capabilities, which rely on the coordinated interplay of billions of neurons. Despite recent advances in characterizing the collective neuronal dynamics, however, it remains a major challenge to understand the principles of how functional neuronal networks develop and maintain these processing capabilities. A popular hypothesis is that neuronal networks self-organize to a critical state [1-3], because in models, criticality maximizes information processing capacities [4-6]. This predicts that biological networks should develop towards a critical state during maturation, and at the same time processing capabilities should increase. We tested this hypothesis using multi-electrode spike recordings in mouse hippocampal and cortical neurons over the first four weeks in vitro. We showed that developing neuronal networks indeed increased their information processing capacities, as quantified by transfer entropy and active information storage [6-8]. The increase in processing capacity was tightly linked to decreasing the distance to criticality (correlation r = 0.68, p < 10; r = 0.55, p < 10 for transfer and storage, respectively). Thereby our results for the first time demonstrate experimentally that approaching criticality with maturation goes in hand with increasing processing capabilities.

Anesthesia-related changes in information transfer may be caused by reduction in local information generation.

In anesthesia research it is an open question how general anesthetics lead to loss of consciousness (LOC). It has been proposed that LOC may be caused by the disruption of cortical information processing, preventing information integration. Therefore, recent studies investigating information processing under anesthesia focused on changes in information transfer, measured by transfer entropy (TE). However, often this complex technique was not applied rigorously, using time series in symbolic representation, or using TE differences without accounting for neural conduction delays, or without accounting for signal history. Here, we used current best-practice in TE estimation to investigate information transfer under anesthesia: We conducted simultaneous recordings in primary visual cortex (V1) and prefrontal cortex (PFC) of head-fixed ferrets in a dark environment under different levels of anesthesia (awake, 0.5% isoflurane, 1.0 % isoflurane). To elucidate reasons for changes in TE, we further quantified information processing within brain areas by estimating active information storage (AIS) as an estimator of predictable information, and Lempel-Ziv complexity (LZC) as an estimator of signal entropy. Under anesthesia, we found a reduction in information transfer (TE) between PFC and V1 with a stronger reduction for the feedback direction (PFC to V1), validating previous results. Furthermore, entropy (LZC) was reduced and activity became more predictable as indicated by higher values of AIS. We conclude that higher anesthesia concentrations indeed lead to reduced inter-areal information transfer, which may be partly caused by decreases in local entropy and increases in local predictability. In revealing a possible reason for reduced TE that is potentially independent of inter-areal coupling, we demonstrate the value of directly quantifying information processing in addition to focusing on dynamic properties such as coupling strength.

Editorial for the research topic: information-based methods for neuroimaging: analyzing structure, function and dynamics.

Editorial for the research topic: information-based methods for neuroimaging: analyzing structure, function and dynamics.

JIDT: An Information-Theoretic Toolkit for Studying the Dynamics of Complex Systems

Complex systems are increasingly being viewed as distributed information processing systems, particularly in the domains of computational neuroscience, bioinformatics and Artificial Life. This trend has resulted in a strong uptake in the use of (Shannon) information-theoretic measures to analyse the dynamics of complex systems in these fields. We introduce the Java Information Dynamics Toolkit (JIDT): a Google code project which provides a standalone, (GNU GPL v3 licensed) open-source code implementation for empirical estimation of information-theoretic measures from time-series data. While the toolkit provides classic information-theoretic measures (e.g. entropy, mutual information, conditional mutual information), it ultimately focusses on implementing higher-level measures for information dynamics. That is, JIDT focusses on quantifying information storage, transfer and modification, and the dynamics of these operations in space and time. For this purpose, it includes implementations of the transfer entropy and active information storage, their multivariate extensions and local or pointwise variants. JIDT provides implementations for both discrete and continuous-valued data for each measure, including various types of estimator for continuous data (e.g. Gaussian, box-kernel and Kraskov-Stoegbauer-Grassberger) which can be swapped at run-time due to Java's object-oriented polymorphism. Furthermore, while written in Java, the toolkit can be used directly in MATLAB, GNU Octave, Python and other environments. We present the principles behind the code design, and provide several examples to guide users.

Reservoir of neurons with adaptive time constants: a hybrid model for robust motor-sensory temporal processing

The ability to precisely quantify time on the scale ofhundreds of milliseconds is critical towards the proces-sing of complex sensory and motor patterns. However,the natures of neural mechanisms for temporal proces-sing (at this scale) in the brain are mostly unknown.Based on experimental data (psychophysics, cell cul-tures, electrophysiology) and theoretical studies, it is lar-gely debated whetherdedicated circuits or intrinsicmechanisms of neural circuits underlie the timing pro-cess [1]. One specific type of timing model, namelystate-dependent networks (SDN) [2], shows that time isencoded in the temporal patterns of activity of neuralpopulations and emerges from the internal dynamics ofrecurrent networks. This can be achieved without theneed of dedicated timing units. However, such intrinsicmodels in their present form have difficulty accountingfor crossmodal transfer [1]. In contrast, recent experi-mental evidence indicates that medial premotor corticalneurons of behaving monkeys show specific intervaltuning across modalities (auditory and visual) [3]. In thiswork we propose a hybrid model, making the hypothesisthat dedicated interval tuning mechanisms of individualneurons augment the intrinsic dynamics of large recur-rent networks (dynamic reservoir). Using a networkmodel of rate-coded neurons starting with random initi-alization of synaptic connections, we propose a learningrule based on local active information storage (LAIS) [4]to adapt neuronal time constants with respect to theinput stimuli to the network. Measured at each spatio-temporal location of the reservoir, LAIS gives a prob-abilistic measure of the amount of information in theprevious state of the neuron that is relevant in predict-ing the next state. Interestingly high LAIS regions in thenetwork correlate to significant events in time (intervals)of the driving stimulus. Furthermore, we combine thiswith mutual information driven intrinsic plasticityscheme in order to stabilize chaotic activity in the net-work. Incoming input drives the network which, in turn,is connected to readout neurons (Figure 1.A) that dis-play the learned behavior for temporally dependent sen-sory motor tasks. Reservoir-to-output connections canbe adapted using both supervised and reward modulatedlearning rules. Using single and multiple interval discri-mination tasks, we show that our network reproduces(across modalities) a linear increase in temporal variabil-ity with increase in interval duration. This correlation isalso observed in experimental data [3]. Furthermore wedemonstrate that our dedicated timing mechanism com-plements the inherent transient dynamics of the net-work by successfully learning complex time dependentmotor behaviors; like handwriting generation (Figure 1.Band C), locomotion pattern transformation and temporalmemory tasks. In essence, our hybrid model demon-stratesthattimecanbeencodedbyacombinationofdedicated and intrinsic mechanisms with the possibilityto ‘learn’ the temporal structure of incoming stimuli [5].

Chaos forgets and remembers: Measuring information creation, destruction, and storage

The hallmark of deterministic chaos is that it creates information---the rate being given by the Kolmogorov-Sinai metric entropy. Since its introduction half a century ago, the metric entropy has been used as a unitary quantity to measure a system's intrinsic unpredictability. Here, we show that it naturally decomposes into two structurally meaningful components: A portion of the created information---the ephemeral information---is forgotten and a portion---the bound information---is remembered. The bound information is a new kind of intrinsic computation that differs fundamentally from information creation: it measures the rate of active information storage. We show that it can be directly and accurately calculated via symbolic dynamics, revealing a hitherto unknown richness in how dynamical systems compute.

Reduced predictable information in brain signals in autism spectrum disorder.

Autism spectrum disorder (ASD) is a common developmental disorder characterized by communication difficulties and impaired social interaction. Recent results suggest altered brain dynamics as a potential cause of symptoms in ASD. Here, we aim to describe potential information-processing consequences of these alterations by measuring active information storage (AIS)—a key quantity in the theory of distributed computation in biological networks. AIS is defined as the mutual information between the past state of a process and its next measurement. It measures the amount of stored information that is used for computation of the next time step of a process. AIS is high for rich but predictable dynamics. We recorded magnetoencephalography (MEG) signals in 10 ASD patients and 14 matched control subjects in a visual task. After a beamformer source analysis, 12 task-relevant sources were obtained. For these sources, stationary baseline activity was analyzed using AIS. Our results showed a decrease of AIS values in the hippocampus of ASD patients in comparison with controls, meaning that brain signals in ASD were either less predictable, reduced in their dynamic richness or both. Our study suggests the usefulness of AIS to detect an abnormal type of dynamics in ASD. The observed changes in AIS are compatible with Bayesian theories of reduced use or precision of priors in ASD.

Local active information storage as a tool to understand distributed neural information processing

Every act of information processing can in principle be decomposed into the component operations of information storage, transfer, and modification. Yet, while this is easily done for today's digital computers, the application of these concepts to neural information processing was hampered by the lack of proper mathematical definitions of these operations on information. Recently, definitions were given for the dynamics of these information processing operations on a local scale in space and time in a distributed system, and the specific concept of local active information storage was successfully applied to the analysis and optimization of artificial neural systems. However, no attempt to measure the space-time dynamics of local active information storage in neural data has been made to date. Here we measure local active information storage on a local scale in time and space in voltage sensitive dye imaging data from area 18 of the cat. We show that storage reflects neural properties such as stimulus preferences and surprise upon unexpected stimulus change, and in area 18 reflects the abstract concept of an ongoing stimulus despite the locally random nature of this stimulus. We suggest that LAIS will be a useful quantity to test theories of cortical function, such as predictive coding.

Information dynamics based self-adaptive reservoir for delay temporal memory tasks

Recurrent neural networks of the reservoir computing (RC) type have been found useful in various time-series processing tasks with inherent non-linearity and requirements of variable temporal memory. Specifically for delayed response tasks involving the transient memorization of information (temporal memory), self-adaptation in RC is crucial for generalization to varying delays. In this work using information theory, we combine a generalized intrinsic plasticity rule with a local information dynamics based schema of reservoir neuron leak adaptation. This allows the RC network to be optimized in a self-adaptive manner with minimal parameter tuning. Local active information storage, measured as the degree of influence of previous activity on the next time step activity of a neuron, is used to modify its leak-rate. This results in RC network with non-uniform leak rate which depends on the time scales of the incoming input. Intrinsic plasticity (IP) is aimed at maximizing the mutual information between each neuron’s input and output while maintaining a mean level of activity (homeostasis). Experimental results on two standard benchmark tasks confirm the extended performance of this system as compared to the static RC (fixed leak and no IP) and RC with only IP. In addition, using both a simulated wheeled robot and a more complex physical hexapod robot, we demonstrate the ability of the system to achieve long temporal memory for solving a basic T-shaped maze navigation task with varying delay time scale.

Complexity at large

Complexity at large

Local measures of information storage in complex distributed computation

Information storage is a key component of intrinsic distributed computation. Despite the existence of appropriate measures for it (e.g. excess entropy), its role in interacting with information transfer and modification to give rise to distributed computation is not yet well-established. We explore how to quantify information storage on a local scale in space and time, so as to understand its role in the dynamics of distributed computation. To assist these explorations, we introduce the active information storage, which quantifies the information storage component that is directly in use in the computation of the next state of a process. We present the first profiles of local excess entropy and local active information storage in cellular automata, providing evidence that blinkers and background domains are dominant information storage processes in these systems. This application also demonstrates the manner in which these two measures of information storage are distinct but complementary. It also reveals other information storage phenomena, including the misinformative nature of local storage when information transfer dominates the computation, and demonstrates that the local entropy rate is a useful spatiotemporal filter for information transfer structure.

Information processing in biomolecule-based biomimetic systems. From macroscopic to nanoscopic scale

This paper deals with information processing in biomolecule-based biomimetic systems (i.e., artificial systems in which a biomolecule is involved, and whose purpose is to mimic biological functions). Several approaches are presented, based on structures ranging from the macroscopic to the nanoscopic scale. The first part is devoted to the biosensing function: the basic principle, the possible transducers, the main applications. The second part presents integrated information on processing functions in macroscopic structures: active transport, biological clocks, mathematical operations, information storage and control functions. In the third part, the evolution of this field toward the nanoscopic scale is described, with the presentation of some methods of creating nanoscopic structures (adsorbed and chemically bound monolayers, black lipid membrane (BLM) and Langmuir-Blodgett techniques), and their application in the field of biosensors. Finally, new trends in information processing by nanoscopic systems are discussed.