Information-theoretic Analysis Research Articles

When to Simply Use Passive RIS as Beamformer: An Information-Theoretic Analysis and a Novel Single-RF MIMO Transceiver Architecture

When to Simply Use Passive RIS as Beamformer: An Information-Theoretic Analysis and a Novel Single-RF MIMO Transceiver Architecture

Information–Theoretic Analysis of Visibility Graph Properties of Extremes in Time Series Generated by a Nonlinear Langevin Equation

In this study, we examined how the nonlinearity α of the Langevin equation influences the behavior of extremes in a generated time series. The extremes, defined according to run theory, result in two types of series, run lengths and surplus magnitudes, whose complex structure was investigated using the visibility graph (VG) method. For both types of series, the information measures of the Shannon entropy measure and Fisher Information Measure were utilized for illustrating the influence of the nonlinearity α on the distribution of the degree, which is the main topological parameter describing the graph constructed by the VG method. The main finding of our study was that the Shannon entropy of the degree of the run lengths and the surplus magnitudes of the extremes is mostly influenced by the nonlinearity, which decreases with with an increase in α. This result suggests that the run lengths and surplus magnitudes of extremes are characterized by a sort of order that increases with increases in nonlinearity.

Kinetic Trajectories of Glucose Uptake in Single Cancer Cells Reveal a Drug-Induced Cell-State Change Within Hours of Drug Treatment.

The development of drug resistance is a nearly universal phenomenon in patients with glioblastoma multiforme (GBM) brain tumors. Upon treatment, GBM cancer cells may initially undergo a drug-induced cell-state change to a drug-tolerant, slow-cycling state. The kinetics of that process are not well understood, in part due to the heterogeneity of GBM tumors and tumor models, which can confound the interpretation of kinetic data. Here, we resolve drug-adaptation kinetics in a patient-derived in vitro GBM tumor model characterized by the epithelial growth factor receptor (EGFR) variant(v)III oncogene treated with an EGFR inhibitor. We use radiolabeled 18F-fluorodeoxyglucose (FDG) to monitor the glucose uptake trajectories of single GBM cancer cells over a 12 h period of drug treatment. Autocorrelation analysis of the single-cell glucose uptake trajectories reveals evidence of a drug-induced cell-state change from a high- to low-glycolytic phenotype after 5-7 h of drug treatment. Information theoretic analysis of a bulk transcriptome kinetic series of the GBM tumor model delineated the underlying molecular mechanisms driving the cellular state change, including a shift from a stem-like mesenchymal state to a more differentiated, slow-cycling astrocyte-like state. Our results demonstrate that complex drug-induced cancer cell-state changes of cancer cells can be captured via measurements of single cell metabolic trajectories and reveal the extremely facile nature of drug adaptation.

A quantum information theoretic analysis of reinforcement learning-assisted quantum architecture search

In the field of quantum computing, variational quantum algorithms (VQAs) represent a pivotal category of quantum solutions across a broad spectrum of applications. These algorithms demonstrate significant potential for realising quantum computational advantage. A fundamental aspect of VQAs involves formulating expressive and efficient quantum circuits (namely ansatz), and automating the search of such ansatz is known as quantum architecture search (QAS). Recently reinforcement learning (RL) techniques is utilized to automate the search for ansatzes, know as RL-QAS. This study investigates RL-QAS for crafting ansatz tailored to the variational quantum state diagonalisation problem. Our investigation includes a comprehensive analysis of various dimensions, such as the entanglement thresholds of the resultant states, the impact of initial conditions on the performance of RL-agent, the phase transition behaviour of correlation in concurrence bounds, and the discrete contributions of qubits in deducing eigenvalues through conditional entropy metrics. We leverage these insights to devise an entanglement-guided admissible ansatz in QAS to diagonalise random quantum states using optimal resources. Furthermore, the methodologies presented herein offer a generalised framework for constructing reward functions within RL-QAS applicable to variational quantum algorithms.

Signatures of hierarchical temporal processing in the mouse visual system.

A core challenge for the brain is to process information across various timescales. This could be achieved by a hierarchical organization of temporal processing through intrinsic mechanisms (e.g., recurrent coupling or adaptation), but recent evidence from spike recordings of the rodent visual system seems to conflict with this hypothesis. Here, we used an optimized information-theoretic and classical autocorrelation analysis to show that information- and correlation timescales of spiking activity increase along the anatomical hierarchy of the mouse visual system under visual stimulation, while information-theoretic predictability decreases. Moreover, intrinsic timescales for spontaneous activity displayed a similar hierarchy, whereas the hierarchy of predictability was stimulus-dependent. We could reproduce these observations in a basic recurrent network model with correlated sensory input. Our findings suggest that the rodent visual system employs intrinsic mechanisms to achieve longer integration for higher cortical areas, while simultaneously reducing predictability for an efficient neural code.

Thompson Sampling for Stochastic Bandits with Noisy Contexts: An Information-Theoretic Regret Analysis.

We study stochastic linear contextual bandits (CB) where the agent observes a noisy version of the true context through a noise channel with unknown channel parameters. Our objective is to design an action policy that can "approximate" that of a Bayesian oracle that has access to the reward model and the noise channel parameter. We introduce a modified Thompson sampling algorithm and analyze its Bayesian cumulative regret with respect to the oracle action policy via information-theoretic tools. For Gaussian bandits with Gaussian context noise, our information-theoretic analysis shows that under certain conditions on the prior variance, the Bayesian cumulative regret scales as O˜(mT), where m is the dimension of the feature vector and T is the time horizon. We also consider the problem setting where the agent observes the true context with some delay after receiving the reward, and show that delayed true contexts lead to lower regret. Finally, we empirically demonstrate the performance of the proposed algorithms against baselines.

Quantum information-theoretical analysis on the two-photon transitions in hydrogen isoelectronic ions under plasma confinement

Quantum information-theoretical analysis on the two-photon transitions in hydrogen isoelectronic ions under plasma confinement

A Twist for Graph Classification: Optimizing Causal Information Flow in Graph Neural Networks

Graph neural networks (GNNs) have achieved state-of-the-art results on many graph representation learning tasks by exploiting statistical correlations. However, numerous observations have shown that such correlations may not reflect the true causal mechanisms underlying the data and thus may hamper the ability of the model to generalize beyond the observed distribution. To address this problem, we propose an Information-based Causal Learning (ICL) framework that combines information theory and causality to analyze and improve graph representation learning to transform information relevance to causal dependence. Specifically, we first introduce a multi-objective mutual information optimization objective derived from information-theoretic analysis and causal learning principles to simultaneously extract invariant and interpretable causal information and reduce reliance on non-causal information in correlations. To optimize this multi-objective objective, we enable a causal disentanglement layer that effectively decouples the causal and non-causal information in the graph representations. Moreover, due to the intractability of mutual information estimation, we derive variational bounds that enable us to transform the above objective into a tractable loss function. To balance the multiple information objectives and avoid optimization conflicts, we leverage multi-objective gradient descent to achieve a stable and efficient transformation from informational correlation to causal dependency. Our approach provides important insights into modulating the information flow in GNNs to enhance their reliability and generalization. Extensive experiments demonstrate that our approach significantly improves the robustness and interpretability of GNNs across different distribution shifts. Visual analysis demonstrates how our method converts informative dependencies in representations into causal dependencies.

Information-Theoretical Analysis of the Cycle of Creation of Knowledge and Meaning in Brains under Multiple Cognitive Modalities.

It is of great interest to develop advanced sensory technologies allowing non-invasive monitoring of neural correlates of cognitive processing in people performing everyday tasks. A lot of progress has been reported in recent years in this research area using scalp EEG arrays, but the high level of noise in the electrode signals poses a lot of challenges. This study presents results of detailed statistical analysis of experimental data on the cycle of creation of knowledge and meaning in human brains under multiple cognitive modalities. We measure brain dynamics using a HydroCel Geodesic Sensor Net, 128-electrode dense-array electroencephalography (EEG). We compute a pragmatic information (PI) index derived from analytic amplitude and phase, by Hilbert transforming the EEG signals of 20 participants in six modalities, which combine various audiovisual stimuli, leading to different mental states, including relaxed and cognitively engaged conditions. We derive several relevant measures to classify different brain states based on the PI indices. We demonstrate significant differences between engaged brain states that require sensory information processing to create meaning and knowledge for intentional action, and relaxed-meditative brain states with less demand on psychophysiological resources. We also point out that different kinds of meanings may lead to different brain dynamics and behavioral responses.

Information-theoretical analysis of the neural code for decoupled face representation.

Processing faces accurately and efficiently is a key capability of humans and other animals that engage in sophisticated social tasks. Recent studies reported a decoupled coding for faces in the primate inferotemporal cortex, with two separate neural populations coding for the geometric position of (texture-free) facial landmarks and for the image texture at fixed landmark positions, respectively. Here, we formally assess the efficiency of this decoupled coding by appealing to the information-theoretic notion of description length, which quantifies the amount of information that is saved when encoding novel facial images, with a given precision. We show that despite decoupled coding describes the facial images in terms of two sets of principal components (of landmark shape and image texture), it is more efficient (i.e., yields more information compression) than the encoding in terms of the image principal components only, which corresponds to the widely used eigenface method. The advantage of decoupled coding over eigenface coding increases with image resolution and is especially prominent when coding variants of training set images that only differ in facial expressions. Moreover, we demonstrate that decoupled coding entails better performance in three different tasks: the representation of facial images, the (daydream) sampling of novel facial images, and the recognition of facial identities and gender. In summary, our study provides a first principle perspective on the efficiency and accuracy of the decoupled coding of facial stimuli reported in the primate inferotemporal cortex.

Information representation in an oscillating neural field model modulated by working memory signals.

We study how stimulus information can be represented in the dynamical signatures of an oscillatory model of neural activity-a model whose activity can be modulated by input akin to signals involved in working memory (WM). We developed a neural field model, tuned near an oscillatory instability, in which the WM-like input can modulate the frequency and amplitude of the oscillation. Our neural field model has a spatial-like domain in which an input that preferentially targets a point-a stimulus feature-on the domain will induce feature-specific activity changes. These feature-specific activity changes affect both the mean rate of spikes and the relative timing of spiking activity to the global field oscillation-the phase of the spiking activity. From these two dynamical signatures, we define both a spike rate code and an oscillatory phase code. We assess the performance of these two codes to discriminate stimulus features using an information-theoretic analysis. We show that global WM input modulations can enhance phase code discrimination while simultaneously reducing rate code discrimination. Moreover, we find that the phase code performance is roughly two orders of magnitude larger than that of the rate code defined for the same model solutions. The results of our model have applications to sensory areas of the brain, to which prefrontal areas send inputs reflecting the content of WM. These WM inputs to sensory areas have been established to induce oscillatory changes similar to our model. Our model results suggest a mechanism by which WM signals may enhance sensory information represented in oscillatory activity beyond the comparatively weak representations based on the mean rate activity.

Propofol Reversibly Attenuates Short-Range Microstate Ordering and 20Hz Microstate Oscillations.

Microstate sequences summarize the changing voltage patterns measured by electroencephalography, using a clustering approach to reduce the high dimensionality of the underlying data. A common approach is to restrict the pattern matching step to local maxima of the global field power (GFP) and to interpolate the microstate fit in between. In this study, we investigate how the anesthetic propofol affects microstate sequence periodicity and predictability, and how these metrics are changed by interpolation. We performed two frequency analyses on microstate sequences, one based on time-lagged mutual information, the other based on Fourier transform methodology, and quantified the effects of interpolation. Resting-state microstate sequences had a 20Hz frequency peak related to dominant 10Hz (alpha) rhythms, and the Fourier approach demonstrated that all five microstate classes followed this frequency. The 20Hz periodicity was reversibly attenuated under moderate propofol sedation, as shown by mutual information and Fourier analysis. Characteristic microstate frequencies could only be observed in non-interpolated microstate sequences and were masked by smoothing effects of interpolation. Information-theoretic analysis revealed faster microstate dynamics and larger entropy rates under propofol, whereas Shannon entropy did not change significantly. In moderate sedation, active information storage decreased for non-interpolated sequences. Signatures of non-equilibrium dynamics were observed in non-interpolated sequences, but no changes were observed between sedation levels. All changes occurred while subjects were able to perform an auditory perception task. In summary, we show that low dose propofol reversibly increases the randomness of microstate sequences and attenuates microstate oscillations without correlation to cognitive task performance. Microstate dynamics between GFP peaks reflect physiological processes that are not accessible in interpolated sequences.

An Effecient Joint Source-Channel Coding Scheme for Wireless Hierarchical Federated Learning and Its Information-Theoretic Analysis

An Effecient Joint Source-Channel Coding Scheme for Wireless Hierarchical Federated Learning and Its Information-Theoretic Analysis

Ratio Shift Keying Modulation for Time-Varying Molecular Communication Channels.

Molecular Communications (MC) is a bio-inspired communication technique that uses molecules to encode and transfer information. Many efforts have been devoted to developing novel modulation techniques for MC based on various distinguishable characteristics of molecules, such as their concentrations or types. In this paper, we investigate a particular modulation scheme called Ratio Shift Keying (RSK), where the information is encoded in the concentration ratio of two different types of molecules. RSK modulation is hypothesized to enable accurate information transfer in dynamic MC scenarios where the time-varying channel characteristics affect both types of molecules equally. To validate this hypothesis, we first conduct an information-theoretical analysis of RSK modulation and derive the capacity of the end-to-end MC channel where the receiver estimates concentration ratio based on ligand-receptor binding statistics in an optimal or suboptimal manner. We then analyze the error performance of RSK modulation in a practical time-varying MC scenario, that is mobile MC, in which both the transmitter and the receiver undergo diffusion-based propagation. Our numerical and analytical results, obtained for varying levels of similarity between the ligand types used for ratio-encoding, and varying number of receptors, show that RSK can significantly outperform the most commonly considered MC modulation technique, concentration shift keying (CSK), in dynamic MC scenarios.

The information culture of teenagers in the context of modern education paradigm

The growing information streams and expanding information space have made information easy accessible while the need for its filtration, critical reevaluation, and clear-cut assessment criteria emerged. The younger generation has to master skills of retrieving, analyzing and using reliable information to support their lives, career and social success. Being skillful Internet users, the teenagers yet demonstrate the low level of information culture. However, the requirement to educate information culture is formulated in The Federal State Educational Standard of Comprehensive Secondary Education (FGOS) in the disciplines “Mathematics”, “Informatics” and “Literature”. The results of the regional stage of the All-Russian Olympiad of School Students – the republican Olympiad in the personal information culture (Republic of Buryatia) – have demonstrated non-conformity of teenagers’ performance with the FGOS standard. No similar Olympiads are held in other RF regions, which makes the study particularly valuable. The authors emphasize that the Olympiad in this subject area has to be implemented in other regions. They applied several research methods to assess information literacy, namely empirical (observation, comparison) and theoretical (analysis, generalization). The analysis is based on students’ answers. The newness of the study lies in characterizing information culture of Buryat schoolchildren and specifying major challenges, substantiating and extending obtained data, and suggesting new vectors of information culture regional studies.

Information-Theoretic Analysis of a Model of CAR-4-1BB-Mediated NFκB Activation

Systems biology utilizes computational approaches to examine an array of biological processes, such as cell signaling, metabolomics and pharmacology. This includes mathematical modeling of CAR T cells, a modality of cancer therapy by which genetically engineered immune cells recognize and combat a cancerous target. While successful against hematologic malignancies, CAR T cells have shown limited success against other cancer types. Thus, more research is needed to understand their mechanisms of action and leverage their full potential. In our work, we set out to apply information theory on a mathematical model of NFκB signaling initiated by the CAR following antigen encounter. First, we estimated channel capacity for CAR-4-1BB-mediated NFκB signal transduction. Next, we evaluated the pathway’s ability to distinguish contrasting “low” and “high” antigen concentration levels, depending on the amount of variability in protein concentrations. Finally, we assessed the fidelity by which NFκB activation reflects the encountered antigen concentration, depending on the prevalence of antigen-positive targets in tumor population. We found that in most scenarios, fold change in the nuclear concentration of NFκB carries a higher channel capacity for the pathway than NFκB’s absolute response. Additionally, we found that most errors in transducing the antigen signal through the pathway skew towards underestimating the concentration of encountered antigen. Finally, we found that disabling IKKβ deactivation could increase signaling fidelity against targets with antigen-negative cells. Our information-theoretic analysis of signal transduction can provide novel perspectives on biological signaling, as well as enable a more informed path to cell engineering.

Information-theoretic analyses of neural data to minimize the effect of researchers' assumptions in predictive coding studies.

Studies investigating neural information processing often implicitly ask both, which processing strategy out of several alternatives is used and how this strategy is implemented in neural dynamics. A prime example are studies on predictive coding. These often ask whether confirmed predictions about inputs or prediction errors between internal predictions and inputs are passed on in a hierarchical neural system-while at the same time looking for the neural correlates of coding for errors and predictions. If we do not know exactly what a neural system predicts at any given moment, this results in a circular analysis-as has been criticized correctly. To circumvent such circular analysis, we propose to express information processing strategies (such as predictive coding) by local information-theoretic quantities, such that they can be estimated directly from neural data. We demonstrate our approach by investigating two opposing accounts of predictive coding-like processing strategies, where we quantify the building blocks of predictive coding, namely predictability of inputs and transfer of information, by local active information storage and local transfer entropy. We define testable hypotheses on the relationship of both quantities, allowing us to identify which of the assumed strategies was used. We demonstrate our approach on spiking data collected from the retinogeniculate synapse of the cat (N = 16). Applying our local information dynamics framework, we are able to show that the synapse codes for predictable rather than surprising input. To support our findings, we estimate quantities applied in the partial information decomposition framework, which allow to differentiate whether the transferred information is primarily bottom-up sensory input or information transferred conditionally on the current state of the synapse. Supporting our local information-theoretic results, we find that the synapse preferentially transfers bottom-up information.

Do we really need dice? The hidden region-size biases of segmentation losses

Most segmentation losses are arguably variants of the Cross-Entropy (CE) or Dice losses. On the surface, these two categories of losses (i.e., distribution based vs. geometry based) seem unrelated, and there is no clear consensus as to which category is a better choice, with varying performances for each across different benchmarks and applications. Furthermore, it is widely argued within the medical-imaging community that Dice and CE are complementary, which has motivated the use of compound CE-Dice losses. In this work, we provide a theoretical analysis, which shows that CE and Dice share a much deeper connection than previously thought. First, we show that, from a constrained-optimization perspective, they both decompose into two components, i.e., a similar ground-truth matching term, which pushes the predicted foreground regions towards the ground-truth, and a region-size penalty term imposing different biases on the size (or proportion) of the predicted regions. Then, we provide bound relationships and an information-theoretic analysis, which uncover hidden region-size biases: Dice has an intrinsic bias towards specific extremely imbalanced solutions, whereas CE implicitly encourages the ground-truth region proportions. Our theoretical results explain the wide experimental evidence in the medical-imaging literature, whereby Dice losses bring improvements for imbalanced segmentation. It also explains why CE dominates natural-image problems with diverse class proportions, in which case Dice might have difficulty adapting to different region-size distributions. Based on our theoretical analysis, we propose a principled and simple solution, which enables to control explicitly the region-size bias. The proposed method integrates CE with explicit terms based on L1 or the KL divergence, which encourage segmenting region proportions to match target class proportions, thereby mitigating class imbalance but without losing generality. Comprehensive experiments and ablation studies over different losses and applications validate our theoretical analysis, as well as the effectiveness of explicit and simple region-size terms. The code is available at https://github.com/by-liu/SegLossBias .

The color communication game

There is clear diversity among speakers of a typical language in how colors are named. What is the impact of this diversity on the people’s ability to communicate about color? Is there a gap between a person’s general understanding of the color terms in their native language and how they understand a particular term that denotes a particular color sample? Seventy English-speaking dyads and 63 Somali-speaking dyads played the Color Communication Game, where the “sender” in each dyad named 30 color samples as they would in any color-naming study, then the “receiver” chose the sample they thought the sender intended to communicate. English speakers played again, under instructions to intentionally communicate color sample identity. Direct comparison of senders’ samples and receivers’ choices revealed categorical understanding of colors without considering color naming data. Although Somali-speaking senders provided fewer color terms, interpersonal Mutual Information (MI) calculated from color naming data was similarly below optimal for both groups, and English-speaking dyads’ MI did not improve with experience. Both groups revealed superior understanding of color terms because receivers showed better exactly-correct selection performance than was predicted by simulation from their senders’ color-naming data. This study highlights limitations on information-theoretic analyses of color naming data.

Information-theoretical analysis of Dirac and nonrelativistic quantum oscillators

Information-theoretical analysis of Dirac and nonrelativistic quantum oscillators