Information-theoretic Analysis Research Articles

Quantum information-theoretical analysis on the two-photon transitions in hydrogen isoelectronic ions under plasma confinement

Quantum information-theoretical analysis on the two-photon transitions in hydrogen isoelectronic ions under plasma confinement

A Twist for Graph Classification: Optimizing Causal Information Flow in Graph Neural Networks

Graph neural networks (GNNs) have achieved state-of-the-art results on many graph representation learning tasks by exploiting statistical correlations. However, numerous observations have shown that such correlations may not reflect the true causal mechanisms underlying the data and thus may hamper the ability of the model to generalize beyond the observed distribution. To address this problem, we propose an Information-based Causal Learning (ICL) framework that combines information theory and causality to analyze and improve graph representation learning to transform information relevance to causal dependence. Specifically, we first introduce a multi-objective mutual information optimization objective derived from information-theoretic analysis and causal learning principles to simultaneously extract invariant and interpretable causal information and reduce reliance on non-causal information in correlations. To optimize this multi-objective objective, we enable a causal disentanglement layer that effectively decouples the causal and non-causal information in the graph representations. Moreover, due to the intractability of mutual information estimation, we derive variational bounds that enable us to transform the above objective into a tractable loss function. To balance the multiple information objectives and avoid optimization conflicts, we leverage multi-objective gradient descent to achieve a stable and efficient transformation from informational correlation to causal dependency. Our approach provides important insights into modulating the information flow in GNNs to enhance their reliability and generalization. Extensive experiments demonstrate that our approach significantly improves the robustness and interpretability of GNNs across different distribution shifts. Visual analysis demonstrates how our method converts informative dependencies in representations into causal dependencies.

Information-Theoretical Analysis of the Cycle of Creation of Knowledge and Meaning in Brains under Multiple Cognitive Modalities.

It is of great interest to develop advanced sensory technologies allowing non-invasive monitoring of neural correlates of cognitive processing in people performing everyday tasks. A lot of progress has been reported in recent years in this research area using scalp EEG arrays, but the high level of noise in the electrode signals poses a lot of challenges. This study presents results of detailed statistical analysis of experimental data on the cycle of creation of knowledge and meaning in human brains under multiple cognitive modalities. We measure brain dynamics using a HydroCel Geodesic Sensor Net, 128-electrode dense-array electroencephalography (EEG). We compute a pragmatic information (PI) index derived from analytic amplitude and phase, by Hilbert transforming the EEG signals of 20 participants in six modalities, which combine various audiovisual stimuli, leading to different mental states, including relaxed and cognitively engaged conditions. We derive several relevant measures to classify different brain states based on the PI indices. We demonstrate significant differences between engaged brain states that require sensory information processing to create meaning and knowledge for intentional action, and relaxed-meditative brain states with less demand on psychophysiological resources. We also point out that different kinds of meanings may lead to different brain dynamics and behavioral responses.

Information-theoretical analysis of the neural code for decoupled face representation.

Processing faces accurately and efficiently is a key capability of humans and other animals that engage in sophisticated social tasks. Recent studies reported a decoupled coding for faces in the primate inferotemporal cortex, with two separate neural populations coding for the geometric position of (texture-free) facial landmarks and for the image texture at fixed landmark positions, respectively. Here, we formally assess the efficiency of this decoupled coding by appealing to the information-theoretic notion of description length, which quantifies the amount of information that is saved when encoding novel facial images, with a given precision. We show that despite decoupled coding describes the facial images in terms of two sets of principal components (of landmark shape and image texture), it is more efficient (i.e., yields more information compression) than the encoding in terms of the image principal components only, which corresponds to the widely used eigenface method. The advantage of decoupled coding over eigenface coding increases with image resolution and is especially prominent when coding variants of training set images that only differ in facial expressions. Moreover, we demonstrate that decoupled coding entails better performance in three different tasks: the representation of facial images, the (daydream) sampling of novel facial images, and the recognition of facial identities and gender. In summary, our study provides a first principle perspective on the efficiency and accuracy of the decoupled coding of facial stimuli reported in the primate inferotemporal cortex.

Information representation in an oscillating neural field model modulated by working memory signals.

We study how stimulus information can be represented in the dynamical signatures of an oscillatory model of neural activity-a model whose activity can be modulated by input akin to signals involved in working memory (WM). We developed a neural field model, tuned near an oscillatory instability, in which the WM-like input can modulate the frequency and amplitude of the oscillation. Our neural field model has a spatial-like domain in which an input that preferentially targets a point-a stimulus feature-on the domain will induce feature-specific activity changes. These feature-specific activity changes affect both the mean rate of spikes and the relative timing of spiking activity to the global field oscillation-the phase of the spiking activity. From these two dynamical signatures, we define both a spike rate code and an oscillatory phase code. We assess the performance of these two codes to discriminate stimulus features using an information-theoretic analysis. We show that global WM input modulations can enhance phase code discrimination while simultaneously reducing rate code discrimination. Moreover, we find that the phase code performance is roughly two orders of magnitude larger than that of the rate code defined for the same model solutions. The results of our model have applications to sensory areas of the brain, to which prefrontal areas send inputs reflecting the content of WM. These WM inputs to sensory areas have been established to induce oscillatory changes similar to our model. Our model results suggest a mechanism by which WM signals may enhance sensory information represented in oscillatory activity beyond the comparatively weak representations based on the mean rate activity.

Propofol Reversibly Attenuates Short-Range Microstate Ordering and 20Hz Microstate Oscillations.

Microstate sequences summarize the changing voltage patterns measured by electroencephalography, using a clustering approach to reduce the high dimensionality of the underlying data. A common approach is to restrict the pattern matching step to local maxima of the global field power (GFP) and to interpolate the microstate fit in between. In this study, we investigate how the anesthetic propofol affects microstate sequence periodicity and predictability, and how these metrics are changed by interpolation. We performed two frequency analyses on microstate sequences, one based on time-lagged mutual information, the other based on Fourier transform methodology, and quantified the effects of interpolation. Resting-state microstate sequences had a 20Hz frequency peak related to dominant 10Hz (alpha) rhythms, and the Fourier approach demonstrated that all five microstate classes followed this frequency. The 20Hz periodicity was reversibly attenuated under moderate propofol sedation, as shown by mutual information and Fourier analysis. Characteristic microstate frequencies could only be observed in non-interpolated microstate sequences and were masked by smoothing effects of interpolation. Information-theoretic analysis revealed faster microstate dynamics and larger entropy rates under propofol, whereas Shannon entropy did not change significantly. In moderate sedation, active information storage decreased for non-interpolated sequences. Signatures of non-equilibrium dynamics were observed in non-interpolated sequences, but no changes were observed between sedation levels. All changes occurred while subjects were able to perform an auditory perception task. In summary, we show that low dose propofol reversibly increases the randomness of microstate sequences and attenuates microstate oscillations without correlation to cognitive task performance. Microstate dynamics between GFP peaks reflect physiological processes that are not accessible in interpolated sequences.

Ratio Shift Keying Modulation for Time-Varying Molecular Communication Channels.

Molecular Communications (MC) is a bio-inspired communication technique that uses molecules to encode and transfer information. Many efforts have been devoted to developing novel modulation techniques for MC based on various distinguishable characteristics of molecules, such as their concentrations or types. In this paper, we investigate a particular modulation scheme called Ratio Shift Keying (RSK), where the information is encoded in the concentration ratio of two different types of molecules. RSK modulation is hypothesized to enable accurate information transfer in dynamic MC scenarios where the time-varying channel characteristics affect both types of molecules equally. To validate this hypothesis, we first conduct an information-theoretical analysis of RSK modulation and derive the capacity of the end-to-end MC channel where the receiver estimates concentration ratio based on ligand-receptor binding statistics in an optimal or suboptimal manner. We then analyze the error performance of RSK modulation in a practical time-varying MC scenario, that is mobile MC, in which both the transmitter and the receiver undergo diffusion-based propagation. Our numerical and analytical results, obtained for varying levels of similarity between the ligand types used for ratio-encoding, and varying number of receptors, show that RSK can significantly outperform the most commonly considered MC modulation technique, concentration shift keying (CSK), in dynamic MC scenarios.

The information culture of teenagers in the context of modern education paradigm

The growing information streams and expanding information space have made information easy accessible while the need for its filtration, critical reevaluation, and clear-cut assessment criteria emerged. The younger generation has to master skills of retrieving, analyzing and using reliable information to support their lives, career and social success. Being skillful Internet users, the teenagers yet demonstrate the low level of information culture. However, the requirement to educate information culture is formulated in The Federal State Educational Standard of Comprehensive Secondary Education (FGOS) in the disciplines “Mathematics”, “Informatics” and “Literature”. The results of the regional stage of the All-Russian Olympiad of School Students – the republican Olympiad in the personal information culture (Republic of Buryatia) – have demonstrated non-conformity of teenagers’ performance with the FGOS standard. No similar Olympiads are held in other RF regions, which makes the study particularly valuable. The authors emphasize that the Olympiad in this subject area has to be implemented in other regions. They applied several research methods to assess information literacy, namely empirical (observation, comparison) and theoretical (analysis, generalization). The analysis is based on students’ answers. The newness of the study lies in characterizing information culture of Buryat schoolchildren and specifying major challenges, substantiating and extending obtained data, and suggesting new vectors of information culture regional studies.

Information-Theoretic Analysis of a Model of CAR-4-1BB-Mediated NFκB Activation

Systems biology utilizes computational approaches to examine an array of biological processes, such as cell signaling, metabolomics and pharmacology. This includes mathematical modeling of CAR T cells, a modality of cancer therapy by which genetically engineered immune cells recognize and combat a cancerous target. While successful against hematologic malignancies, CAR T cells have shown limited success against other cancer types. Thus, more research is needed to understand their mechanisms of action and leverage their full potential. In our work, we set out to apply information theory on a mathematical model of NFκB signaling initiated by the CAR following antigen encounter. First, we estimated channel capacity for CAR-4-1BB-mediated NFκB signal transduction. Next, we evaluated the pathway’s ability to distinguish contrasting “low” and “high” antigen concentration levels, depending on the amount of variability in protein concentrations. Finally, we assessed the fidelity by which NFκB activation reflects the encountered antigen concentration, depending on the prevalence of antigen-positive targets in tumor population. We found that in most scenarios, fold change in the nuclear concentration of NFκB carries a higher channel capacity for the pathway than NFκB’s absolute response. Additionally, we found that most errors in transducing the antigen signal through the pathway skew towards underestimating the concentration of encountered antigen. Finally, we found that disabling IKKβ deactivation could increase signaling fidelity against targets with antigen-negative cells. Our information-theoretic analysis of signal transduction can provide novel perspectives on biological signaling, as well as enable a more informed path to cell engineering.

Information-theoretic analyses of neural data to minimize the effect of researchers' assumptions in predictive coding studies.

Studies investigating neural information processing often implicitly ask both, which processing strategy out of several alternatives is used and how this strategy is implemented in neural dynamics. A prime example are studies on predictive coding. These often ask whether confirmed predictions about inputs or prediction errors between internal predictions and inputs are passed on in a hierarchical neural system-while at the same time looking for the neural correlates of coding for errors and predictions. If we do not know exactly what a neural system predicts at any given moment, this results in a circular analysis-as has been criticized correctly. To circumvent such circular analysis, we propose to express information processing strategies (such as predictive coding) by local information-theoretic quantities, such that they can be estimated directly from neural data. We demonstrate our approach by investigating two opposing accounts of predictive coding-like processing strategies, where we quantify the building blocks of predictive coding, namely predictability of inputs and transfer of information, by local active information storage and local transfer entropy. We define testable hypotheses on the relationship of both quantities, allowing us to identify which of the assumed strategies was used. We demonstrate our approach on spiking data collected from the retinogeniculate synapse of the cat (N = 16). Applying our local information dynamics framework, we are able to show that the synapse codes for predictable rather than surprising input. To support our findings, we estimate quantities applied in the partial information decomposition framework, which allow to differentiate whether the transferred information is primarily bottom-up sensory input or information transferred conditionally on the current state of the synapse. Supporting our local information-theoretic results, we find that the synapse preferentially transfers bottom-up information.

Do we really need dice? The hidden region-size biases of segmentation losses

Most segmentation losses are arguably variants of the Cross-Entropy (CE) or Dice losses. On the surface, these two categories of losses (i.e., distribution based vs. geometry based) seem unrelated, and there is no clear consensus as to which category is a better choice, with varying performances for each across different benchmarks and applications. Furthermore, it is widely argued within the medical-imaging community that Dice and CE are complementary, which has motivated the use of compound CE-Dice losses. In this work, we provide a theoretical analysis, which shows that CE and Dice share a much deeper connection than previously thought. First, we show that, from a constrained-optimization perspective, they both decompose into two components, i.e., a similar ground-truth matching term, which pushes the predicted foreground regions towards the ground-truth, and a region-size penalty term imposing different biases on the size (or proportion) of the predicted regions. Then, we provide bound relationships and an information-theoretic analysis, which uncover hidden region-size biases: Dice has an intrinsic bias towards specific extremely imbalanced solutions, whereas CE implicitly encourages the ground-truth region proportions. Our theoretical results explain the wide experimental evidence in the medical-imaging literature, whereby Dice losses bring improvements for imbalanced segmentation. It also explains why CE dominates natural-image problems with diverse class proportions, in which case Dice might have difficulty adapting to different region-size distributions. Based on our theoretical analysis, we propose a principled and simple solution, which enables to control explicitly the region-size bias. The proposed method integrates CE with explicit terms based on L1 or the KL divergence, which encourage segmenting region proportions to match target class proportions, thereby mitigating class imbalance but without losing generality. Comprehensive experiments and ablation studies over different losses and applications validate our theoretical analysis, as well as the effectiveness of explicit and simple region-size terms. The code is available at https://github.com/by-liu/SegLossBias .

The color communication game

There is clear diversity among speakers of a typical language in how colors are named. What is the impact of this diversity on the people’s ability to communicate about color? Is there a gap between a person’s general understanding of the color terms in their native language and how they understand a particular term that denotes a particular color sample? Seventy English-speaking dyads and 63 Somali-speaking dyads played the Color Communication Game, where the “sender” in each dyad named 30 color samples as they would in any color-naming study, then the “receiver” chose the sample they thought the sender intended to communicate. English speakers played again, under instructions to intentionally communicate color sample identity. Direct comparison of senders’ samples and receivers’ choices revealed categorical understanding of colors without considering color naming data. Although Somali-speaking senders provided fewer color terms, interpersonal Mutual Information (MI) calculated from color naming data was similarly below optimal for both groups, and English-speaking dyads’ MI did not improve with experience. Both groups revealed superior understanding of color terms because receivers showed better exactly-correct selection performance than was predicted by simulation from their senders’ color-naming data. This study highlights limitations on information-theoretic analyses of color naming data.

Information-theoretical analysis of Dirac and nonrelativistic quantum oscillators

Information-theoretical analysis of Dirac and nonrelativistic quantum oscillators

Substantiation of creation of transdermal forms of drug delivery with antihypertensive action

The article presents the theoretical justification for the choice of active pharmaceutical ingredients of antihypertensive action for the creation of new forms of delivery - transdermal therapeutic systems.
 Purpose: monitoring of studies on the use of angiotensin-converting enzyme (ACE) inhibitors in the development of transdermal drugs for the treatment of hypertension.
 Materials: open access electronic resources of scientific periodicals from around the world: Google Scholar (https://scholar.google.com/); PubMed (https://pubmed.ncbi.nlm.nih.gov/); Sciencedirect (https://www.sciencedirect.com/); Pubchem (https://pubchem.ncbi.nlm.nih.gov/); National Library of Ukraine named after V. I. Vernadskyi (http://www.nbuv.gov.ua/); Ukrainian Institute of Intellectual Property (Ukrpatent) (https://sis.ukrpatent.org/uk/); Industrial Property (https://base.uipv.org/searchBul/); USPTO. The United States Patent and Trademark Office (https://www.uspto.gov/patents/search); European Patent Office (EPO) (https://www.epo.org/).
 Methods of the research: information search, theoretical analysis and systematization of data from scientific sources, logical analysis.
 Results: a review of scientific sources was carried out, and an analysis of ACE inhibitors as candidates for the creation of transdermal forms for the treatment of arterial hypertension was carried out. The pharmacological and physicochemical aspects of the possibility of their use for introduction through the skin are defined. It has been established that the search and development of the latest means of treatment of arterial hypertension, which would significantly increase the duration and quality of life of patients, are extremely relevant. Transdermal drug delivery systems are one of the pharmaceutical products being developed on the world market, and their use allows overcoming the associated disadvantages of other delivery routes.
 Conclusions: the review of domestic and foreign literature confirmed the relevance of biopharmaceutical research in the field of development of innovative dosage forms - transdermal therapeutic systems for the treatment of hypertension.
 The choice of promising antihypertensive active pharmaceutical ingredients of the ACE group (enalapril maleate, lisinopril dihydrate, and captopril) is substantiated, taking into account their specific physicochemical properties that are suitable for penetration through the skin.
 The market for transdermal drug delivery is increasing, and there is a prospect of higher growth rates in this market in the coming years

Non-Markov models of single-molecule dynamics from information-theoretical analysis of trajectories.

Whether single-molecule trajectories, observed experimentally or in molecular simulations, can be described using simple models such as biased diffusion is a subject of considerable debate. Memory effects and anomalous diffusion have been reported in a number of studies, but directly inferring such effects from trajectories, especially given limited temporal and/or spatial resolution, has been a challenge. Recently, we proposed that this can be achieved with information-theoretical analysis of trajectories, which is based on the general observation that non-Markov effects make trajectories more predictable and, thus, more "compressible" by lossless compression algorithms. Toy models where discrete molecular states evolve in time were shown to be amenable to such analysis, but its application to continuous trajectories presents a challenge: the trajectories need to be digitized first, and digitization itself introduces non-Markov effects that depend on the specifics of how trajectories are sampled. Here we develop a milestoning-based method for information-theoretical analysis of continuous trajectories and show its utility in application to Markov and non-Markov models and to trajectories obtained from molecular simulations.

Negation of a probability distribution: An information theoretic analysis

Yager (2014) proposed a transformation for negating the happening of an uncertain event which opposes the occurrence of any uncertain event by redistributing its probability equally among the other outcomes. Yager’s negation applied repeatedly on any probability distribution converges to the uniform distribution (all events having identical probability of occurrence). In most cases, uniform distribution is the maximum entropy probability distribution (MEPD) as it has maximum uncertainty inherent in it. However, if some information is available regarding the occurrence of events associated with a random experiment, then MEPD may or may not be a uniform distribution. In this work, we have first explored some new properties of Yager’s negation and then investigated Yager’s negation when we have some additional information about the probability distribution other than the natural constraints. The MEPD in the presence of this additional information is determined using the negation of probabilities. It is shown that the existence of MEPD (and as a result the maximum entropy) largely depends on the parameters of the additional constraints. Some numerical examples have been considered for comparing the MEPD with and without additional constraints.

On the relation between the subadditivity cone and the quantum entropy cone

Given a multipartite quantum system, what are the possible ways to impose mutual independence among some subsystems, and the presence of correlations among others, such that there exists a quantum state which satisfies these demands? This question and the related notion of a pattern of marginal independence (PMI) were introduced in [1], and then argued in [2] to be central in the derivation of the holographic entropy cone. Here we continue the general information theoretic analysis of the PMIs allowed by strong subadditivity (SSA) initiated in [1]. We show how the computation of these PMIs simplifies when SSA is replaced by a weaker constraint, dubbed Klein’s condition (KC), which follows from the necessary condition for the saturation of subadditivity (SA). Formulating KC in the language of partially ordered sets, we show that the set of PMIs compatible with KC forms a lattice, and we investigate several of its structural properties. One of our main results is the identification of a specific lower dimensional face of the SA cone that contains on its boundary all the extreme rays (beyond Bell pairs) that can possibly be realized by quantum states. We verify that for four or more parties, KC is strictly weaker than SSA, but nonetheless the PMIs compatible with SSA can easily be derived from the KC-compatible ones. For the special case of 1-dimensional PMIs, we conjecture that KC and SSA are in fact equivalent. To make the presentation self-contained, we review the key ingredients from lattice theory as needed.

A discriminative information-theoretical analysis of the regularity gradient in inflectional morphology

Over the last decades, several independent lines of research in morphology have questioned the hypothesis of a direct correspondence between sublexical units and their mental correlates. Word and paradigm models of morphology shifted the fundamental part-whole relation in an inflection system onto the relation between individual inflected word forms and inflectional paradigms. In turn, the use of artificial neural networks of densely interconnected parallel processing nodes for morphology learning marked a radical departure from a morpheme-based view of the mental lexicon. Lately, in computational models of Discriminative Learning, a network architecture has been combined with an uncertainty reducing mechanism that dispenses with the need for a one-to-one association between formal contrasts and meanings, leading to the dissolution of a discrete notion of the morpheme.The paper capitalises on these converging lines of development to offer a unifying information-theoretical, simulation-based analysis of the costs incurred in processing (ir)regularly inflected forms belonging to the verb systems of English, German, French, Spanish and Italian. Using Temporal Self-Organising Maps as a computational model of lexical storage and access, we show that a discriminative, recurrent neural network, based on Rescorla-Wagner’s equations, can replicate speakers’ exquisite sensitivity to widespread effects of word frequency, paradigm entropy and morphological (ir)regularity in lexical processing. The evidence suggests an explanatory hypothesis linking Word and paradigm morphology with principles of information theory and human perception of morphological structure. According to this hypothesis, the ways more or less regularly inflected words are structured in the mental lexicon are more related to a reduction in processing uncertainty and maximisation of predictive efficiency than to economy of storage.

Information-theoretic Analysis of Longitude Distribution of Photospheric Magnetic Fields from MDI/HMI Synoptic Maps: Evidence for Rossby Waves

Much of the research on the magnetic activity of the Sun has been focused on its axisymmetric component. However, the longitudinal complexity plays a fundamental role in the solar magnetic activity. Rossby waves have recently been proposed as a fundamental mechanism regarding the nonaxisymmetric nature of the solar magnetic fields. Here, we use HMI and MDI magnetic field synoptic maps to evaluate the magnetic field structures’ (mainly active regions) organization and propagation as a function of time and latitude. We demonstrate, using information theory, that the organization of longitudinal structures observed on synoptic maps is proportional to the level of activity at a given latitude. We further show that this organization on the longitudinal structures is persistent and due to long-lived features. The drift velocity of these long-lived photospheric features is inferred and is shown to significantly vary with latitude, and is compatible with the phase speed of tachocline magnetic Rossby waves with a toroidal field in the range of 5–10 kG. Our results suggest that Rossby waves contribute to the organization and propagation of photospheric magnetic features on the timescale of several months and beyond.

An information theoretic Bayesian uncertainty analysis of AEM systems over Menindee Lake, Australia

SUMMARY Long-range, active-source airborne electromagnetic (AEM) systems for near-surface conductivity imaging fall into two categories: helicopter (rotary-wing) borne or fixed-wing aircraft borne. A multitude of factors such as flying height, transmitter loop area and current, source waveforms, aerodynamic stability and data stacking times contribute to the geological resolvability of the subsurface. A comprehensive comparison of the relative merits of each system considering all such factors is difficult, but test flights over well-constrained subsurface geology with downhole induction logs are extremely useful for resolution studies. However, given the non-linear nature of the electromagnetic inverse problem, handling transmitter–receiver geometries in fixed-wing aircraft is especially challenging. As a consequence of this non-linearity, inspecting the closeness of downhole conductivities to deterministic inversion results is not sufficient for studying resolvability. A more comprehensive picture is provided by examining the variation in probability mass of the depth-wise Bayesian posterior conductivity distributions for each kind of AEM system within an information theoretic framework. For this purpose, probabilistic inversions of data must be carried out. Each acquiring system should fly over the same geology, survey noise levels must be measured and the same prior probabilities on conductivity must be used. With both synthetic models as well as real data from over the Menindee calibration range in New South Wales, Australia, we shed new light on the matter of AEM inverse model uncertainty. We do this using two information theoretic attributes derived from different Kullback–Leibler divergences—Bayesian information gain, and a strictly proper scoring rule, to assess posterior probabilities estimated by a novel Bayesian inversion scheme. The inversion marginalizes fixed-wing geometry attributes as generic nuisance parameters during Markov chain sampling. This is the first time-domain AEM study we know of, that compares nuisance marginalized subsurface posterior conductivities from a fixed-wing system, with rotary-wing derived posterior conductivities. We also compare field results with induction log data where available. Finally, we estimate the information gain in each case via a covariate shift adaptation technique that has not been used before in geophysical work. Our findings have useful implications in AEM system selection, as well as in the design of better deterministic AEM inversion algorithms.