Mutual Information Rate Research Articles

Performance enhancement with geometrically shaped 4/8D-SP-50QAM for DDO-OFDM system

We design the 4-dimensional set-partitioning 50-ary quadrature amplitude modulation (4D-SP-50QAM) and 8D-SP-50QAM with 7 bit/symbol in direct detection optical orthogonal frequency division multiplexing (DDO-OFDM) systems. Quadrant multiplexing (QM) and orthant symmetry (OS) are respectively used to reduce the number of constellation points and the complexity of constellation mapping, while improving system bandwidth utilization. The proposed 8D-SP-50QAM scheme outperforms 4D-SP-50QAM, 4D-OS-64QAM and 4D 128-ary polarization-ring-switching (4D-128PRS) in terms of generalized mutual information (GMI) and bit error rate (BER) performance at various fiber length. In addition, the GMI gain increases gradually with the increase of fiber length, proving that this scheme can effectively improve system capacity and nonlinear tolerance.

Bounds on the rates of statistical divergences and mutual information via stochastic thermodynamics.

Statistical divergences are important tools in data analysis, information theory, and statistical physics, and there exist well-known inequalities on their bounds. However, in many circumstances involving temporal evolution, one needs limitations on the rates of such quantities instead. Here, several general upper bounds on the rates of some f-divergences are derived, valid for any type of stochastic dynamics (both Markovian and non-Markovian), in terms of information-like and/or thermodynamic observables. As special cases, the analytical bounds on the rate of mutual information are obtained. The major role in all those limitations is played by temporal Fisher information, characterizing the speed of global system dynamics, and some of them contain entropy production, suggesting a link with stochastic thermodynamics. Indeed, the derived inequalities can be used for estimation of minimal dissipation and global speed in thermodynamic stochastic systems. Specific applications of these inequalities in physics and neuroscience are given, which include the bounds on the rates of free energy and work in nonequilibrium systems, limits on the speed of information gain in learning synapses, as well as the bounds on the speed of predictive inference and learning rate. Overall, the derived bounds can be applied to any complex network of interacting elements, where predictability and thermodynamics of network dynamics are of prime concern.

Testing dynamic correlations and nonlinearity in bivariate time series through information measures and surrogate data analysis.

The increasing availability of time series data depicting the evolution of physical system properties has prompted the development of methods focused on extracting insights into the system behavior over time, discerning whether it stems from deterministic or stochastic dynamical systems. Surrogate data testing plays a crucial role in this process by facilitating robust statistical assessments. This ensures that the observed results are not mere occurrences by chance, but genuinely reflect the inherent characteristics of the underlying system. The initial process involves formulating a null hypothesis, which is tested using surrogate data in cases where assumptions about the underlying distributions are absent. A discriminating statistic is then computed for both the original data and each surrogate data set. Significantly deviating values between the original data and the surrogate data ensemble lead to the rejection of the null hypothesis. In this work, we present various surrogate methods designed to assess specific statistical properties in random processes. Specifically, we introduce methods for evaluating the presence of autodependencies and nonlinear dynamics within individual processes, using Information Storage as a discriminating statistic. Additionally, methods are introduced for detecting coupling and nonlinearities in bivariate processes, employing the Mutual Information Rate for this purpose. The surrogate methods introduced are first tested through simulations involving univariate and bivariate processes exhibiting both linear and nonlinear dynamics. Then, they are applied to physiological time series of Heart Period (RR intervals) and respiratory flow (RESP) variability measured during spontaneous and paced breathing. Simulations demonstrated that the proposed methods effectively identify essential dynamical features of stochastic systems. The real data application showed that paced breathing, at low breathing rate, increases the predictability of the individual dynamics of RR and RESP and dampens nonlinearity in their coupled dynamics.

An Asynchronous Training-free SSVEP-BCI Detection Algorithm for Non-Equal Prior Probability Scenarios.

SSVEP-based brain-computer interface (BCI) systems have received a lot of attention due to their relatively high Signal to Noise Ratio (SNR) and less training requirements. Most of the existing steady-state visual evoked potential (SSVEP) detection algorithms treat the prior probability of each alternative target being selected as equal. In this study, the prior probability distribution of alternative targets was introduced into the SSVEP recognition algorithm, and an asynchronous training-free SSVEP-BCI detection algorithm for non-equal prior probability scenarios was proposed. This algorithm is based on the Spatio-temporal equalization multi-window technique (STE-MW) and introduces the Maximum A Posteriori criterion (MAP), which makes full use of prior information to improve the performance of the asynchronous training-free BCI system. In addition, we proposed a mutual information-based performance evaluation metric called Mutual information rate (MIR) specifically for non-equal prior probability scenarios. This evaluation framework is designed to provide a more accurate estimation of the information transmission performance of BCI systems in such scenarios. A 10-target simulated vehicle control offline experiment involving 17 subjects showed that the proposed method improved the average MIR by 6.48%. Online free control experiments involving 12 subjects showed that the proposed method improved the average MIR by 14.93%, and significantly reduced the average instruction time. The proposed algorithm is more suitable for practical engineering application scenarios that are asynchronous and training-free; the extremely high accuracy is guaranteed while maintaining a low false alarm rate, which can be applied to asynchronous BCI systems that require high stability.

A frequency domain multiplexing scheme based on kernel density estimation for neural communication systems

Transmitting information in engineered neural communication systems is a promising solution to delay-sensitive applications for the Internet of Bio-Nanothings (IoBNTs). As widely used in wired and wireless communication systems, introducing multiplexing into neural communication system could improve channel transmission efficiency. In this paper, we model a neural communication system for IoBNTs and propose a neural signal multiplexing scheme for this system, based on frequency-division multiplexing (FDM) principles. The whole system including channel modeling, neural encoding, demultiplexing scheme, and decoding method using kernel density estimation (KDE) are presented. The optimal parameters for KDE and bit error probability are analyzed, and the performance of the proposed strategy is evaluated in terms of error rate and mutual information rate. The work can help researchers better understanding the underlying mechanism of neural multiplexing and pave the way for the implementation of IoBNT applications.

Mutual Information Rate of Gaussian and Truncated Gaussian Inputs on Intensity-Driven Signal Transduction Channels

Mutual Information Rate of Gaussian and Truncated Gaussian Inputs on Intensity-Driven Signal Transduction Channels

Decomposing the Mutual Information Rate of Heart Period and Respiration Variability Series to Assess Cardiorespiratory Interactions.

Heart rate variability results from the coupled activity of the cardiovascular and cardiorespiratory systems, which have their own internal regulation mechanisms but also interact with each other and with the autonomic nervous system to maintain homeostasis. In this work, the assessment of these physiological mechanisms is carried out decomposing the Mutual Information Rate (MIR), an information-theoretic measure of the interdependence between coupled processes, into terms of entropy rate or conditional mutual information related respectively to complexity and causality measures. These measures are computed using a non-parametric approach based on nearest-neighbors. The proposed framework is first tested on simulated autoregressive processes and then applied to experimental data consisting of heart period and respiratory time series measured in healthy subjects monitored at rest and during head-up tilt. Our results evidence that MIR decomposition is able to highlight the interdependence of short-term physiological mechanisms of cardiorespiratory interactions during postural stress.

Doubly Constrained Waveform Optimization for Integrated Sensing and Communications

This paper investigates threshold-constrained joint waveform optimization for an integrated sensing and communication (ISAC) system. Unlike existing studies, we employ mutual information (MI) and sum rate (SR) as sensing and communication metrics, respectively, and optimize the waveform under constraints to both metrics simultaneously. This provides significant flexibility in meeting system performance. We formulate three different optimization problems that constrain the radar performance only, the communication performance only, and the ISAC performance, respectively. New techniques are developed to solve the original problems, which are NP-hard and cannot be directly solved by conventional semi-definite programming (SDP) techniques. Novel gradient descent methods are developed to solve the first two problems. For the third non-convex optimization problem, we transform it into a convex problem and solve it via convex toolboxes. We also disclose the connections between three optimizations using numerical results. Finally, simulation results are provided and validate the proposed optimization solutions.

Comparison of discretization strategies for the model-free information-theoretic assessment of short-term physiological interactions

This work presents a comparison between different approaches for the model-free estimation of information-theoretic measures of the dynamic coupling between short realizations of random processes. The measures considered are the mutual information rate (MIR) between two random processes X and Y and the terms of its decomposition evidencing either the individual entropy rates of X and Y and their joint entropy rate, or the transfer entropies from X to Y and from Y to X and the instantaneous information shared by X and Y. All measures are estimated through discretization of the random variables forming the processes, performed either via uniform quantization (binning approach) or rank ordering (permutation approach). The binning and permutation approaches are compared on simulations of two coupled non-identical Hènon systems and on three datasets, including short realizations of cardiorespiratory (CR, heart period and respiration flow), cardiovascular (CV, heart period and systolic arterial pressure), and cerebrovascular (CB, mean arterial pressure and cerebral blood flow velocity) measured in different physiological conditions, i.e., spontaneous vs paced breathing or supine vs upright positions. Our results show that, with careful selection of the estimation parameters (i.e., the embedding dimension and the number of quantization levels for the binning approach), meaningful patterns of the MIR and of its components can be achieved in the analyzed systems. On physiological time series, we found that paced breathing at slow breathing rates induces less complex and more coupled CR dynamics, while postural stress leads to unbalancing of CV interactions with prevalent baroreflex coupling and to less complex pressure dynamics with preserved CB interactions. These results are better highlighted by the permutation approach, thanks to its more parsimonious representation of the discretized dynamic patterns, which allows one to explore interactions with longer memory while limiting the curse of dimensionality.

Network inference combining mutual information rate and statistical tests

In this paper, we present a method that combines information-theoretical and statistical approaches to infer connectivity in complex networks using time-series data. The method is based on estimations of the Mutual Information Rate for pairs of time-series and on statistical significance tests for connectivity acceptance using the false discovery rate method for multiple hypothesis testing. We provide the mathematical background on Mutual Information Rate, discuss the statistical significance tests and the false discovery rate. Further on, we present results for correlated normal-variates data, coupled circle and coupled logistic maps, coupled Lorenz systems and coupled stochastic Kuramoto phase oscillators. Following up, we study the effect of noise on the presented methodology in networks of coupled stochastic Kuramoto phase oscillators and of coupling heterogeneity degree on networks of coupled circle maps. We show that the method can infer the correct number and pairs of connected nodes, by means of receiver operating characteristic curves. In the more realistic case of stochastic data, we demonstrate its ability to infer the structure of the initial connectivity matrices. The method is also shown to recover the initial connectivity matrices for dynamics on the nodes of Erdős-Rényi and small-world networks with varying coupling heterogeneity in their connections. The highlight of the proposed methodology is its ability to infer the underlying network connectivity based solely on the recorded datasets.

Noise-induced synchronization stochastic resonance in Hodgkin–Huxley neurons

In this paper, based on the Hodgkin–Huxley (H-H) neuron model, three distinct types of noises are introduced into the input of the neuron. The noises are Gaussian white noise (GWN), Lévy noise and Colored noise, separately. The phenomenon of stochastic resonance under and above threshold in one-dimensional H-H neuron is discussed. First, the responses of H-H neuron to different input signal frequency, amplitude and constant offset stimulation are researched. The neuron membrane potential diagram and spectrum diagram are given. Then, the mutual information rate (MIR) is taken as the evaluation index. The results show that adding a noise of certain intensity can increase the response of neuron to the orignal signal and the MIR between the output and input of neuron has unimodal property, which proves the existence of stochastic resonance under and above the threshold. We also find that neurons have frequency selectivity and different transmission performance for different amplitude of input signals. In addition, the constant offset stimulus should not be too large. Finally, it is discovered that the adaptability of subthreshold stochastic resonance and suprathreshold stochastic resonance (SSR) characteristics characteristics of one-dimensional H-H neurons induced by three distinct types of noises is different. What is more important is that the results of this paper will be instrumental in the future researching on the affect of these three noises in the stochastic resonance of neuron networks.

Markov Chain Modeling of the End-to-End Molecular Communication System Using Ligand Receiver

Molecular communication (MC) makes use of molecules as information carriers for nanoscale and macroscale applications. In this paper, an end-to-end biochemical communication modeling of an MC system is presented. The transmitter is assumed to be a point source, and the receiver is embedded with finite ligand receptors. The inherent memory in the fluid channel is represented by a Markov chain and additionally a novel Markov chain is proposed which takes into effect the aggregated decision of the receiver corresponding to the number of molecules received on the finite individual receptors. The proposed discrete-time Markov chain can potentially replace the block of the finite number of Markov chains for different ligand receptors followed by a threshold comparator. Moreover, the capacity of the ligand channel is analyzed for independent and identically distributed (IID) binary source and upper and lower bounds on the mutual information (MI) rate are derived for the Markov inputs to the ligand channel. The performance analysis for the end-to-end MC system is done by deriving the analytical expressions of the bit error rate for both IID and Markov inputs. Finally, we find the simulated probability of error and observe that the analytical results are corroborated by the simulated ones.

Robust adaptive orthogonal frequency division multiplexing integrated radar and communications waveform design under clutter circumstances

To improve the target recognition and classification performance of integrated radar and communications system in signal-dependent interference, we propose an innovative robust adaptive orthogonal frequency division multiplexing integrated radar and communications waveform design taking into account the clutter. We set the uncertainty class for clutter and other prior information to solve the problem of system performance degradation caused by inaccurate information in actual environments. Then, we formulate the optimization function to maximize the normalized weighted sum of mutual information and data information rate subject to the constraint of the total transmitted energy. To solve the optimization problem, we propose a maximum marginal allocation algorithm that transforms the original problem into a multi-stage energy distribution problem, and performs optimal energy distribution in each stage according to the optimization criteria. Moreover, by considering a more practical scenario of multiple downlink communication users, a joint scheme for channel allocation and waveform design is presented to ensure the fairness of multiple communication users. Simulation results show the effectiveness and feasibility of the designed waveform and channel allocation method.

A Deterministic Algorithm for the Capacity of Finite-State Channels

We propose two modified versions of the classical gradient ascent method to compute the capacity of finite-state channels with Markovian inputs. For the case that the channel mutual information rate is strongly concave in a parameter taking values in a compact convex subset of some Euclidean space, our first algorithm proves to achieve polynomial accuracy in polynomial time and, moreover, for some special families of finite-state channels our algorithm can achieve exponential accuracy in polynomial time under some technical conditions. For the case that the channel mutual information rate may not be strongly concave, our second algorithm proves to be at least locally convergent.

Measuring the Rate of Information Exchange in Point-Process Data With Application to Cardiovascular Variability.

The amount of information exchanged per unit of time between two dynamic processes is an important concept for the analysis of complex systems. Theoretical formulations and data-efficient estimators have been recently introduced for this quantity, known as the mutual information rate (MIR), allowing its continuous-time computation for event-based data sets measured as realizations of coupled point processes. This work presents the implementation of MIR for point process applications in Network Physiology and cardiovascular variability, which typically feature short and noisy experimental time series. We assess the bias of MIR estimated for uncoupled point processes in the frame of surrogate data, and we compensate it by introducing a corrected MIR (cMIR) measure designed to return zero values when the two processes do not exchange information. The method is first tested extensively in synthetic point processes including a physiologically-based model of the heartbeat dynamics and the blood pressure propagation times, where we show the ability of cMIR to compensate the negative bias of MIR and return statistically significant values even for weakly coupled processes. The method is then assessed in real point-process data measured from healthy subjects during different physiological conditions, showing that cMIR between heartbeat and pressure propagation times increases significantly during postural stress, though not during mental stress. These results document that cMIR reflects physiological mechanisms of cardiovascular variability related to the joint neural autonomic modulation of heart rate and arterial compliance.

Free Ride on LDPC Coded Transmission

In this paper, we formulate the problem to cope with the transmission of extra bits over an existing coded transmission link (referred to as coded payload link) without any cost of extra transmission energy or extra bandwidth. This is possible since a gap to the channel capacity typically exists for a practical code. A new concept, termed as <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">accessible capacity</i> , is introduced to specify the maximum rate at which the superposition transmission of extra bits is reliable and has a negligible effect on the performance of the coded payload link. For a binary-input output-symmetric (BIOS) memoryless channel, the accessible capacity can be characterized as the difference between the channel capacity and the mutual information rate of the coded payload link, which can be numerically evaluated for very short payload codes. For a general payload code, we present a simple lower bound on the accessible capacity, given by the channel capacity minus the coding rate of the payload code. We then focus on the scenarios where low-density parity-check (LDPC) codes are implemented for the payload link. We propose to transmit extra bits by random superposition for encoding, and exhaustive search (with the aid of statistical learning) for decoding. We further propose, by establishing an auxiliary channel (called <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">syndrome channel</i> ) induced from “zero-forcing” over the binary field, to transmit extra bits with structured codes such as repetition codes and first-order Reed-Muller (RM) codes. Numerical results show that up to 60 extra bits can be reliably transmitted along with a rate-1/2 LDPC code of length 8064.

Stochastic facilitation in heteroclinic communication channels.

Biological neural systems encode and transmit information as patterns of activity tracing complex trajectories in high-dimensional state spaces, inspiring alternative paradigms of information processing. Heteroclinic networks, naturally emerging in artificial neural systems, are networks of saddles in state space that provide a transparent approach to generate complex trajectories via controlled switches among interconnected saddles. External signals induce specific switching sequences, thus dynamically encoding inputs as trajectories. Recent works have focused either on computational aspects of heteroclinic networks, i.e., Heteroclinic Computing, or their stochastic properties under noise. Yet, how well such systems may transmit information remains an open question. Here, we investigate the information transmission properties of heteroclinic networks, studying them as communication channels. Choosing a tractable but representative system exhibiting a heteroclinic network, we investigate the mutual information rate (MIR) between input signals and the resulting sequences of states as the level of noise varies. Intriguingly, MIR does not decrease monotonically with increasing noise. Intermediate noise levels indeed maximize the information transmission capacity by promoting an increased yet controlled exploration of the underlying network of states. Complementing standard stochastic resonance, these results highlight the constructive effect of stochastic facilitation (i.e., noise-enhanced information transfer) on heteroclinic communication channels and possibly on more general dynamical systems exhibiting complex trajectories in state space.

Mutual information-based recommender system using autoencoder

Nowadays, most of the websites like Amazon, YouTube and Netflix use collaborative filtering methods to recommend various types of items to users. There are two principal categories of collaborative filtering; memory-based and model-based. The memory-based methods use the users’ similarity measures and have several advantages over the model-based techniques, including being easily explained and easy modeling updates with new ratings and items. However, the memory-based methods’ performance reduces when the data is sparse, and unlike the model-based methods, memory-based methods are not scalable. In this paper, we propose a method that exploit the benefits of both similarity-based and model-based approaches. We address both the reliability and the online updating problems based on a novel user-similarity based method. To calculate the new similarity metric we use the predicted user rating vectors in the autoencoder’s output and apply mutual information to the predicted vectors in order to find similar users. We depict a similarity graph according to the mutual information rate, which is calculated for each pair of users. We implement the proposed method on the Netflix movie recommendation dataset. According to our experiments, the proposed approach has a significant advantage over the other methods, such as the standard autoencoder, the matrix factorization, and the similarity-based methods.

An Information-Theoretic Framework to Measure the Dynamic Interaction Between Neural Spike Trains

While understanding the interaction patterns among simultaneous recordings of spike trains from multiple neuronal units is a key topic in neuroscience, existing methods either do not consider the inherent point-process nature of spike trains or are based on parametric assumptions. This work presents an information-theoretic framework for the model-free, continuous-time estimation of both undirected (symmetric) and directed (Granger-causal) interactions between spike trains. The framework computes the mutual information rate (MIR) and the transfer entropy rate (TER) for two point processes X and Y, showing that the MIR between X and Y can be decomposed as the sum of the TER along the directions X → Y and Y → X. We present theoretical expressions and introduce strategies to estimate efficiently the two measures through nearest neighbor statistics. Using simulations of independent and coupled point processes, we show the accuracy of MIR and TER to assess interactions even for weakly coupled and short realizations, and demonstrate the superiority of continuous-time estimation over the standard discrete-time approach. We also apply the MIR and TER to real-world data, specifically, recordings from in-vitro preparations of spontaneously-growing cultures of cortical neurons. Using this dataset, we demonstrate the ability of MIR and TER to describe how the functional networks between recording units emerge over the course of the maturation of the neuronal cultures. the proposed framework provides principled measures to assess undirected and directed spike train interactions with more efficiency and flexibility than previous discrete-time or parametric approaches, opening new perspectives for the analysis of point-process data in neuroscience and many other fields.

A protein structural study based on the centrality analysis of protein sequence feature networks.

In this paper, we use network approaches to analyze the relations between protein sequence features for the top hierarchical classes of CATH and SCOP. We use fundamental connectivity measures such as correlation (CR), normalized mutual information rate (nMIR), and transfer entropy (TE) to analyze the pairwise-relationships between the protein sequence features, and use centrality measures to analyze weighted networks constructed from the relationship matrices. In the centrality analysis, we find both commonalities and differences between the different protein 3D structural classes. Results show that all top hierarchical classes of CATH and SCOP present strong non-deterministic interactions for the composition and arrangement features of Cystine (C), Methionine (M), Tryptophan (W), and also for the arrangement features of Histidine (H). The different protein 3D structural classes present different preferences in terms of their centrality distributions and significant features.