Mutual Information Measure Research Articles

Tuning Pair and Higher‐Order Mutual Information Measures in Oscillator Systems with N$N$‐Dependent Frequencies

Tuning Pair and Higher‐Order Mutual Information Measures in Oscillator Systems with N$N$‐Dependent Frequencies

Network mutual information measures for graph similarity

A wide range of tasks in network analysis, such as clustering network populations or identifying anomalies in temporal graph streams, require a measure of the similarity between two graphs. To provide a meaningful data summary for downstream scientific analyses, the graph similarity measures used for these tasks must be principled, interpretable, and capable of distinguishing meaningful overlapping network structure from statistical noise at different scales of interest. Here we derive a family of graph mutual information measures that satisfy these criteria and are constructed using only fundamental information theoretic principles. Our measures capture the information shared among networks according to different encodings of their structural information, with our mesoscale mutual information measure allowing for network comparison under any specified network coarse-graining. We test our measures in a range of applications on real and synthetic network data, finding that they effectively highlight intuitive aspects of network similarity across scales in a variety of systems.

Cross-Modal Clustering With Deep Correlated Information Bottleneck Method.

Cross-modal clustering (CMC) intends to improve the clustering accuracy (ACC) by exploiting the correlations across modalities. Although recent research has made impressive advances, it remains a challenge to sufficiently capture the correlations across modalities due to the high-dimensional nonlinear characteristics of individual modalities and the conflicts in heterogeneous modalities. In addition, the meaningless modality-private information in each modality might become dominant in the process of correlation mining, which also interferes with the clustering performance. To tackle these challenges, we devise a novel deep correlated information bottleneck (DCIB) method, which aims at exploring the correlation information between multiple modalities while eliminating the modality-private information in each modality in an end-to-end manner. Specifically, DCIB treats the CMC task as a two-stage data compression procedure, in which the modality-private information in each modality is eliminated under the guidance of the shared representation of multiple modalities. Meanwhile, the correlations between multiple modalities are preserved from the aspects of feature distributions and clustering assignments simultaneously. Finally, the objective of DCIB is formulated as an objective function based on a mutual information measurement, in which a variational optimization approach is proposed to ensure its convergence. Experimental results on four cross-modal datasets validate the superiority of the DCIB. Code is released at https://github.com/Xiaoqiang-Yan/DCIB.

Characterizing the nonmonotonic behavior of mutual information along biochemical reaction cascades.

Cells sense environmental signals and transmit information intracellularly through changes in the abundance of molecular components. Such molecular abundances can be measured in single cells and exhibit significant heterogeneity in clonal populations even in identical environments. Experimentally observed joint probability distributions can then be used to quantify the covariability and mutual information between molecular abundances along signaling cascades. However, because stationary state abundances along stochastic biochemical reaction cascades are not conditionally independent, their mutual information is not constrained by the data-processing inequality. Here, we report the conditions under which the mutual information between stationary state abundances increases along a cascade of biochemical reactions. This nonmonotonic behavior can be intuitively understood in terms of noise propagation and time-averaging stochastic fluctuations that are short-lived compared to an extrinsic signal. Our results reemphasize that mutual information measurements of stationary state distributions of cellular components may be of limited utility for characterizing cellular signaling processes because they do not measure information transfer.

Mutual information sums and relations with interaction energies in N-particle quantum systems.

The sums of position- and momentum-space mutual information measures are used to examine the pairwise and higher-order statistical correlation in the ground states of N-particle coupled oscillators. Analytical expressions for these measures are shown to be related to the logarithmic interaction energies of these states, plus those of mirror states where the intensities of the one- and two-body potentials are interchanged, and the nature of the attractive or repulsive interaction is opposite to that in the original state. The measures separate the contributions from the interactionsand those from the effective interactions due to marginalization into different terms. The pairwise mutual information sum is linearly related to the Shannon entropy sum in two particle systems, while the total correlation sum exhibits a similar relationship in three particle ones. In the latter instance, the interaction information sum can be related to entropy differences. This illustrates how entropy sums are connected to correlation measure sums in these systems. All measures approach zero with large N, when the magnitudes of the one- and two-body potentials are fixed. The pair mutual information and total correlation sums decay monotonically with N in the presence of an attractive potential and monotonically increase with a repulsive potential. On the other hand, the interaction information sum exhibits a minimum at small N, with an attractive potential. This is a consequence of the higher-order correlations governing behavior at smaller N while the pairwise ones dominate at larger N. Results are presented when the magnitude of the one-body potential is set to N.

Copula-Based Mutual Information Measures and Mutual Entropy: A Brief Survey

Copula-Based Mutual Information Measures and Mutual Entropy: A Brief Survey

Consistency of the central bank of the Republic of Türkiye’s policy stance with its discourses: a fuzzy logic analysis

ABSTRACT The purpose of the study is to use fuzzy logic methods to assess the stance of the Central Bank of the Republic of Türkiye (CBRT) towards monetary policy. These methods enable us to examine the coherence and consistency between the CBRT’s discourses and policy stance with its declarations covering the period 2011:09 - 2019:12. Key variables are included that effectively explain the proxy for the central bank’s policy stance by mutual information measure for variable selection. The policy stance of the CBRT is classified by fuzzy C-means clustering within the scope of the rule-based fuzzy logic model. This analysis allows us to group the policy stance as loose, moderate, and tight. Within the analysis timeframe, the CBRT maintained a rhetoric of consistent tightness in statements regarding its monetary policy stance. However, the only overlap occurred between June 2018 and July 2019, when the policy stance declared by the CBRT and the model results aligned. In the remaining periods, the policy stance declared by the CBRT does not correspond with the model outputs.

Hybrid Filtering and Probabilistic Techniques for Privacy-Preserving Community Detection in OSNs

Online Social Networks (OSNs) face the major challenge of protecting participant's privacy, due to the high dimensionality and volume of the data. In real-time social networks, where hundreds of personal details of people are shared every day, there remains a significant threat to privacy. Privacy preservation is challenging for a community detection problem due to the high computational complexity and memory requirements, especially in larger real-world OSN graphs. Although weighted nodes provide better results, as they allow capturing the frequencies of the values, the privacy preservation of sensitive attributes such as specific profiles becomes harder compared to these models. This problem can result in queries and subsequent learnings from social network profiles of specific individuals, which may be of personal, political or otherwise concern. Online social networks (OSNs) grapple with significant privacy challenges due to the extensive dimensions and vast quantities of data involved. This research fills the void in current privacy-preserving community detection methodologies, which face problems in computational complexity and memory usage in large-scale OSNs. The proposed framework seeks to bolster privacy preservation through a comprehensive multi-step process. Data filtering uses a blended data filter system to remove outliers and irrelevant data, thus enhancing the quality of the input data. Density-Oriented Clustering phase employs a density-oriented clustering model to identify communities, with each cluster representing a distinct community. Privacy Preservation component introduces a new privacy preservation technique for sensitive OSN attributes, surpassing existing k-anonymization methods. The developed density-based social network community detection model and its novel privacy-preserving scheme are evaluated using the datasets Yelp, Football, Zachary and Dolphin from the SNAP dataset. Experimental results on these datasets embed a comprehensive evaluation based on the order of each node and the graph-based networks, where each node is laden with weights as proximity values, indicating the semantic proximity between communities and individuals. The proposed framework employs the normalized mutual information (NMI), modularity (Q), Rand index and runtime measurements to demonstrate widespread advantages in the multi-dimensional functional space, including greater accuracy, cluster compatibility, and computational tractability over the existing prominent traditional models for OSNs.

Deep learning insights into non-universality in the halo mass function

ABSTRACT The abundance of dark matter haloes is a key cosmological probe in forthcoming galaxy surveys. The theoretical understanding of the halo mass function (HMF) is limited by our incomplete knowledge of the origin of non-universality and its cosmological parameter dependence. We present a deep-learning model which compresses the linear matter power spectrum into three independent factors which are necessary and sufficient to describe the $z=0$ HMF from the state-of-the-art Aemulus emulator to sub-per cent accuracy in a wCDM$+N_\mathrm{eff}$ parameter space. Additional information about growth history does not improve the accuracy of HMF predictions if the matter power spectrum is already provided as input, because required aspects of the former can be inferred from the latter. The three factors carry information about the universal and non-universal aspects of the HMF, which we interrogate via the information-theoretic measure of mutual information. We find that non-universality is captured by recent growth history after matter-dark-energy equality and $N_{\rm eff}$ for $M\sim 10^{13} \, \mathrm{M_\odot }\, h^{-1}$ haloes, and by $\Omega _{\rm m}$ for $M\sim 10^{15} \, \mathrm{M_\odot }\, h^{-1}$. The compact representation learnt by our model can inform the design of emulator training sets to achieve high emulator accuracy with fewer simulations.

Analyzing the Quality of Distorted Images by the Normalized Mutual Information Measure

This research explores how different types of distorting algorithms impact the Full-Reference image quality assessment, particularly when subjective quality evaluations are incorporated. We draw upon the TID2013 database, which contains 3000 images distorted by 24 distinct algorithms, in conjunction with Mean Opinion Scores (MOS) for quality ratings. We compare the results of Normalized Mutual Information (NMI) for image quality score with W2, based on Weibull distribution, the common PSNR similarity measure and MOS. We advocate for integrating of NMI into the repertoire of image quality assessment metrics.

Measurement of information content of Perovskite solar cell’s synthesis descriptors related to performance parameters

AbstractPerovskite solar cells (PSC) are formed by different layers composed of thin films of various materials, in which the properties of every thin layer affect the performance of the cell. The identification of those most relevant properties (or descriptors) has a significant impact on the optimization and cost reduction of the Perovskite solar cell. This relevance is typically evaluated by adjusting a model using subsets of features, but in the present work, we propose to use the mutual information measure to quantify the statistical association between input descriptors and Perovskite solar cell performance parameters (Voc, Jsc, FF, PCE). As a result, it is found that ion X is the factor that most impacts the performance of the solar cell. On the other hand, variables such as band gap, Perovskite layer thickness, and A and B ions are also important. In this work, we identify some of the most important factors affecting Perovskite solar cells’ performance, and it could help to improve the efficiency of Perovskite solar cells. In addition, this proposed method could also be applied to other types of functional coatings, thin films, and surfaces.

Strong Asymptotic Composition Theorems for Mutual Information Measures

We characterize the growth of the Sibson and Arimoto mutual informations and α-maximal leakage, of any order that is at least unity, between a random variable and a growing set of noisy, conditionally independent and identically-distributed observations of the random variable. Each of these measures increases exponentially fast to a limit that is order- and measure-dependent, with an exponent that is order- and measure-independent.

A computational linguistic methodology for assessing semiotic structure in prehistoric art and the meaning of southern Scandinavian Mesolithic ornamentation

Non-figurative prehistoric art is comparatively common yet challenging to decode and hence to understand. At the heart of the difficulty of assessing the presence of semiotic structure in prehistoric art is a lack of appropriate, replicable, and case-transferable methodologies. We here propose a novel approach derived from computational linguistics, in which k-skip-n-gram (skipgram) models and associated pointwise mutual information (PMI) measures are customised to the analysis of prehistoric art. In applying this methodology to a large corpus of portable art from the South Scandinavian Mesolithic, we demonstrate how the mutual relationship between individual motifs can be established. In the case of Mesolithic portable art, our analysis strongly suggests that there is no evident semiotic structure – this was likely not a form of proto-writing – but salient changes of motif occurrence over time are detectable. These changes are sensitive to changes in population density, structure, and connectedness, and may relate to increased territoriality in the Late Mesolithic. The method presented here is readily case-transferable and renders possible further linguistic and semiotic analyses of prehistoric art.

Private outsourced model predictive control via secure multi-party computation

Protecting data privacy is essential in developing practical cloud-based control systems to benefit from remote and distributed computation while avoiding the risks of disclosing sensitive information to potentially untrusted parties. This paper proposes a privacy-preserving framework to outsource the computation of model predictive control law to several untrusted cloud-based servers. A secret sharing scheme is employed to maintain the privacy of the system’s data in all the stages of the control loop in a secure multi-party computation environment. In this regard, a privacy-preserving algorithm is derived to securely implement a projected gradient method to solve the underlying optimization problem, without revealing private data to external eavesdroppers and curious cloud-based servers. The proposed method is solely based on secret sharing, and all computations can be performed securely by cloud servers without the need to designate target nodes or engage the system’s actuator in computing the intermediate steps. Therefore, the proposed method is less computationally demanding than the existing results based on homomorphic encryption and is applicable to systems with limited computing resources. Information-theoretic privacy assessments based on mutual information measures are provided. The efficacy of the proposed method is investigated on the cruise control problem of a freight train system.

The causality measure of partial mutual information from mixed embedding (PMIME) revisited.

The measure of partial mutual information from mixed embedding (PMIME) is an information theory-based measure to accurately identify the direct and directional coupling, termed Granger causality or simply causality, between the observed variables or subsystems of a high-dimensional dynamical and complex system, without any a priori assumptions about the nature of the coupling relationship. In its core, it is a forward selection procedure that aims to iteratively identify the lag-dependence structure of a given observed variable (response) to all the other observed variables (candidate drivers). This model-free approach is capable of detecting nonlinear interactions, abundantly present in real-world complex systems, and it was shown to perform well on multivariate time series of moderately high dimension. However, the PMIME presents some inefficiencies in its performance mainly when applied on strongly stochastic (linear or nonlinear) systems as it may falsely detect non-existent relationships. Moreover, and by construction, the measure cannot extract purely synergetic relationships present in a system. In the current work, the issue of false detections is addressed by introducing an improved resampling significance test and a procedure of rechecking the identified drivers (backward revision). Regarding the inability to detect synergetic relationships, the PMIME is further enhanced by checking pairs as candidate drivers for the response variable after having considered all drivers individually. The effects of these modifications are investigated in a systematic simulation study on properly designed systems involving strong stochasticity, regressor terms with synergetic effects, and a system dimension ranging from 3 to 30. The overall results of the simulations indicate that these modifications indeed improve the performance of PMIME and alleviate to a significant degree the issues of the original algorithm. Guidelines for balancing between accuracy and computational efficiency are also given, particularly relevant for real-world applications. Finally, the measure performance is investigated in the study of futures of various government bonds and stock market indices in the period around COVID-19 pandemic.

Multitask Image Clustering via Deep Information Bottleneck.

Multitask image clustering approaches intend to improve the model accuracy on each task by exploring the relationships of multiple related image clustering tasks. However, most existing multitask clustering (MTC) approaches isolate the representation abstraction from the downstream clustering procedure, which makes the MTC models unable to perform unified optimization. In addition, the existing MTC relies on exploring the relevant information of multiple related tasks to discover their latent correlations while ignoring the irrelevant information between partially related tasks, which may also degrade the clustering performance. To tackle these issues, a multitask image clustering method named deep multitask information bottleneck (DMTIB) is devised, which aims at conducting multiple related image clustering by maximizing the relevant information of multiple tasks while minimizing the irrelevant information among them. Specifically, DMTIB consists of a main-net and multiple subnets to characterize the relationships across tasks and the correlations hidden in a single clustering task. Then, an information maximin discriminator is devised to maximize the mutual information (MI) measurement of positive samples and minimize the MI of negative ones, in which the positive and negative sample pairs are constructed by a high-confidence pseudo-graph. Finally, a unified loss function is devised for the optimization of task relatedness discovery and MTC simultaneously. Empirical comparisons on several benchmark datasets, NUS-WIDE, Pascal VOC, Caltech-256, CIFAR-100, and COCO, show that our DMTIB approach outperforms more than 20 single-task clustering and MTC approaches.

Continuous Patient-Independent Estimation of Respiratory Rate and Blood Pressure Using Robust Spectro-Temporal Features Derived From Photoplethysmogram Only.

Objective: A patient-independent approach for continuous estimation of vital signs using robust spectro-temporal features derived from only photoplethysmogram (PPG) signal. Methods: In the pre-processing stage, we remove baseline shifts and artifacts of the PPG signal using Incremental Merge Segmentation with adaptive thresholding. From the cleaned PPG, we extract multiple parameters independent of individual patient PPG morphology for both Respiration Rate (RR) and Blood Pressure (BP). In addition, we derived a set of novel spectral and statistical features strongly correlated to BP. We proposed robust correlation-based feature selection methods for accurate RR estimates. For fewer computations and accurate measurements of BP, the most significant features are selected using correlation and mutual information measures in the feature engineering part. Finally, RR and BP are estimated using breath counting and a neural network regression model, respectively. Results: The proposed approach outperforms the current state-of-the-art in both RR and BP. The RR algorithm results in mean absolute errors (median, 25th-75th percentiles) of 0.4 (0.1-0.7) for CapnoBase dataset and 0.5(0.3-2.8) for BIDMC dataset without discarding any data window. Similarly, BP approach has been validated on a large dataset derived from MIMIC-II ([Formula: see text]1700 records) which has errors (mean absolute, standard deviation) of 5.0(6.3) and 3.0(4.0) for systolic and diastolic BP, respectively. The results meet the American Association for the Advancement of Medical Instrumentation (AAMI) and British Hypertension Society (BHS) Class A criteria. Conclusion: By using robust features and feature selection methods, we alleviated patient dependency to have reliable estimates of vitals.

Sustainable Topic Modeling for Legal Moroccan Arabic Language: A Challenging Study on BERTopic Technique

Topic Modeling approaches face difficulties in processing legal texts because of their unique characteristics, such as the length of the texts and the specialized terminology used within them. The process of topic modeling involves finding a text's semantic structure. This way, specific approaches are needed. When the legal documents are presented has a lot to do with what topics are important. This paper aims to explain and evaluate BERTopic's application to topic modeling in legal documents. In this research, we experiment with BERTopic by utilizing its several pre-trained Arabic language models as embeddings. Performance evaluation employs the Normalized Pointwise Mutual Information (NPMI) measure. Notably, in comparison to multilingual pre-trained models, our findings reveal that BERTopic using Arabic monolingual pre-trained models exhibits superior performance, offering insights into sustainable and efficient topic modeling for legal documents.

A mutual information measure of phase-amplitude coupling using gamma generalized linear models.

Cross frequency coupling (CFC) between electrophysiological signals in the brain is a long-studied phenomenon and its abnormalities have been observed in conditions such as Parkinson's disease and epilepsy. More recently, CFC has been observed in stomach-brain electrophysiologic studies and thus becomes an enticing possible target for diseases involving aberrations of the gut-brain axis. However, current methods of detecting coupling, specifically phase-amplitude coupling (PAC), do not attempt to capture the phase and amplitude statistical relationships. In this paper, we first demonstrate a method of modeling these joint statistics with a flexible parametric approach, where we model the conditional distribution of amplitude given phase using a gamma distributed generalized linear model (GLM) with a Fourier basis of regressors. We perform model selection with minimum description length (MDL) principle, demonstrate a method for assessing goodness-of-fit (GOF), and showcase the efficacy of this approach in multiple electroencephalography (EEG) datasets. Secondly, we showcase how we can utilize the mutual information, which operates on the joint distribution, as a canonical measure of coupling, as it is non-zero and non-negative if and only if the phase and amplitude are not statistically independent. In addition, we build off of previous work by Martinez-Cancino et al., and Voytek et al., and show that the information density, evaluated using our method along the given sample path, is a promising measure of time-resolved PAC. Using synthetically generated gut-brain coupled signals, we demonstrate that our method outperforms the existing gold-standard methods for detectable low-levels of phase-amplitude coupling through receiver operating characteristic (ROC) curve analysis. To validate our method, we test on invasive EEG recordings by generating comodulograms, and compare our method to the gold standard PAC measure, Modulation Index, demonstrating comparable performance in exploratory analysis. Furthermore, to showcase its use in joint gut-brain electrophysiology data, we generate topoplots of simultaneous high-density EEG and electrgastrography recordings and reproduce seminal work by Richter et al. that demonstrated the existence of gut-brain PAC. Using simulated data, we validate our method for different types of time-varying coupling and then demonstrate its performance to track time-varying PAC in sleep spindle EEG and mismatch negativity (MMN) datasets. Our new measure of PAC using Gamma GLMs and mutual information demonstrates a promising new way to compute PAC values using the full joint distribution on amplitude and phase. Our measure outperforms the most common existing measures of PAC, and show promising results in identifying time varying PAC in electrophysiological datasets. In addition, we provide for using our method with multiple comparisons and show that our measure potentially has more statistical power in electrophysiologic recordings using simultaneous gut-brain datasets.

Application of independent component analysis-based dimensionality reduction technique in effective information extraction of high-dimensional high-frequency data

Abstract In this paper, after analyzing the existing independent component analysis algorithms, the high-dimensional high-frequency data are preprocessed by whitening, and the objective functions of crag, negative entropy, approximate negative entropy, and mutual information are set. The independent component analysis model is designed to separate the independence between signals by maximizing non-Gaussianity, estimating great likelihood, and minimizing mutual information measures. In order to verify that the dimensionality reduction technique based on independent component analysis can effectively extract high-dimensional, high-frequency data information, it is applied to the industry’s closing index data and financial market data. The results show that the stock price synthesized using the six major independent factors is almost the same as the original stock price trend of DG, and the difference in stock price evaluation is within 5, indicating that the six major independent factors play a decisive role in the stock price trend. The study shows that the dimensionality reduction technique based on independent component analysis can analyze the volatility of stock prices and obtain more effective information from high-dimensional, high-frequency data.