Conditional Mutual Information Research Articles

New improved model for joint segmentation and registration of multi-modality images: application to medical images

Joint segmentation and registration of images is a focused area of research nowadays. Jointly segmenting and registering noisy images and images having weak boundaries/intensity inhomogeneity is a challenging task. In medical image processing, joint segmentation and registration are essential methods that aid in distinguishing structures and aligning images for precise diagnosis and therapy. However, these methods encounter challenges, such as computational complexity and sensitivity to variations in image quality, which may reduce their effectiveness in real-world applications. Another major issue is still attaining effective joint segmentation and registration in the presence of artifacts or anatomical deformations. In this paper, a new nonparametric joint model is proposed for the segmentation and registration of multi-modality images having weak boundaries/noise. For segmentation purposes, the model will be utilizing local binary fitting data term and for registration, it is utilizing conditional mutual information. For regularization of the model, we are using linear curvature. The new proposed model is more efficient to segmenting and registering multi-modality images having intensity inhomogeneity, noise and/or weak boundaries. The proposed model is also tested on the images obtained from the freely available CHOAS dataset and compare the results of the proposed model with the other existing models using statistical measures such as the Jaccard similarity index, relative reduction, Dice similarity coefficient and Hausdorff distance. It can be seen that the proposed model outperforms the other existing models in terms of quantitatively and qualitatively.

Learning task-relevant representations via rewards and real actions for reinforcement learning

The input of visual reinforcement learning often contains redundant information, which will reduce the decision efficiency and decrease the performance of the agent. To address this issue, task-relevant representations of the input are usually learned, where only task-related information is preserved for the decision making. While in the literature, auxiliary tasks that are constructed by using reward signals, optimal policy or by extracting controllable elements in the input are commonly adopted to learn the task-relevant representations, the methods based on reward signals do not work well in sparse reward environments, the effectiveness of methods using optimal policy relies heavily on the optimality degree of the given policy, and the methods by extracting the controllable elements will ignore the uncontrollable task-relevant information in the input. To alleviate these problems and to learn better task-relevant representations, in this paper, we first encourage the encoder to encode controllable parts in the input by maximizing the conditional mutual information between the representations and agent’s real actions. And then as reward signals are directly related to the underlying tasks , they are used to make more task-related information encoded irrespective of whether such information is controllable or not. Finally, a temporal coherence constraint is incorporated into the whole framework to reduce the task-irrelevant information in the representations. Experiments on Distracting DeepMind Control Suite and autonomous driving simulator CARLA show that our proposed approach can achieve better performances than some state-of-the-art (SOTA) baselines, which demonstrates the method’s effectiveness in enhancing the agent’s decision efficiency and overall performances. Code is available at https://github.com/DMU-XMU/Learning-Task-relevant-Representations-via-Rewards-and-Real-Actions-for-Reinforcement-Learning.git.

Neural Joint Entropy Estimation.

Estimating the entropy of a discrete random variable is a fundamental problem in information theory and related fields. This problem has many applications in various domains, including machine learning, statistics, and data compression. Over the years, a variety of estimation schemes have been suggested. However, despite significant progress, most methods still struggle when the sample is small, compared to the variable's alphabet size. In this work, we introduce a practical solution to this problem, which extends the work of McAllester and Statos. The proposed scheme uses the generalization abilities of cross-entropy estimation in deep neural networks (DNNs) to introduce improved entropy estimation accuracy. Furthermore, we introduce a family of estimators for related information-theoretic measures, such as conditional entropy and mutual information (MI). We show that these estimators are strongly consistent and demonstrate their performance in a variety of use cases. First, we consider large alphabet entropy estimation. Then, we extend the scope to MI estimation. Next, we apply the proposed scheme to conditional MI estimation, as we focus on independence testing tasks. Finally, we study a transfer entropy (TE) estimation problem. The proposed estimators demonstrate improved performance compared to existing methods in all of these setups.

Situation-Dependent Causal Influence-Based Cooperative Multi-Agent Reinforcement Learning

Learning to collaborate has witnessed significant progress in multi-agent reinforcement learning (MARL). However, promoting coordination among agents and enhancing exploration capabilities remain challenges. In multi-agent environments, interactions between agents are limited in specific situations. Effective collaboration between agents thus requires a nuanced understanding of when and how agents' actions influence others.To this end, in this paper, we propose a novel MARL algorithm named Situation-Dependent Causal Influence-Based Cooperative Multi-agent Reinforcement Learning (SCIC), which incorporates a novel Intrinsic reward mechanism based on a new cooperation criterion measured by situation-dependent causal influence among agents.Our approach aims to detect inter-agent causal influences in specific situations based on the criterion using causal intervention and conditional mutual information. This effectively assists agents in exploring states that can positively impact other agents, thus promoting cooperation between agents.The resulting update links coordinated exploration and intrinsic reward distribution, which enhance overall collaboration and performance.Experimental results on various MARL benchmarks demonstrate the superiority of our method compared to state-of-the-art approaches.

Diagnosing and Rectifying Fake OOD Invariance: A Restructured Causal Approach

Invariant representation learning (IRL) encourages the prediction from invariant causal features to labels deconfounded from the environments, advancing the technical roadmap of out-of-distribution (OOD) generalization. Despite spotlights around, recent theoretical result verified that some causal features recovered by IRLs merely pretend domain-invariantly in the training environments but fail in unseen domains. The fake invariance severely endangers OOD generalization since the trustful objective can not be diagnosed and existing causal remedies are invalid to rectify. In this paper, we review a IRL family (InvRat) under the Partially and Fully Informative Invariant Feature Structural Causal Models (PIIF SCM /FIIF SCM) respectively, to certify their weaknesses in representing fake invariant features, then, unify their causal diagrams to propose ReStructured SCM (RS-SCM). RS-SCM can ideally rebuild the spurious and the fake invariant features simultaneously. Given this, we further develop an approach based on conditional mutual information with respect to RS-SCM, then rigorously rectify the spurious and fake invariant effects. It can be easily implemented by a small feature selection subnet introduced in the IRL family, which is alternatively optimized to achieve our goal. Experiments verified the superiority of our approach to fight against the fake invariant issue across a variety of OOD generalization benchmarks.

Petz recovery from subsystems in conformal field theory

We probe the multipartite entanglement structure of the vacuum state of a CFT in 1+1 dimensions, using recovery operations that attempt to reconstruct the density matrix in some region from its reduced density matrices on smaller subregions. We use an explicit recovery channel known as the twirled Petz map, and study distance measures such as the fidelity, relative entropy, and trace distance between the original state and the recovered state. One setup we study in detail involves three contiguous intervals A, B and C on a spatial slice, where we can view these quantities as measuring correlations between A and C that are not mediated by the region B that lies between them. We show that each of the distance measures is both UV finite and independent of the operator content of the CFT, and hence depends only on the central charge and the cross-ratio of the intervals. We evaluate these universal quantities numerically using lattice simulations in critical spin chain models, and derive their analytic forms in the limit where A and C are close using the OPE expansion. In the case where A and C are far apart, we find a surprising non-commutativity of the replica trick with the OPE limit. For all values of the cross-ratio, the fidelity is strictly better than a general information-theoretic lower bound in terms of the conditional mutual information. We also compare the mutual information between various subsystems in the original and recovered states, which leads to a more qualitative understanding of the differences between them. Further, we introduce generalizations of the recovery operation to more than three adjacent intervals, for which the fidelity is again universal with respect to the operator content.

Slimming Neural Networks Using Adaptive Connectivity Scores.

In general, deep neural network (DNN) pruning methods fall into two categories: 1) weight-based deterministic constraints and 2) probabilistic frameworks. While each approach has its merits and limitations, there are a set of common practical issues such as trial-and-error to analyze sensitivity and hyper-parameters to prune DNNs, which plague them both. In this work, we propose a new single-shot, fully automated pruning algorithm called slimming neural networks using adaptive connectivity scores (SNACS). Our proposed approach combines a probabilistic pruning framework with constraints on the underlying weight matrices, via a novel connectivity measure, at multiple levels to capitalize on the strengths of both approaches while solving their deficiencies. In SNACS, we propose a fast hash-based estimator of adaptive conditional mutual information (ACMI), that uses a weight-based scaling criterion, to evaluate the connectivity between filters and prune unimportant ones. To automatically determine the limit up to which a layer can be pruned, we propose a set of operating constraints that jointly define the upper pruning percentage limits across all the layers in a deep network. Finally, we define a novel sensitivity criterion for filters that measures the strength of their contributions to the succeeding layer and highlights critical filters that need to be completely protected from pruning. Through our experimental validation, we show that SNACS is faster by over 17× the nearest comparable method and is the state-of-the-art single-shot pruning method across four standard Dataset-DNN pruning benchmarks: CIFAR10-VGG16, CIFAR10-ResNet56, CIFAR10-MobileNetv2, and ILSVRC2012-ResNet50.

Unraveling implicit human behavioral effects on dynamic characteristics of Covid-19 daily infection rates in Taiwan.

We investigate the dynamic characteristics of Covid-19 daily infection rates in Taiwan during its initial surge period, focusing on 79 districts within the seven largest cities. By employing computational techniques, we extract 18 features from each district-specific curve, transforming unstructured data into structured data. Our analysis reveals distinct patterns of asymmetric growth and decline among the curves. Utilizing theoretical information measurements such as conditional entropy and mutual information, we identify major factors of order-1 and order-2 that influence the peak value and curvature at the peak of the curves, crucial features characterizing the infection rates. Additionally, we examine the impact of geographic and socioeconomic factors on the curves by encoding each of the 79 districts with two binary characteristics: North-vs-South and Urban-vs-Suburban. Furthermore, leveraging this data-driven understanding at the district level, we explore the fine-scale behavioral effects on disease spread by examining the similarity among 96 age-group-specific curves within urban districts of Taipei and suburban districts of New Taipei City, which collectively represent a substantial portion of the nation's population. Our findings highlight the implicit influence of human behaviors related to living, traveling, and working on the dynamics of Covid-19 transmission in Taiwan.

Research on bearing ZC feature selection method based on DWCMI

Abstract In light of the issues related to the omission of crucial features and the incorrect selection of redundant features in existing feature selection methods for zero-crossing features, this paper presents a feature selection method based on dynamic weights condition mutual information (DWCMI). In this method, the main factor of the objective function for feature evaluation is conditional mutual information, while also incorporating a complementary evaluation criterion based on conditional mutual information to address the issue of misselecting redundant features. By introducing a dynamic weight coefficient, we can accurately measure the importance of selected features by assessing their dynamic change in mutual information, thereby avoiding any oversight of crucial features. In the process of designing the algorithm, computational efficiency is improved by buffering and reusing previously calculated mutual information. This approach avoids the issue of repeatedly calculating the mutual information. The necessity, effectiveness, and high efficiency of the DWCMI method have been verified through simulation and experimentation.

Matrix product states and the decay of quantum conditional mutual information

A uniform matrix product state defined on a tripartite system of spins, denoted by ABC, is shown to be an approximate quantum Markov chain when the size of subsystem B, denoted |B|, is large enough. The quantum conditional mutual information (QCMI) is investigated and proved to be bounded by a function proportional to exp(−q(|B| − K) + 2K ln |B|), with q and K computable constants. The properties of the bounding function are derived by a new approach, with a corresponding improved value given for its asymptotic decay rate q. We show the improved value of q to be optimal. Numerical investigations of the decay of QCMI are reported for a collection of matrix product states generated by selecting the defining isometry with respect to Haar measure.

Improving Estimates of Land–Atmosphere Coupling Through a Novel Framework of Land Aridity Classification

AbstractThe evapotranspiration (ET) regime is an illustration of the water‐energy interactions at the land surface and is important for regional climate. In this study, we propose a revised framework for land aridity classification based on the ET regime and the conditional mutual information method. The proposed framework effectively captures the influences of short‐term water and energy supply on ET and highlights their potential roles in extreme events. This framework can be a supplement to the traditional classification schemes that rely on long‐term mean climate. Furthermore, we examine the sensitivity of this land aridity framework to different model physics parameterization schemes. The cumulus schemes are found to have the most important impact, followed by the radiation and microphysics schemes, while the planetary boundary layer schemes have the weakest impact. These findings provide valuable insights for the identification of land aridity and its relationship with land–atmosphere interactions.

Holographic entanglement distillation from the surface state correspondence

We study correlations between geometric subfactors living on the Ryu-Takayanagi surface that bounds the entanglement wedge. Using the surface-state correspondence and the bit threads program, we are able to calculate mutual information and conditional mutual information between subfactors. This enables us to count the shared Bell pairs between subfactors, and we propose an entanglement distillation procedure over these subsystems via a SWAP gate protocol. We comment on extending to multipartite entanglement.

Quantum conditional mutual information of W state in non-inertial frames

Quantum conditional mutual information (QCMI) is a versatile information theoretic measure. It is used to find the amount of correlations between two qubits from the perspective of a third qubit. In this work, we characterise the QCMI of tripartite W-states when some of the qubits are in accelerated motion. Here for our investigations we consider a massless fermionic field in the single-mode approximation. We consider all possible situations with respect to the acceleration of the qubits. From our results, we observe that QCMI can either increase or decrease depending on the role of the qubit being accelerated. Finally we discuss the connection between QCMI and correlations by studying the biseparable and separable states.

Feature Selection for Unbalanced Distribution Hybrid Data Based on ${k}$-Nearest Neighborhood Rough Set

Neighborhood rough sets are now widely used to process numerical data. Nevertheless, most of the existing neighborhood rough sets are not able to distinguish class mixture samples well when dealing with classification problems. That is, it cannot effectively classify categories when dealing with data with an unbalanced distribution. Because of this, in this aiticle, we propose a new feature selection method that takes into consideration both heterogeneous data and feature interaction. The proposed model well integrates the ascendancy of δ-neighborhood and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">k</i> -nearest neighbor. Such heterogeneous data can be handled better than existing neighborhood models. We utilize information entropy theories such as mutual information and conditional mutual information, and employ an iterative strategy to define the importance of each feature in decision-making. Furthermore, we design a feature extraction algorithm based on above idea. Experimental results displays that the raised algorithm has superior effect than some existing algorithms, particularly the δ-neighborhood rough set model and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">k</i> -nearest neighborhood rough set model.

Causal explaining guided domain generalization for rotating machinery intelligent fault diagnosis

Post-hoc explaining approaches for deep learning (DL) models has attracted much attention in safety–critical applications such as rotating machinery intelligent fault diagnosis (IFD). However, with the help of the explanation techniques, the models are still fragile to domain shifts caused by varying speeds and loads without help for improving their cross-domain performance. Since humans in the decision-making loop are essential for a reliable diagnostic system to determine the reliability of diagnosis, this paper proposes a causal explaining guided domain generalization (CXDG) method to realize trust worthy IFD with human in the decision loop. Specifically, an explaining model is trained with the conditional mutual information, which is a causal strength metric, and utilized to tell the causal features in the input data as the attributions of the diagnostic model. A translation process of the attributions is proposed to make the explaining process understandable. Furthermore, the aim of this paper is not only explaining but also beyond that the diagnostic model is guided to focus on the causal features to improve its generalization ability in unseen domains. The effectiveness of the method is validated on two experiment datasets. The results show that the proposed method can both explain the attributions of the diagnosis model and be beneficial to the generalization ability.

Comparative Analysis of Machine Learning Algorithms for Heart Disease Prediction

One of the more difficult ailments is heart disease, which affected a large number of people worldwide. Heart illness must be promptly and accurately diagnosed in order to be treated, especially in the field of cardiology. In this work, we suggested a machine learning-based approach for diagnosing cardiac disease that is both effective and accurate. The system was created using classification algorithms, which Standard feature selection algorithms like Relief, Minimal redundancy maximal relevance, Least absolute shrinkage selection operator, and Local learning have been used to omit unnecessary and redundant features. Other feature selection algorithms include Support vector machine, Logistic regression, Artificial neural network, K nearest neighbor, Nave bays, and Decision tree. Additionally, we provided a brand-new, quick conditional mutual information feature selection approach to address.

Active Antinoise Fuzzy Dominance Rough Feature Selection Using Adaptive K-Nearest Neighbors

Feature selection methods with anti-noise performance are effective dimensionality reduction methods for classification tasks with noise. However, there are few studies on robust feature selection methods for monotonic classification tasks. The fuzzy dominance rough set (FDRS) model is a nontrivial knowledge acquisition tool, which is widely used in feature selection of monotonic classification tasks. Nonetheless, this model has been proved in practice to be generally poorly fault-tolerance, and only one noisy sample can cause huge interference in acquiring knowledge. In view of these two issues, this paper firstly designs an adaptive <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$K$</tex-math></inline-formula> -nearest neighbors strategy to calculate the density of samples. The noisy samples are identified according to their densities, and then an active anti-noise fuzzy dominance rough set model is proposed. Then, in the active anti-noise fuzzy dominance rough approximation space, the class-separability is evaluated by the approximation operators of the proposed model, and the feature-redundancy is evaluated by the fuzzy ranking conditional mutual information. On this basis, a feature evaluation index is designed comprehensively considering class-separability and feature-redundancy. Finally, a feature selection algorithm is designed to select the feature subset with the highest classification performance. The experimental results show that the proposed algorithm has better robustness and classification performance.

On the relation between quantum Darwinism and approximate quantum Markovianity

There are strong evidences in the literature that quantum non-Markovianity would hinder the presence of Quantum Darwinism. In this Letter, we study the relation between quantum Darwinism and approximate quantum Markovianity for open quantum systems by exploiting the properties of quantum conditional mutual information. We show that for approximately Markovian quantum processes the conditional mutual information still has the scaling property for Quantum Darwinism. Then two general bounds on the backflow of information are obtained, with which we can show that the presence of Quantum Darwinism restricts the information backflow and the quantum non-Markovianity must be small.

Estimating global identifiability using conditional mutual information in a Bayesian framework

A novel information-theoretic approach is proposed to assess the global practical identifiability of Bayesian statistical models. Based on the concept of conditional mutual information, an estimate of information gained for each model parameter is used to quantify the identifiability with practical considerations. No assumptions are made about the structure of the statistical model or the prior distribution while constructing the estimator. The estimator has the following notable advantages: first, no controlled experiment or data is required to conduct the practical identifiability analysis; second, unlike popular variance-based global sensitivity analysis methods, different forms of uncertainties, such as model-form, parameter, or measurement can be taken into account; third, the identifiability analysis is global, and therefore independent of a realization of the parameters. If an individual parameter has low identifiability, it can belong to an identifiable subset such that parameters within the subset have a functional relationship and thus have a combined effect on the statistical model. The practical identifiability framework is extended to highlight the dependencies between parameter pairs that emerge a posteriori to find identifiable parameter subsets. The applicability of the proposed approach is demonstrated using a linear Gaussian model and a non-linear methane-air reduced kinetics model. It is shown that by examining the information gained for each model parameter along with its dependencies with other parameters, a subset of parameters that can be estimated with high posterior certainty can be found.

Differential Privacy Data Release Scheme Using Microaggregation With Conditional Feature Selection

Differential privacy (DP) has achieved great progress in addressing the user privacy presrvation issues related to data analysis in the Internet of Things (IoT) services and applications. However, the existing DP models tend to overlook the effect of the correlation of data features on the utility of the smart IoT data. To mitigate this gap, we propose a Microaggregation-based Differential Privacy method using Conditional Mutual Information (M-DPCMI) for pre-release data processing and feature selection. With the new method, we leverage the anonymous microaggregation approach to improve data utility while preventing potential sensitive IoT user information leakage. In addition, M-DPCMI is theoretically proved to satisfy the definition of differential privacy and experimentally validated over real datasets. It is shown that the new model achieves better data utility than the state-of-the-art DP methods.