Mutual Information Optimization Research Articles

Variational Distillation for Multi-View Learning.

Information Bottleneck (IB) provides an information-theoretic principle for multi-view learning by revealing the various components contained in each viewpoint. This highlights the necessity to capture their distinct roles to achieve view-invariance and predictive representations but remains under-explored due to the technical intractability of modeling and organizing innumerable mutual information (MI) terms. Recent studies show that sufficiency and consistency play such key roles in multi-view representation learning, and could be preserved via a variational distillation framework. But when it generalizes to arbitrary viewpoints, such strategy fails as the mutual information terms of consistency become complicated. This paper presents Multi-View Variational Distillation (MV 2D), tackling the above limitations for generalized multi-view learning. Uniquely, MV 2D can recognize useful consistent information and prioritize diverse components by their generalization ability. This guides an analytical and scalable solution to achieving both sufficiency and consistency. Additionally, by rigorously reformulating the IB objective, MV 2D tackles the difficulties in MI optimization and fully realizes the theoretical advantages of the information bottleneck principle. We extensively evaluate our model on diverse tasks to verify its effectiveness, where the considerable gains provide key insights into achieving generalized multi-view representations under a rigorous information-theoretic principle.

Disentangled Self-Attention with Auto-Regressive Contrastive Learning for Neural Group Recommendation

Disentangled Self-Attention with Auto-Regressive Contrastive Learning for Neural Group Recommendation

A Twist for Graph Classification: Optimizing Causal Information Flow in Graph Neural Networks

Graph neural networks (GNNs) have achieved state-of-the-art results on many graph representation learning tasks by exploiting statistical correlations. However, numerous observations have shown that such correlations may not reflect the true causal mechanisms underlying the data and thus may hamper the ability of the model to generalize beyond the observed distribution. To address this problem, we propose an Information-based Causal Learning (ICL) framework that combines information theory and causality to analyze and improve graph representation learning to transform information relevance to causal dependence. Specifically, we first introduce a multi-objective mutual information optimization objective derived from information-theoretic analysis and causal learning principles to simultaneously extract invariant and interpretable causal information and reduce reliance on non-causal information in correlations. To optimize this multi-objective objective, we enable a causal disentanglement layer that effectively decouples the causal and non-causal information in the graph representations. Moreover, due to the intractability of mutual information estimation, we derive variational bounds that enable us to transform the above objective into a tractable loss function. To balance the multiple information objectives and avoid optimization conflicts, we leverage multi-objective gradient descent to achieve a stable and efficient transformation from informational correlation to causal dependency. Our approach provides important insights into modulating the information flow in GNNs to enhance their reliability and generalization. Extensive experiments demonstrate that our approach significantly improves the robustness and interpretability of GNNs across different distribution shifts. Visual analysis demonstrates how our method converts informative dependencies in representations into causal dependencies.

Multiagent Reinforcement Learning With Graphical Mutual Information Maximization.

Communication learning is an important research direction in the multiagent reinforcement learning (MARL) domain. Graph neural networks (GNNs) can aggregate the information of neighbor nodes for representation learning. In recent years, several MARL methods leverage GNN to model information interactions between agents to coordinate actions and complete cooperative tasks. However, simply aggregating the information of neighboring agents through GNNs may not extract enough useful information, and the topological relationship information is ignored. To tackle this difficulty, we investigate how to efficiently extract and utilize the rich information of neighbor agents as much as possible in the graph structure, so as to obtain high-quality expressive feature representation to complete the cooperation task. To this end, we present a novel GNN-based MARL method with graphical mutual information (MI) maximization to maximize the correlation between input feature information of neighbor agents and output high-level hidden feature representations. The proposed method extends the traditional idea of MI optimization from graph domain to multiagent system, in which the MI is measured from two aspects: agent features information and agent topological relationships. The proposed method is agnostic to specific MARL methods and can be flexibly integrated with various value function decomposition methods. Considerable experiments on various benchmarks demonstrate that the performance of our proposed method is superior to the existing MARL methods.

RIS-Assisted MIMO OFDM Dual-Function Radar-Communication based on Mutual Information Optimization

RIS-Assisted MIMO OFDM Dual-Function Radar-Communication based on Mutual Information Optimization

Optimization of cross-cultural communication model for ethnic minorities based on self-similarity and comparative learning

Abstract The super-resolution algorithm of self-similarity is utilized in this paper to construct an image training set based on the multi-scale self-similarity of images and reconstruct the super-resolution of images. The visual question-and-answer method of contrast learning ensures full coverage of key targets, which makes the optimization of mutual information more reliable and stable to construct a cross-cultural communication model for ethnic minorities. The results show that compared with the cross-modal audio-video instance discrimination model, the accuracy of TOP1 at the visual clip level is 3.04% higher, and the accuracy of TOP5 at the video level is 2.62% higher for the model designed in this paper. This paper's design model can enhance the ability of cross-cultural communication among ethnic minorities, as indicated.

Multiple-Sensor Detection System Design for Tea Identification Based on Mutual Information Array Optimization Scheme

In this paper, a multi-metal oxide sensor (MOS) detection system is developed for the identification of different grades of tea leaves. To cope with the interference brought by redundant information and further improve the detection capability of the system and reduce the number of sensors, sensor array optimization algorithms are further implemented. The main idea of sensor array optimization is to select sensors according to evaluation index which ranks the importance of sensors. This paper optimizes the sensor array based on information theory. Specifically, a weighted modified mutual information optimization algorithm that simultaneously considers relevancy, redundancy, and complementarity is developed. The experimental results show that the weighted modified mutual information optimization method can obtain the best system performance with the minimum number of sensors.

An analysis of detectable leakage rate for oil pipelines based on acoustic wave method

Oil pipeline leakage will not only cause economic losses, but also pollute the environment, so the leakage detection of pipelines is very important. The acoustic wave method is widely used in pipeline leak detection, and the leak acoustic signal collected by the acoustic wave sensor often contains a lot of noise, which makes it impossible to accurately determine the inflection point of the signal curve and reduces the accuracy of pipeline leak detection. This paper proposes a denoising algorithm based on mutual information optimization complete ensemble empirical mode decomposition with adaptive noise combined with cross-spectral analysis. Compared with other methods, this method can accurately select the effective intrinsic modal function for signal reconstruction, the denoising effect is more obvious, and the original information is preserved to a greater extent. Acoustic waves are attenuated during the propagation process, and will be affected by factors such as pipe connection ports and elbows, making it impossible to accurately determine the amplitude of acoustic waves around the pipeline. According to the propagation characteristics of acoustic waves and various factors that affect the propagation of acoustic waves, this paper establishes a model for calculating the amplitude of acoustic waves, which can accurately determine the amplitude of acoustic waves everywhere in the pipeline. Finally, according to the model, the relationship between pipeline characteristics and detectable leakage rate is analysed. Field experiments show that the proposed model is accurate and the denoising algorithm is efficient. The minimum detectable leakage rate of the oil pipeline can reach 0.43% when the acoustic wave method is used for leak detection.

Cooperative Design for MIMO Radar- Communication Spectral Sharing System Based on Mutual Information Optimization

This paper considers a novel cooperative design for radar-communication spectral sharing system with multiple-input multiple-output (MIMO) structure based on mutual information optimization. We present a new spectral sharing framework aiming at maximizing the radar mutual information. Specifically, the transmit sequences including radar waveform and communication codebook are designed simultaneously based on radar and communication system mutual information optimization. Subsequently, an iterative procedure is devised to maximize the radar mutual information under the constraints including the transmit power, the constant modulus requirement for radar waveform, and the communication system mutual information performance requirement constraint. The proposed approach exploits the alternating optimization based on the Minorization-Maximization (MM) framework, Lagrange algorithm, and the alternating direction method of multipliers (ADMM) techniques to decompose the original design into two simpler subproblems with a closed-form solution. Simulation results are presented to demonstrate the advantage of the proposed schemes.

Fast computation of mutual information in the frequency domain with applications to global multimodal image alignment

Multimodal image alignment is the process of finding spatial correspondences between images formed by different imaging techniques or under different conditions, to facilitate heterogeneous data fusion and correlative analysis. The information-theoretic concept of mutual information (MI) is widely used as a similarity measure to guide multimodal alignment processes, where most works have focused on local maximization of MI that typically works well only for small displacements; this points to a need for global maximization of MI, which has previously been computationally infeasible due to the high run-time complexity of existing algorithms. We propose an efficient algorithm for computing MI for all discrete displacements (formalized as the cross-mutual information function (CMIF)), which is based on cross-correlation computed in the frequency domain. We show that the algorithm is equivalent to a direct method while asymptotically superior in terms of run-time. Furthermore, we propose a method for multimodal image alignment for transformation models with few degrees of freedom (e.g. rigid) based on the proposed CMIF-algorithm. We evaluate the efficacy of the proposed method on three distinct benchmark datasets, of aerial images, cytological images, and histological images, and we observe excellent success-rates (in recovering known rigid transformations), overall outperforming alternative methods, including local optimization of MI as well as several recent deep learning-based approaches. We also evaluate the run-times of a GPU implementation of the proposed algorithm and observe speed-ups from 100 to more than 10,000 times for realistic image sizes compared to a GPU implementation of a direct method. Code is shared as open-source at \url{github.com/MIDA-group/globalign}.

Intensity-Mosaic: automatic panorama mosaicking of disordered images with insufficient features.

Purpose: Handling low-quality and few-feature medical images is a challenging task in automatic panorama mosaicking. Current mosaicking methods for disordered input images are based on feature point matching, whereas in this case intensity-based registration achieves better performance than feature-point registration methods. We propose a mosaicking method that enables the use of mutual information (MI) registration for mosaicking randomly ordered input images with insufficient features. Approach: Dimensionality reduction is used to map disordered input images into a low dimensional space. Based on the low dimensional representation, the image global correspondence can be recognized efficiently. For adjacent image pairs, we optimize the MI metric for registration. The panorama is then created after image blending. We demonstrate our method on relatively lower-cost handheld devices that acquire images from the retina in vivo, kidney ex vivo, and bladder phantom, all of which contain sparse features. Results: Our method is compared with three baselines: AutoStitch, "dimension reduction + SIFT," and "MI-Only." Our method compared to the first two feature-point based methods exhibits 1.25 (ex vivo microscope dataset) to two times (in vivo retina dataset) rate of mosaic completion, and MI-Only has the lowest complete rate among three datasets. When comparing the subsequent complete mosaics, our target registration errors can be 2.2 and 3.8 times reduced when using the microscopy and bladder phantom datasets. Conclusions: Using dimensional reduction increases the success rate of detecting adjacent images, which makes MI-based registration feasible and narrows the search range of MI optimization. To the best of our knowledge, this is the first mosaicking method that allows automatic stitching of disordered images with intensity-based alignment, which provides more robust and accurate results when there are insufficient features for classic mosaicking methods.

Radar and Communication Co-Existence Design Based on Mutual Information Optimization

In this brief, we proposed and analyzed a novel algorithmic framework for radar and communication co-existence design based on mutual information (MI) optimization. The objects of interest in design are the radar transmit code, receive filter, and the communication system code-book, whereby we state the design problem as a constrained maximization of the communication MI subject to a number of constraints guaranteeing both the communication and radar performance, through a similarity constraint with some standard waveform and the effective power of radar interference under the power limited conditions. The proposed approach is based on iterative alternating optimization of three suitably designed subproblems, whose solutions are derived and analyzed in closed form, and the convergence is analytically proved. Finally, we assess the effectiveness of the proposed approaches by numerical results.

Fuzzy Clustering Based Image Denoising and Improved Support Vector Machine (ISVM) Based Nearest Target for Retina Images

A developing automated retinal disease diagnostic system based on image analysis has now demonstrated the ability in clinical research. Though, the accuracy of these systems has been negotiated repeatedly, generally due to the basic effort in perceiving the abnormal structures as well as due to deficits in the image gaining that affects image quality. Use the fuzzy clustering; the noises contained in the samples are omitted from the above. Unless the noises will be taken away from the samples instead dimension reduction initializes the optimization of Mutual Information (MI) as just a coarse localization process that narrows the domain of optimization and tries to avoid local optimization. Furthermore, the suggested work closer to the retina picture being done using the Improved Support Vector Machine (ISVM) system used in the area-based registration, offering a reliable approach. It is the first matching template algorithm for retina images with tiny template images of unconstrained retinal areas to the best understanding.

RetinaMatch: Efficient Template Matching of Retina Images for Teleophthalmology.

Retinal template matching and registration is an important challenge in teleophthalmology with low-cost imaging devices. However, the images from such devices generally have a small field of view (FOV) and image quality degradations, making matching difficult. In this paper, we develop an efficient and accurate retinal matching technique that combines dimension reduction and mutual information (MI), called RetinaMatch. The dimension reduction initializes the MI optimization as a coarse localization process, which narrows the optimization domain and avoids local optima. The effectiveness of RetinaMatch is demonstrated on the open fundus image database STARE with simulated reduced FOV and anticipated degradations, and on retinal images acquired by adapter-based optics attached to a smartphone. RetinaMatch achieves a success rate over 94% on human retinal images with the matched target registration errors below 2 pixels on average, excluding the observer variability, outperforming standard template matching solutions. In the application of measuring vessel diameter repeatedly, single pixel errors are expected. In addition, our method can be used in the process of image mosaicking with area-based registration, providing a robust approach when feature-based methods fail. To the best of our knowledge, this is the first template matching algorithm for retina images with small template images from unconstrained retinal areas. In the context of the emerging mixed reality market, we envision automated retinal image matching and registration methods as transformative for advanced teleophthalmology and long-term retinal monitoring.

Large-Deformation Image Registration of CT-TEE for Surgical Navigation of Congenital Heart Disease.

The surgical treatment of congenital heart disease requires navigational assistance with transesophageal echocardiography (TEE); however, TEE images are often difficult to interpret and provide very limited anatomical information. Registering preoperative CT images to intraoperative TEE images provides surgeons with richer and more useful anatomical information. Yet, CT and TEE images differ substantially in terms of scale and geometry. In the present research, we propose a novel method for the registration of CT and TEE images for navigation during surgical repair of large defects in patients with congenital heart disease. Valve data was used for the coarse registration to determine the basic location. This was followed by the use of an enhanced probability model map to overcome gray-level differences between the two imaging modalities. Finally, the rapid optimization of mutual information was achieved by migrating parameters. This method was tested on a dataset of 240 images from 12 infant, children (≤ 3 years old), and adult patients with congenital heart disease. Compared to the “bronze standard” registration, the proposed method was more accurate with an average Dice coefficient of 0.91 and an average root mean square of target registration error of 1.2655 mm.

Golden Angle Modulation

Quadrature amplitude modulation (QAM), with its uniform distribution, exhibits an asymptotic shaping-loss of ${\pi \mathrm {e}/6}$ (≈ 1.53 dB) with increasing signal-to-noise-ratio compared to the additive white Gaussian noise Shannon capacity. With inspiration gained from special (leaf, flower petal, and seed) packing arrangements (spiral phyllotaxis) found among plants, a novel, shape-versatile, circular symmetric, modulation scheme, the golden angle modulation (GAM) is introduced. Disc-shaped, and complex Gaussian approximating bell-shaped, GAM-signal constellations are considered. For bell-GAM, a high-rate approximation, and a mutual information optimization formulation, are developed. Bell-GAM overcomes the asymptotic shaping-loss seen in QAM, and offers Shannon capacity approaching performance. Transmitter resource limited links, such as space probe-to-earth, and mobile-to-basestation, are cases where GAM could be particularly valuable.

Optimizing the echo state network based on mutual information for modeling fed-batch bioprocesses

Echo state networks (ESNs) have become one of the most effective dynamic neural networks because of its excellent fitting performance in real-valued time series modeling tasks and simple training processes. The original ESN concept used randomly fixed created reservoirs, and this concept is considered to be one of its main advantages. However, ESNs have been criticized for its randomly created connectivity and weight parameters. Determining the appropriate weight parameters for a given task is an important problem. An optimization method based on mutual information (MI) is proposed in this study to optimize the input scaling parameters and the structure of ESN to address the aforementioned problem and improve the performance of ESN. The MI optimization method mainly consists of two parts: First, the scaling parameters of multiple inputs are adjusted based on the MI between the network inputs and outputs. Second, some output weight connections are pruned for optimization based on the MI between reservoir states. Furthermore, three MI-ESN models are proposed for a fed-batch penicillin fermentation process. Our experimental outcomes reveal that the obtained MI-ESN models outperform the ESN models without optimization and other traditional neural networks.

MRI-only brain radiotherapy verification using cone beam computed tomography

Introduction CBCT are frequently used for patient treatment verification. Hence, generated synthetic-CT (sCT) for MRI-only treatment planning workflow needs to be assessed for image guided radiation therapy (IGRT) using CBCT as well. Purpose For brain cancer patients, we investigated the effectiveness of using sCT that are generated from MRI instead of CT for CBCT-based IGRT. Material/methods Twelve patients with co-registered CT and conventional MRI were used to compute a library of patches. For three patients with two CBCT each, two sCT were generated from MRI using “non-local means” patch-based method (NMPBM) or “sparse” patch-based method (SPBM). CT and sCT alignments with CBCT were performed with MI-Powell. MI-Powell performs Powell optimization of mutual information between volumes and estimates alignment Euler angles and translation. Rigid transformations obtained for CT/CBCT registration were considered as the gold standard. Mean Target Registration Error (mTRE) obtained for sCT/CBCT alignment was computed. mTRE is the average distance between transformed CBCT voxels centers, within head region, by estimated and gold transformation respectively. Moreover, the average errors of displacement (Δ t ) and orientation (Δ θ ) from CT/CBCT registration transformation were computed. Results For sCT NMPBM /CBCT registration assessment, average mTRE = 1.36 ± 0.29 mm, Δ t = 0.92 ±0.2 mm and Δ θ = 1.02 ±0.34°. For sCT SPBM /CBCT registration assessment, average mTRE = 1.4 ± 0.3 mm, Δ t = 0.91 ± 0.21 mm and Δ θ = 1.08 ± 0.43°. Both NMPBM and SPBM were therefore accurate and gave an intermediate sCT suitable for CBCT-based IGRT. Conclusion Based on our results for brain site, CBCT can be used for patient position verification with sCT as a reference. Further work will be done for assessment in other clinical sites with challenging organ motion.

Mutual information optimization based dynamic log-polar image registration

Log-polar transformation (LPT) is widely used in image registration due to its scale and rotation invariant properties. Through LPT, rotation and scale transformation can be made into translation displacement in log-polar coordinates, and phase correlation technique can be used to get the displacement. In LPT based image registration, constant samples in digitalization processing produce less precise and effective results. Thus, dynamic log-polar transformation (DLPT) is used in this paper. DLPT is a method that generates several sample sets in axes to produce several results and only the effective results are used to get the final results by using statistical approach. Therefore, DLPT can get more precise and effective transformation results than the conventional LPT. Mutual information (MI) is a similarity measure to align two images and has been used in image registration for a long time. An optimal transform for image registration can be obtained by maximizing MI between the two images. Image registration based on MI is robust in noisy, occlusion and illumination changing circumstance. In this paper, we study image registration using MI and DLPT. Experiments with digitalizing images and with real image datasets are performed, and the experimental results show that the combination of MI with DLPT is an effective and precise method for image registration.

Network Formation Games in Cooperative MIMO Interference Systems

This paper considers the cooperative optimization of mutual information in the MIMO Gaussian interference channel in a fully distributed manner via game theory. Null shaping constraints are enforced in the design of transmit covariance matrices to enable interference mitigation among links. The transmit covariance matrices leading to the Nash Equilibrium (NE) are derived, and the existence and uniqueness of the NE is analyzed. The formation of the cooperative sets, that represent the cooperation relationship among links, is considered as coalition games and network formation games. We prove that the proposed coalition formation (CF) and coalition graph formation (CGF) algorithms are Nash-stable, and the proposed network formation (NF) algorithm converges to a Nash Equilibrium. Simulation results show that the proposed CF and CGF algorithms have significant advantages when the antennas at the transmitters is large, and the proposed NF algorithm enhances the sum rate of the system apparently even at low signal-to-noise ratio region and/or with small number of transmit antennas.