Computation Of Mutual Information Research Articles

MVEB: Self-Supervised Learning With Multi-View Entropy Bottleneck.

Self-supervised learning aims to learn representation that can be effectively generalized to downstream tasks. Many self-supervised approaches regard two views of an image as both the input and the self-supervised signals, assuming that either view contains the same task-relevant information and the shared information is (approximately) sufficient for predicting downstream tasks. Recent studies show that discarding superfluous information not shared between the views can improve generalization. Hence, the ideal representation is sufficient for downstream tasks and contains minimal superfluous information, termed minimal sufficient representation. One can learn this representation by maximizing the mutual information between the representation and the supervised view while eliminating superfluous information. Nevertheless, the computation of mutual information is notoriously intractable. In this work, we propose an objective termed multi-view entropy bottleneck (MVEB) to learn minimal sufficient representation effectively. MVEB simplifies the minimal sufficient learning to maximizing both the agreement between the embeddings of two views and the differential entropy of the embedding distribution. Our experiments confirm that MVEB significantly improves performance. For example, it achieves top-1 accuracy of 76.9% on ImageNet with a vanilla ResNet-50 backbone on linear evaluation. To the best of our knowledge, this is the new state-of-the-art result with ResNet-50.

Enhancing protection in high-dimensional data: Distributed differential privacy with feature selection

The computational cost for implementing data privacy protection tends to rise as the dimensions increase, especially on correlated datasets. For this reason, a faster data protection mechanism is needed to handle high-dimensional data while balancing utility and privacy. This study introduces an innovative framework to improve the performance by leveraging distributed computing strategies. The framework integrates specific feature selection algorithms and distributed mutual information computation, which is crucial for sensitivity assessment. Additionally, it is optimized using a hyperparameter tuning technique based on Bayesian optimization, which focuses on minimizing either a combined score of the Bayesian information criterion (BIC) and Akaike’s Information Criterion (AIC) or by minimizing the Maximal Information Coefficient (MIC) score individually. Extensive testing on 12 datasets with tens to thousands of features was conducted for classification and regression tasks. With our method, the sensitivity of the resulting data is lower than alternative approaches, requiring less perturbation for an equivalent level of privacy. Using a novel Privacy Deviation Coefficient (PDC) metric, we assess the performance disparity between original and perturbed data. Overall, there is a significant execution time improvement of 64.30% on the computation, providing valuable insights for practical applications.

Towards federated feature selection: Logarithmic division for resource-conscious methods

Feature selection is a popular preprocessing step to reduce the dimensionality of the data while preserving the important information. In this paper, we propose an efficient and green feature selection method based on information theory, with the novelty of using the logarithmic division and resorting to fixed-point precision. Moreover, we extend these advancements by adapting the Mutual Information calculation for federated scenarios. The results of experiments conducted on several datasets indicate the potential of our proposal, as it does not incur significant information loss compared to the double-precision method, both in the features selected and in the subsequent classification step. Our method has shown significant potential when applied in federated scenarios, where experimentation demonstrated a lossless feature selection process and maintains classification results compared with centralised versions. These findings open up possibilities towards a new family of green feature selection methods, which would help to minimise energy consumption, lower carbon emissions and increase adaptability to Internet of Things environments.

Covert Surveillance Video Transmission with QAM Constellations Modulation

This study explores the covert transmission of surveillance videos using QAM (quadrature amplitude modulation) constellations. By analyzing transmission strategies used in surveillance video systems, we adopt KL divergence as the constraint for covertness performance and mutual information to characterize transmission rates. Utilizing the Taylor series expansion method, we simplify the complex integral calculations of mutual information and KL divergence into summation operations. This simplification not only reduces computational complexity but also ensures precise calculations. Theoretical derivations establish the necessary conditions for the covert transmission of surveillance videos, optimizing modulation probabilities for constellation points. Experimental results and simulation validations demonstrate the superior performance of our proposed method in terms of both information transfer and precision.

Archetypal analysis of COVID-19 in Montana, USA, March 13, 2020 to April 26, 2022.

Infectious disease data can often involve complex spatial patterns intermixed with temporal trends. Archetypal Analysis is a method to mine complex spatio-temporal data, and can be used to discover the dynamics of spatial patterns. The application of Archetypal Analysis to epidemiological data is relatively new, and here we present one of the first applications on COVID-19 data from March 13, 2020 to April 26, 2022, for the counties of Montana, USA. We present three views of the data set decomposed with Archetypal Analysis. First, we evaluate the entire 56 county data set. Second, we use a mutual information calculation to remove counties whose dynamics are mainly independent from the other counties, reducing the set to 17 counties. Finally, we analyze the top ten counties in terms of population size to focus on the dynamics in the large cities in the state. For each data set, we analyze four significant disease outbreaks across Montana. Archetypal Analysis uncovers distinct spatial patterns for each outbreak and demonstrates that each has a unique trajectory across the state.

Mutual Information Guided Diffusion for Zero-Shot Cross-Modality Medical Image Translation.

Cross-modality data translation has attracted great interest in medical image computing. Deep generative models show performance improvement in addressing related challenges. Nevertheless, as a fundamental challenge in image translation, the problem of zero-shot learning cross-modality image translation with fidelity remains unanswered. To bridge this gap, we propose a novel unsupervised zero-shot learning method called Mutual Information guided Diffusion Model, which learns to translate an unseen source image to the target modality by leveraging the inherent statistical consistency of Mutual Information between different modalities. To overcome the prohibitive high dimensional Mutual Information calculation, we propose a differentiable local-wise mutual information layer for conditioning the iterative denoising process. The Local-wise-Mutual-Information-Layer captures identical cross-modality features in the statistical domain, offering diffusion guidance without relying on direct mappings between the source and target domains. This advantage allows our method to adapt to changing source domains without the need for retraining, making it highly practical when sufficient labeled source domain data is not available. We demonstrate the superior performance of MIDiffusion in zero-shot cross-modality translation tasks through empirical comparisons with other generative models, including adversarial-based and diffusion-based models. Finally, we showcase the real-world application of MIDiffusion in 3D zero-shot learning-based cross-modality image segmentation tasks.

Variational Data-Free Knowledge Distillation for Continual Learning.

Deep neural networks suffer from catastrophic forgetting when trained on sequential tasks in continual learning. Various methods rely on storing data of previous tasks to mitigate catastrophic forgetting, which is prohibited in real-world applications considering privacy and security issues. In this paper, we consider a realistic setting of continual learning, where training data of previous tasks are unavailable and memory resources are limited. We contribute a novel knowledge distillation-based method in an information-theoretic framework by maximizing mutual information between outputs of previously learned and current networks. Due to the intractability of computation of mutual information, we instead maximize its variational lower bound, where the covariance of variational distribution is modeled by a graph convolutional network. The inaccessibility of data of previous tasks is tackled by Taylor expansion, yielding a novel regularizer in network training loss for continual learning. The regularizer relies on compressed gradients of network parameters. It avoids storing previous task data and previously learned networks. Additionally, we employ self-supervised learning technique for learning effective features, which improves the performance of continual learning. We conduct extensive experiments including image classification and semantic segmentation, and the results show that our method achieves state-of-the-art performance on continual learning benchmarks.

An improvised coral reef optimization-based image registration using modified mutual information method

Computer visualisation and medical applications require image registration. It includes transforming collective picture data into the familiar coordinate scheme. Metaheuristic-based methods were developed to explain the issue and improve the efficiency and accuracy of conventional image registration techniques due to their limitations. We describe a hybrid medical image registration technique using bio-inspired meta-heuristic algorithms: Betteroffspring and multi-crossover strategies increase convergent time and solution quality with an improvised coral reef optimization with modified mutual information (ICOR-MMI) algorithm. This new optimization method (ICORMMI) proposes that coral reefs expand, compete for space, and reproduce. A linear weighted sum of image intensity and contour flow model intensity is added to the mutual information calculation. Including statistical and spatial image data improves image registration. The established technique has been tested and verified using multiple medical image data sets, some of which contain single-modality and mixed-modality images (CT, MRI). The registration validates the proposed model's accuracy and efficiency and shows posture's contribution by incorporating statistical and geographical image data. This strategy is adapted to the real-coding problem and tested for realtime issues. The ICOR-MMI algorithm-based hybrid approach outperforms current results in time efficiency and toughness.

Semantic OcTree Mapping and Shannon Mutual Information Computation for Robot Exploration

Autonomous robot operation in unstructured and unknown environments requires efficient techniques for mapping and exploration using streaming range and visual observations. Information-based exploration techniques, such as Cauchy–Schwarz quadratic mutual information and fast Shannon mutual information, have successfully achieved active binary occupancy mapping with range measurements. However, as we envision robots performing complex tasks specified with semantically meaningful concepts, it is necessary to capture semantics in the measurements, map representation, and exploration objective. This work presents semantic octree mapping and Shannon mutual information computation for robot exploration. We develop a Bayesian multiclass mapping algorithm based on an octree data structure, where each voxel maintains a categorical distribution over semantic classes. We derive a closed-form efficiently computable lower bound of the Shannon mutual information between a multiclass octomap and a set of range-category measurements using semantic run-length encoding of the sensor rays. The bound allows rapid evaluation of many potential robot trajectories for autonomous exploration and mapping. We compare our method against state-of-the-art exploration techniques and apply it in a variety of simulated and real-world experiments.

Whole-Slide Imaging, Mutual Information Registration for Multiplex Immunohistochemistry and Immunofluorescence

Multiplex immunohistochemistry/immunofluorescence (mIHC/mIF) is a developing technology that facilitates the evaluation of multiple, simultaneous protein expressions at single-cell resolution while preserving tissue architecture. These approaches have shown great potential for biomarker discovery, yet many challenges remain. Importantly, streamlined cross-registration of multiplex immunofluorescence images with additional imaging modalities and immunohistochemistry (IHC) can help increase the plex and/or improve the quality of the data generated by potentiating downstream processes such as cell segmentation.To address this problem, a fully automated process was designed to perform a hierarchical, parallelizable, and deformable registration of multiplexed digital whole-slide images (WSIs). We generalized the calculation of mutual information as a registration criterion to an arbitrary number of dimensions, making it well suited for multiplexed imaging. We also used the self-information of a given IF channel as a criterion to select the optimal channels to use for registration. Additionally, as precise labeling of cellular membranes in situ is essential for robust cell segmentation, a pan-membrane immunohistochemical staining method was developed for incorporation into mIF panels or for use as an IHC followed by cross-registration. In this study, we demonstrate this process by registering whole-slide 6-plex/7-color mIF images with whole-slide brightfield mIHC images, including a CD3 and a pan-membrane stain. Our algorithm, WSI, mutual information registration (WSIMIR), performed highly accurate registration allowing the retrospective generation of an 8-plex/9-color, WSI, and outperformed 2 alternative automated methods for cross-registration by Jaccard index and Dice similarity coefficient (WSIMIR vs automated WARPY, P < .01 and P < .01, respectively, vs HALO + transformix, P = .083 and P = .049, respectively). Furthermore, the addition of a pan-membrane IHC stain cross-registered to an mIF panel facilitated improved automated cell segmentation across mIF WSIs, as measured by significantly increased correct detections, Jaccard index (0.78 vs 0.65), and Dice similarity coefficient (0.88 vs 0.79).

Learning Representations by Graphical Mutual Information Estimation and Maximization.

The rich content in various real-world networks such as social networks, biological networks, and communication networks provides unprecedented opportunities for unsupervised machine learning on graphs. This paper investigates the fundamental problem of preserving and extracting abundant information from graph-structured data into embedding space without external supervision. To this end, we generalize conventional mutual information computation from vector space to graph domain and present a novel concept, Graphical Mutual Information (GMI), to measure the correlation between input graph and hidden representation. Except for standard GMI which considers graph structures from a local perspective, our further proposed GMI++ additionally captures global topological properties by analyzing the co-occurrence relationship of nodes. GMI and its extension exhibit several benefits: First, they are invariant to the isomorphic transformation of input graphs-an inevitable constraint in many existing methods; Second, they can be efficiently estimated and maximized by current mutual information estimation methods; Lastly, our theoretical analysis confirms their correctness and rationality. With the aid of GMI, we develop an unsupervised embedding model and adapt it to the specific anomaly detection task. Extensive experiments indicate that our GMI methods achieve promising performance in various downstream tasks, such as node classification, link prediction, and anomaly detection.

KIND: A Novel Image-Mutual-Information-Based Decision Fusion Method for Saturation Attack Detection in SD-IoT

Software-defined networking for IoT (SD-IoT), as an emerging architecture, is suffering from numerous security issues. Saturation attacks against the SDN switches and controllers are major security concerns in SD-IoT. When using the Dempster–Shafer evidence theory (DSE) to detect various saturation attacks, the simple mutual information calculation may cause information loss problem, leading to the decrease of detection accuracy. Therefore, how to avoid information loss problem to improve the detection performance is a key issue. Aiming to solve the above-mentioned issues, we propose KIND, a novel image mutual information-based decision fusion method for saturation attack detection in SD-IoT. The main idea of KIND is that it converts the probability matrix of binary classifiers to images and detects the saturation attacks by fusing these images. More specifically, when executing the evidence fusion, each evidence is converted to an image by a nonlinear transformation method. Each image is converted from the related probability-supported matrix. Then, the evidence can be fixed by comparing structural similarities among different images. After that, all evidence can be utilized to obtain the final detection results using the conducted combination rule. The evaluation results demonstrate that KIND can achieve high detection performance, which outperforms other state-of-the-art methods, in terms of TPR, TNR, FPR, FNR, accuracy, precision, recall, F-score, confusion matrix, ROC curves, PR curves, Chi-square test, correlation coefficient, and the quantitative strategy test. In conclusion, KIND can detect saturation attacks with high precision while causing acceptable storage overload.

Fast computation of mutual information in the frequency domain with applications to global multimodal image alignment

Multimodal image alignment is the process of finding spatial correspondences between images formed by different imaging techniques or under different conditions, to facilitate heterogeneous data fusion and correlative analysis. The information-theoretic concept of mutual information (MI) is widely used as a similarity measure to guide multimodal alignment processes, where most works have focused on local maximization of MI that typically works well only for small displacements; this points to a need for global maximization of MI, which has previously been computationally infeasible due to the high run-time complexity of existing algorithms. We propose an efficient algorithm for computing MI for all discrete displacements (formalized as the cross-mutual information function (CMIF)), which is based on cross-correlation computed in the frequency domain. We show that the algorithm is equivalent to a direct method while asymptotically superior in terms of run-time. Furthermore, we propose a method for multimodal image alignment for transformation models with few degrees of freedom (e.g. rigid) based on the proposed CMIF-algorithm. We evaluate the efficacy of the proposed method on three distinct benchmark datasets, of aerial images, cytological images, and histological images, and we observe excellent success-rates (in recovering known rigid transformations), overall outperforming alternative methods, including local optimization of MI as well as several recent deep learning-based approaches. We also evaluate the run-times of a GPU implementation of the proposed algorithm and observe speed-ups from 100 to more than 10,000 times for realistic image sizes compared to a GPU implementation of a direct method. Code is shared as open-source at \url{github.com/MIDA-group/globalign}.

Multivariate Gaussian Copula Mutual Information to Estimate Functional Connectivity with Less Random Architecture.

Recognition of a brain region’s interaction is an important field in neuroscience. Most studies use the Pearson correlation to find the interaction between the regions. According to the experimental evidence, there is a nonlinear dependence between the activities of different brain regions that is ignored by Pearson correlation as a linear measure. Typically, the average activity of each region is used as input because it is a univariate measure. This dimensional reduction, i.e., averaging, leads to a loss of spatial information across voxels within the region. In this study, we propose using an information-theoretic measure, multivariate mutual information (mvMI), as a nonlinear dependence to find the interaction between regions. This measure, which has been recently proposed, simplifies the mutual information calculation complexity using the Gaussian copula. Using simulated data, we show that the using this measure overcomes the mentioned limitations. Additionally using the real resting-state fMRI data, we compare the level of significance and randomness of graphs constructed using different methods. Our results indicate that the proposed method estimates the functional connectivity more significantly and leads to a smaller number of random connections than the common measure, Pearson correlation. Moreover, we find that the similarity of the estimated functional networks of the individuals is higher when the proposed method is used.

Active object search using a pyramid approach to determine the next-best-view

&lt;p&gt;The development of service robotics continues to arouse interest in the scientific community due to the complexity of the activities performed like interaction in human environments, identifying and manipulating objects, and even learning by themselves. This paper proposed to improve the perception of the environment by searching for objects in service robotics tasks. We present the development and implementation of an active object search method based on three main phases: Firstly, image pyramid segmentation to examine in detail the im- age features. Second step, object detection at each level of the pyramid through a local feature descriptor and a mutual information calculation. Finally, the next camera position selection through analyzing the object detections accumulation in the pyramid. To evaluate the implementation of the proposed method, we use a NAO robot in a familiar place for humans, such as an office or a home. Ordinary objects are part of our database with the premise that a robot must know them before looking for an object. The results in the experiments showed an acceptable performance in simulation and with a real platform.&lt;/p&gt;

MiCS-P:Parallel mutual-information computation of big categorical data on spark

Mutual information can effectively measure the correlation between categorical attributes. However, it is found to be quite computationally intensive and time consuming process for enormous size and different distribution data sets. It involves steps for computation of marginal entropies, probability distribution and so on. Spark is a fast, general-purpose parallel framework designed specifically for large-scale data processing. Main motivation of this paper is to provide an intelligent method for parallel mutual information calculation based on Spark computing environment with maintaining the synchronization between different computing nodes. Proposed method named MiCS-P has been able to execute with different number of computing nodes, and gives significant speedup working with different dimensions and sizes of data sets. The MiCS-P algorithm adopts column-wise transformation scheme, which is conducive to the calculation of mutual information between a large number of feature pairs. And to alleviate imbalanced load causing long execution times, we implement a two-phase virtual partitioning scheme running on Spark.

Improved Relief Weight Feature Selection Algorithm Based on Relief and Mutual Information

As the classic feature selection algorithm, the Relief algorithm has the advantages of simple computation and high efficiency, but the algorithm itself is limited to only dealing with binary classification problems, and the comprehensive distinguishing ability of the feature subsets composed of the former K features selected by the Relief algorithm is often redundant, as the algorithm cannot select the ideal feature subset. When calculating the correlation and redundancy between characteristics by mutual information, the computation speed is slow because of the high computational complexity and the method’s need to calculate the probability density function of the corresponding features. Aiming to solve the above problems, we first improve the weight of the Relief algorithm, so that it can be used to evaluate a set of candidate feature sets. Then we use the improved joint mutual information evaluation function to replace the basic mutual information computation and solve the problem of computation speed and correlation, and redundancy between features. Finally, a compound correlation feature selection algorithm based on Relief and joint mutual information is proposed using the evaluation function and the heuristic sequential forward search strategy. This algorithm can effectively select feature subsets with small redundancy and strong classification characteristics, and has the excellent characteristics of faster calculation speed.

Integrative statistical analyses of multiple liquid biopsy analytes in metastatic breast cancer

BackgroundSingle liquid biopsy analytes (LBAs) have been utilized for therapy selection in metastatic breast cancer (MBC). We performed integrative statistical analyses to examine the clinical relevance of using multiple LBAs: matched circulating tumor cell (CTC) mRNA, CTC genomic DNA (gDNA), extracellular vesicle (EV) mRNA, and cell-free DNA (cfDNA).MethodsBlood was drawn from 26 hormone receptor-positive, HER2-negative MBC patients. CTC mRNA and EV mRNA were analyzed using a multi-marker qPCR. Plasma from CTC-depleted blood was utilized for cfDNA isolation. gDNA from CTCs was isolated from mRNA-depleted CTC lysates. CTC gDNA and cfDNA were analyzed by targeted sequencing. Hierarchical clustering was performed within each analyte, and its results were combined into a score termed Evaluation of multiple Liquid biopsy analytes In Metastatic breast cancer patients All from one blood sample (ELIMA.score), which calculates the contribution of each analyte to the overall survival prediction. Singular value decomposition (SVD), mutual information calculation, k-means clustering, and graph-theoretic analysis were conducted to elucidate the dependence between individual analytes.ResultsA combination of two/three/four LBAs increased the prevalence of patients with actionable signals. Aggregating the results of hierarchical clustering of individual LBAs into the ELIMA.score resulted in a highly significant correlation with overall survival, thereby bolstering evidence for the additive value of using multiple LBAs. Computation of mutual information indicated that none of the LBAs is independent of the others, but the ability of a single LBA to describe the others is rather limited—only CTC gDNA could partially describe the other three LBAs. SVD revealed that the strongest singular vectors originate from all four LBAs, but a majority originated from CTC gDNA. After k-means clustering of patients based on parameters of all four LBAs, the graph-theoretic analysis revealed CTC ERBB2 variants only in patients belonging to one particular cluster.ConclusionsThe additional benefits of using all four LBAs were objectively demonstrated in this pilot study, which also indicated a relative dominance of CTC gDNA over the other LBAs. Consequently, a multi-parametric liquid biopsy approach deconvolutes the genomic and transcriptomic complexity and should be considered in clinical practice.

Multi-Modality Non-rigid Image Registration Using Local Similarity Estimations

In this study, we focus on improving the efficiency and accuracy of nonrigid multi-modality registration of medical images. In this regard, we analyze the potentials of using the point similarity measurement approach as an alternative to global computation of mutual information (MI), which is still the most renown multi-modality similarity measure. The improvement capabilities are illustrated using the popular B-spline transformation model. The proposed solution is a combination of three related improvements of the most straightforward implementation, i.e., efficient computation of the voxel displacement field, local estimation of similarity and usage of a static image intensity dependence estimate. Five image registration prototypes were implemented to show contribution and dependence of the proposed improvements. When all the proposed improvements are applied, a significant reduction of computational cost and increased accuracy are obtained. The concept offers additional improvement opportunities by incorporating prior knowledge and machine learning techniques into the static intensity dependence estimation.

Construction of wind turbine performance model based on SCADA data

Compared with traditional wind turbine performance evaluation methods such as operating power curve fitting, the wind turbine performance model construction method using SCADA data on machine learning has gradually attracted attention in the industry in recent years. In this technical route, the choice of input SCADA data type will directly affect the accuracy of the wind turbine performance model construction. However, in the past, selecting key SCADA data types in the performance evaluation of wind turbines were mostly subjective judgments or correlation analyses. This paper proposed a key SCADA data type selection method based on mutual information calculation and a wind turbine performance model construction method based on deep neural network training. The selected key SCADA data types based on mutual information were applied to the deep neural network to construct the wind turbine performance model, and the actual SCADA operation data of the wind farm was applied to test the model. The results show that the model construction method’s accuracy and generalizability using the above technical route can satisfy the industry demand in the same wind farm with the same wind turbine type.