Terms Of Mutual Information Research Articles

Multi-modal image fusion technique for enhancing image quality with multi-scale decomposition algorithm

ABSTRACT Multi-modality image fusion is an effective technique for fusing complementary information from multi-modality images into an integrated image. The additional information obtained from fused images not only enhances visibility to the human eye, but also complements the limitations of each image. The existing image fusion techniques lacks in preserving the quality of image. Therefore to preserve the structure information and perform the detailed information of source images, a novel image fusion scheme based on Multi-Scale Singular Value Decomposition (MSVD) is proposed. Initially, source images are decomposed into detailed and smooth layer by means of MSVD. Then low- and high-frequency components are extracted with the help of proposed filtering technique and subtraction rule. For efficiently selecting appropriate components, Optimisation algorithm is performed by maximising the mutual information of images. Finally, based on additive fusion rule, the fused images are reconstructed through inverse of MSVD approach. The efficiency of the proposed fusion rule is determined by the comparative analysis with existing image fusion method in terms of Mutual Information (MI), Standard Deviation, Entropy, Correlation etc. The experimental outcome shows the superiority of proposed method and it outperforms state-of-the-art methods by providing better quality image with good clarity. Furthermore, the fused image illustrates better visual effect and edge consistency.

Research synthesis of information theory measures of uncertainty: Meta-analysis of entropy and mutual information of diagnostic tests.

Assessing the performance of diagnostic tests requires evaluation of the amount of diagnostic uncertainty a test reduces. Statistical measures, such as sensitivity and specificity, currently dominating the evidence-based medicine (EBM) and related fields, cannot explicitly measure this reduction in diagnostic uncertainty. Mutual information (MI), an information theory statistic, explicitly quantifies diagnostic uncertainty by measuring information gain before vs after diagnostic testing. In this paper, we propose the use of MI as a single measure to express diagnostic test performance and demonstrate how it can be used in the meta-analysis of diagnostic test studies. We use two case studies from the literature to demonstrate the applicability of MI meta-analysis in assessing diagnostic performance. Meta-analysis of studies evaluating (a) ultrasonography (US) to detect endometrial cancer and (b) magnetic resonance angiography to detect arterial stenosis. The results of MI meta-analyses are comparable to those of traditional statistical measures' meta-analyses. However, the results of MI are easier to understand as it relates directly to the extent of uncertainty a diagnostic test can reduce. For example, the US test, diagnosing endometrial cancer, is 40% specific and 94% sensitive. The combination of these values is difficult to interpret and may lead to inappropriate assessment (eg, one could favour the test due to its high sensitivity, ignoring its low specificity). In terms of MI, however, a single metric shows that the test reduces diagnostic uncertainty by 10%, which many users may consider small under most circumstances. We have demonstrated the suitability of MI in assessing the performance of diagnostic tests, which can facilitate easier interpretation of the true utility of diagnostic tests. Similarly, to the guidance for interpretation of effect size of treatment interventions, we also propose the guidelines for interpretation of the utility of diagnostic tests based on the magnitude of reduction in diagnostic uncertainty.

Variational Information Bottleneck for Semi-Supervised Classification.

In this paper, we consider an information bottleneck (IB) framework for semi-supervised classification with several families of priors on latent space representation. We apply a variational decomposition of mutual information terms of IB. Using this decomposition we perform an analysis of several regularizers and practically demonstrate an impact of different components of variational model on the classification accuracy. We propose a new formulation of semi-supervised IB with hand crafted and learnable priors and link it to the previous methods such as semi-supervised versions of VAE (M1 + M2), AAE, CatGAN, etc. We show that the resulting model allows better understand the role of various previously proposed regularizers in semi-supervised classification task in the light of IB framework. The proposed IB semi-supervised model with hand-crafted and learnable priors is experimentally validated on MNIST under different amount of labeled data.

Non-Uniform Embedding Scheme and Low-Dimensional Approximation Methods for Causality Detection.

Information causality measures have proven to be very effective in uncovering the connectivity patterns of multivariate systems. The non-uniform embedding (NUE) scheme has been developed to address the “curse of dimensionality”, since the estimation relies on high-dimensional conditional mutual information (CMI) terms. Although the NUE scheme is a dimension reduction technique, the estimation of high-dimensional CMIs is still required. A possible solution is the utilization of low-dimensional approximation (LA) methods for the computation of CMIs. In this study, we aim to provide useful insights regarding the effectiveness of causality measures that rely on NUE and/or on LA methods. In a comparative study, three causality detection methods are evaluated, namely partial transfer entropy (PTE) defined using uniform embedding, PTE using the NUE scheme (PTENUE), and PTE utilizing both NUE and an LA method (LATE). Results from simulations on well known coupled systems suggest the superiority of PTENUE over the other two measures in identifying the true causal effects, having also the least computational cost. The effectiveness of PTENUE is also demonstrated in a real application, where insights are presented regarding the leading forces in financial data.

Towards a generalization of information theory for hierarchical partitions.

Complex systems often exhibit multiple levels of organization covering a wide range of physical scales, so the study of the hierarchical decomposition of their structure and function is frequently convenient. To better understand this phenomenon, we introduce a generalization of information theory that works with hierarchical partitions. We begin revisiting the recently introduced hierarchical mutual information (HMI), and show that it can be written as a level by level summation of classical conditional mutual information terms. Then, we prove that the HMI is bounded from above by the corresponding hierarchical joint entropy. In this way, in analogy to the classical case, we derive hierarchical generalizations of many other classical information-theoretic quantities. In particular, we prove that, as opposed to its classical counterpart, the hierarchical generalization of the variation of information is not a metric distance, but it admits a transformation into one. Moreover, focusing on potential applications of the existing developments of the theory, we show how to adjust by chance the HMI. We also corroborate and analyze all the presented theoretical results with exhaustive numerical computations, and include an illustrative application example of the introduced formalism. Finally, we mention some open problems that should be eventually addressed for the proposed generalization of information theory to reach maturity.

Adversarial Disentanglement with Grouped Observations

We consider the disentanglement of the representations of the relevant attributes of the data (content) from all other factors of variations (style) using Variational Autoencoders. Some recent works addressed this problem by utilizing grouped observations, where the content attributes are assumed to be common within each group, while there is no any supervised information on the style factors. In many cases, however, these methods fail to prevent the models from using the style variables to encode content related features as well. This work supplements these algorithms with a method that eliminates the content information in the style representations. For that purpose the training objective is augmented to minimize an appropriately defined mutual information term in an adversarial way. Experimental results and comparisons on image datasets show that the resulting method can efficiently separate the content and style related attributes and generalizes to unseen data.

Health monitoring scheme‐based Forster resonance energy transfer nanocommunications in the Internet of Biological Nanothings

SummaryIn this paper, a health monitoring approach for a targeted drug delivery (TDD) system taking into consideration the binding of drug to a target cell is proposed. The proposed scheme is mainly based on Forster resonance energy transfer (FRET) nanocommunications and the physical phenomenon such as protein–protein interactions in biological systems, in order to propose a molecular communication system with various forms for single and/or multiple ligand–receptor binding. Inspired by classical wireless communication system, the analysis of proposed scheme in terms of bit error rate (BER) and mutual information between the transmitter nanomachine (TN) and receiver nanomachine (RN) is presented. Subsequently, we are able to obtain a closed‐form expression for BER for the proposed scheme. The proposed scheme can be employed in the promising Internet of Biological Nanothings (IoBNT) paradigm. The numerical results reveal that the proposed scheme with multiple donors multiple acceptors system analogous to multiple input multiple output (MIMO) communication system is a superior scheme for transmission therapeutic drug information with achievable BER.

Molecular communication in three-dimensional diffusive channel with mobile nanomachines

A three-dimensional (3-D) diffusive mobile-molecular communication system is considered inside the unbounded environment. We consider a point transmitter and a perfectly absorbing spherical receiver which can move randomly in any of the three coordinates. The closed-form expression for expected hitting rate for 3-D unbounded communication channel with nanomachines mobility is derived. The performance of considered system is further evaluated in terms of probability of detection (PD), probability of false alarm (PFA), and probability of error (Pe). The system is further analyzed in terms of average mutual information and average maximum achievable rate. Furthermore, an optimum decision threshold, at which minimum probability of error and maximum information rate occur is also derived. The analytical results are verified through particle-based and Monte-Carlo simulations.

Protecting Your Shopping Preference With Differential Privacy

Online banks may disclose consumers’ shopping preferences due to various attacks. With differential privacy, each consumer can disturb his consumption amount locally before sending it to online banks. However, directly applying differential privacy in online banks will incur problems in reality because existing differential privacy schemes do not consider handling the noise boundary problem. In this paper, we propose an Optimized Differential prIvate Online tRansaction scheme (O-DIOR) for online banks to set boundaries of consumption amounts with added noises. We then revise O-DIOR to design a RO-DIOR scheme to select different boundaries while satisfying the differential privacy definition. Moreover, we provide in-depth theoretical analysis to prove that our schemes are capable to satisfy the differential privacy constraint. Finally, to evaluate the effectiveness, we have implemented our schemes in mobile payment experiments. Experimental results illustrate that the relevance between the consumption amount and online bank amount is reduced significantly, and the privacy losses are less than 0.5 in terms of mutual information.

On the Difference between the Information Bottleneck and the Deep Information Bottleneck.

Combining the information bottleneck model with deep learning by replacing mutual information terms with deep neural nets has proven successful in areas ranging from generative modelling to interpreting deep neural networks. In this paper, we revisit the deep variational information bottleneck and the assumptions needed for its derivation. The two assumed properties of the data, X and Y, and their latent representation T, take the form of two Markov chains and . Requiring both to hold during the optimisation process can be limiting for the set of potential joint distributions . We, therefore, show how to circumvent this limitation by optimising a lower bound for the mutual information between T and Y: , for which only the latter Markov chain has to be satisfied. The mutual information can be split into two non-negative parts. The first part is the lower bound for , which is optimised in deep variational information bottleneck (DVIB) and cognate models in practice. The second part consists of two terms that measure how much the former requirement is violated. Finally, we propose interpreting the family of information bottleneck models as directed graphical models, and show that in this framework, the original and deep information bottlenecks are special cases of a fundamental IB model.

An Uplink SCMA Codebook Design Combining Probabilistic Shaping and Geometric Shaping

Sparse Code Multiple Access (SCMA) is a promising non-orthogonal multiple access scheme owning to the shaping gain of its multi-dimensional codebook. Most of the existing designs of the codebook are based on geometric shaping (GS), while probabilistic shaping (PS) has important advantages in increasing channel capacity and reducing bit error rate. In this paper, an uplink SCMA codebook optimization algorithm which combines PS and GS is proposed. The algorithm adopts the bare bones particle swarm optimization algorithm based on maximizing the AMI. Based on the non-equiprobable codebook, a message passing algorithm with non-equalprobable distribution is introduced as the multiuser detection algorithm for SCMA scheme. We theoretically analyze the superiority of the codebooks combining PS and GS over others in terms of the average mutual information (AMI). Simulation results show that our proposed codebooks outperform the reference GS based codebooks under different overloading, different channels and different codebook sizes, and with the increase of overloading or codebook sizes, the gains are greater, which is up to 2.1dB gain in terms of block error rate performance. This observation also validates the theoretical analysis based on the AMI.

Sabotage Attack Detection for Additive Manufacturing Systems

This paper presents a novel multi-modal sabotage attack detection system for Additive Manufacturing (AM) machines. By utilizing multiple side-channels, we improve system state estimation significantly in comparison to uni-modal techniques. Besides, we analyze the value of each side-channel for performing attack detection in terms of mutual information shared with the machine control parameters. We evaluate our system on real-world test cases and achieve an attack detection accuracy of 98.15%. AM, or 3D Printing, is seeing practical use for the rapid prototyping and production of industrial parts. The digitization of such systems not only makes AM a crucial technology in Industry 4.0 but also presents a broad attack surface that is vulnerable to kinetic cyberattacks. In the field of AM security, sabotage attacks are cyberattacks that introduce inconspicuous defects to a manufactured component at any specific process of the AM digital process chain, resulting in the compromise of the component’s structural integrity and load-bearing capabilities. Defense mechanisms that detect such attacks using side-channel analysis have been studied. However, most current works focus on modeling the state of AM systems using a single side-channel, thus limiting their effectiveness at attack detection. In this paper, we demonstrate the value of a multi-modal sabotage attack detection system in comparison to uni-modal techniques.

Object-based feature extraction for hyperspectral data using firefly algorithm

Object-based classification methods can improve the accuracy of hyperspectral image classification due to the fact that they incorporate spatial information into the classification procedure by assigning neighboring pixels into the same class. In this paper, a new object-based feature extraction method is proposed which makes use of information theory to reduce the Bayes error. In this way, the proposed method exploits higher order statistics for feature extraction which are very effective for non Gaussian data such as hyperspectral images. The criterion to be minimized is composed of three mutual information terms. The first and second terms, consider the maximal relevance and minimal redundancy, respectively, while the third term takes into account the segmentation map containing disjoint spatial regions. To obtain the segmentation map, we apply the firefly clustering algorithm whose fitness function simultaneously considers the intra-distance between samples and their cluster centroids, and inter-distance between centroids of any two clusters. Our experimental results, performed using a variety of hyperspectral scenes, indicate that the proposed framework gives better classification results than some state-of the-art spectral–spatial feature extraction methods.

Unary Coding Controlled Simultaneous Wireless Information and Power Transfer

Radio frequency (RF) signals have been relied upon for both wireless information delivery and wireless charging to the massively deployed low-power Internet of Things (IoT) devices. Extensive efforts have been invested in physical layer and medium-access-control layer design for coordinating simultaneous wireless information and power transfer (SWIPT) in RF bands. Different from the existing works, we study the coding controlled SWIPT from the information theoretical perspective with practical transceiver. Due to its practical decoding implementation and its flexibility on the codeword structure, unary code is chosen for joint information and energy encoding. Wireless power transfer (WPT) performance in terms of energy harvested per binary sign and of battery overflow/underflow probability is maximised by optimising the codeword distribution of coded information source, while satisfying required wireless information transfer (WIT) performance in terms of mutual information. Furthermore, a Genetic Algorithm (GA) aided coding design is proposed to reduce the computational complexity. Numerical results characterise the SWIPT performance and validate the optimality of our proposed GA aided unary coding design.

Continuous-variable ramp quantum secret sharing with Gaussian states and operations

We aim to quantify and mitigate quantum-information leakage in continuous-variable quantum secret sharing (CV QSS). Here we introduce a technique for certifying CV ramp quantum secret-sharing (RQSS) schemes in the framework of quantum interactive-proof systems. We devise pseudocodes in order to represent the sequence of steps taken to solve the certification problem. Furthermore, we derive the expression for quantum mutual information between the quantum secret extracted by any multi-player structure and the share held by the referee corresponding to the Tyc–Rowe–Sanders CV QSS scheme. We solve by converting the Tyc–Rowe–Sanders position representation for the state into a Wigner function from which the covariance matrix can be found, then insert the covariance matrix into the standard formula for CR quantum mutual information to obtain quantum mutual information in terms of squeezing. Our quantum mutual information result quantifies the leakage of the RQSS schemes.

Multi‐exposure image fusion technique using multi‐resolution blending

This study proposes a multi-exposure image fusion (MEF) technique that takes multi-exposure input images and produces a high-quality output image without any artefacts. The proposed technique consists of four steps. In the first step, three quality measures (contrast, saturation, and well exposedness) are measured. Secondly, the colour dissimilarity approach is used to detect moving objects. Thirdly, the authors calculate the weight maps using three quality measures (contrast, saturation, and well exposedness) and colour dissimilarity feature. Finally, the fused image is generated using the multi-resolution blending utilising pyramid decomposition. The vital advantage of the proposed technique is that it blends the multi-exposure images very well and avoids the seams effectively. The method can be used for consumer cameras as the presented technique is quite fast. Experimental results prove that the proposed method produces high dynamic range images without ghost artefacts. Furthermore, the comparison, objectively and subjectively, of the proposed technique in terms of mutual information, MEF structural similarity index, and natural image quality evaluator, with state-of-the-art techniques, shows significant improvement of the proposed scheme over existing techniques.

Finger movements recognition using minimally redundant features of wavelet denoised EMG

Developing prosthetic hands with high functionality and ease of use is the focus of current research in the area of Electromyogram (EMG) based prosthesis control. Although individuals with upper limb loss can perform grasping operations with currently available prosthetic hands, more intuitive control of finger movements is required to replicate the complex motor functions of human hands. A significant challenge is to classify the finger movements with higher recognition rates using a smaller number of EMG channels. This paper reports a novel criterion for selection of minimally redundant EMG feature set for classification of 10-class finger movements using two-channel EMG. The feature set is selected from wavelet denoised EMG at four decomposition levels using minimum redundancy in terms of mutual information. A set of five features: root mean square, simple square integral, slope sign change, peak frequency and power spectral ratio have been selected from 31 time and frequency domain features. Using the current state of the art classification technique based on support vector machine, we achieved 10-fold cross-validation recognition rate of 96.5 ± 0.13%. The experimental study shows that the feature set with minimum redundancy in terms of mutual information ensures the highest recognition rate with reduced computational cost.

Efficient feature selection using shrinkage estimators

Information theoretic feature selection methods quantify the importance of each feature by estimating mutual information terms to capture: the relevancy, the redundancy and the complementarity. These terms are commonly estimated by maximum likelihood, while an under-explored area of research is how to use shrinkage methods instead. Our work suggests a novel shrinkage method for data-efficient estimation of information theoretic terms. The small sample behaviour makes it particularly suitable for estimation of discrete distributions with large number of categories (bins). Using our novel estimators we derive a framework for generating feature selection criteria that capture any high-order feature interaction for redundancy and complementarity. We perform a thorough empirical study across datasets from diverse sources and using various evaluation measures. Our first finding is that our shrinkage based methods achieve better results, while they keep the same computational cost as the simple maximum likelihood based methods. Furthermore, under our framework we derive efficient novel high-order criteria that outperform state-of-the-art methods in various tasks.

Quantifying Information Conveyed by Large Neuronal Populations.

Quantifying mutual information between inputs and outputs of a large neural circuit is an important open problem in both machine learning and neuroscience. However, evaluation of the mutual information is known to be generally intractable for large systems due to the exponential growth in the number of terms that need to be evaluated. Here we show how information contained in the responses of large neural populations can be effectively computed provided the input-output functions of individual neurons can be measured and approximated by a logistic function applied to a potentially nonlinear function of the stimulus. Neural responses in this model can remain sensitive to multiple stimulus components. We show that the mutual information in this model can be effectively approximated as a sum of lower-dimensional conditional mutual information terms. The approximations become exact in the limit of large neural populations and for certain conditions on the distribution of receptive fields across the neural population. We empirically find that these approximations continue to work well even when the conditions on the receptive field distributions are not fulfilled. The computing cost for the proposed methods grows linearly in the dimension of the input and compares favorably with other approximations.

The Synergy and Cycle Values in Regional Innovation Systems: The Case of Norway

The innovation capacity of a system can be measured as the synergy in interactions among its parts. Synergy can be considered as a consequence of negative entropies among three parts of the system. We analyze the development of synergy value in the Norwegian innovation system in terms of mutual information among geographical, sectorial, and size distributions of firms. We use three different techniques for the evaluation of the evolution of synergy over time: rescaled range analysis, DFT, and geographical synergy decomposition. The data was provided by Statistics Norway for all Norwegian firms registered in the database between 2002 and 2014. The results suggest that the synergy at the level of both the country and its seven regions show non-chaotic oscillatory behavior which resonates in a set of natural frequencies. The finding of a set of frequencies implies a complex Triple-Helix structure, composed of many elementary triple helices, which can be theorized in terms of a fractal TH manifold.