Use Of Mutual Information Research Articles

Using Mutual Information and Information Gain Ratios on Entrepreneurial Research: An Empirical Case for Greek I.T. Start-Ups

The majority of Entrepreneurial quantitative research focuses on Correlation Coefficients. However, new statistical analysis based on Entropy, such as Mutual Information and Information Gain Ratios cast a new light on understanding the relationships among variables and offer a view of non-linear relationships.The study examines key entrepreneurial variables using Mutual Information and Information Gain Ratios and compares findings using the same dataset which examined I.T. Greek Start-Ups. Use of Mutual Information and Information Gain ratios reveals much more relationships between the variables examined, in comparison to Pearson Correlation. Furthermore, the study compares results from Pearson Correlation and Mutual Information and Information Gain ratios to drawn new conclusions on the perceptions of Greek I.T. start-up founders. The findings indicate that use of Mutual Information reveals a set of factors that contribute to entrepreneurial perception of success which differs significantly from the conclusions based on Correlation Coefficient Analysis. More specifically factors such as Operation Years and Previous Start-Ups play a far more crucial role than B2B and Sales. The study offers an original contribution to entrepreneurial science, introducing the use of entropy-based mathematical ratios, such as Mutual Information and Information Gain in Entrepreneurial Research. The study highlights that use and results derived of such ratios enable researchers to identify more information regarding (non-linear) relationships between variables, compared to Correlation Coefficient methods.

Nimbus++: Revisiting Efficient Function Signature Recovery with Depth Data Analysis

Function signature recovery is vital for many binary analysis tasks, led by control-flow integrity enhancement. To minimize human effort, existing works attempt to replace rule-based methods with learning-based methods. These works put a lot of work into improving the system’s performance, but this had the unintended consequence of increasing resource usage. However, recovering the function signature is more about providing information for subsequent tasks, e.g. reverse engineering, so both efficiency and performance are significant. To identify the fundamental factors that increase efficiency, we attempt to optimize data-driven systems throughout their lifecycle from a data perspective. To this end, we perform detailed data analysis on a carefully collected dataset. After analysis and exploration, selective input is adopted and a multi-task learning (MTL) structure is introduced for function feature recovery to make full use of mutual information, and the computing resource overhead is optimized based on the observation of information deviation and sub-task relationship. The resource usage of the entire process is significantly reduced by our suggested solution, named Nimbus++ for efficient function signature recovery, without sacrificing performance. Our test findings demonstrate that we even surpass the state-of-the-art method’s prediction accuracy across all function signature recovery tasks by about 1% with just about 12.5% of the processing time.

Modeling Categorical Variables by Mutual Information Decomposition.

This paper proposed the use of mutual information (MI) decomposition as a novel approach to identifying indispensable variables and their interactions for contingency table analysis. The MI analysis identified subsets of associative variables based on multinomial distributions and validated parsimonious log-linear and logistic models. The proposed approach was assessed using two real-world datasets dealing with ischemic stroke (with 6 risk factors) and banking credit (with 21 discrete attributes in a sparse table). This paper also provided an empirical comparison of MI analysis versus two state-of-the-art methods in terms of variable and model selections. The proposed MI analysis scheme can be used in the construction of parsimonious log-linear and logistic models with a concise interpretation of discrete multivariate data.

Mutual information: Measuring nonlinear dependence in longitudinal epidemiological data.

Given a large clinical database of longitudinal patient information including many covariates, it is computationally prohibitive to consider all types of interdependence between patient variables of interest. This challenge motivates the use of mutual information (MI), a statistical summary of data interdependence with appealing properties that make it a suitable alternative or addition to correlation for identifying relationships in data. MI: (i) captures all types of dependence, both linear and nonlinear, (ii) is zero only when random variables are independent, (iii) serves as a measure of relationship strength (similar to but more general than R2), and (iv) is interpreted the same way for numerical and categorical data. Unfortunately, MI typically receives little to no attention in introductory statistics courses and is more difficult than correlation to estimate from data. In this article, we motivate the use of MI in the analyses of epidemiologic data, while providing a general introduction to estimation and interpretation. We illustrate its utility through a retrospective study relating intraoperative heart rate (HR) and mean arterial pressure (MAP). We: (i) show postoperative mortality is associated with decreased MI between HR and MAP and (ii) improve existing postoperative mortality risk assessment by including MI and additional hemodynamic statistics.

A robust estimator of mutual information for deep learning interpretability

We develop the use of mutual information (MI), a well-established metric in information theory, to interpret the inner workings of deep learning (DL) models. To accurately estimate MI from a finite number of samples, we present GMM-MI (pronounced ‘Jimmie’), an algorithm based on Gaussian mixture models that can be applied to both discrete and continuous settings. GMM-MI is computationally efficient, robust to the choice of hyperparameters and provides the uncertainty on the MI estimate due to the finite sample size. We extensively validate GMM-MI on toy data for which the ground truth MI is known, comparing its performance against established MI estimators. We then demonstrate the use of our MI estimator in the context of representation learning, working with synthetic data and physical datasets describing highly non-linear processes. We train DL models to encode high-dimensional data within a meaningful compressed (latent) representation, and use GMM-MI to quantify both the level of disentanglement between the latent variables, and their association with relevant physical quantities, thus unlocking the interpretability of the latent representation. We make GMM-MI publicly available in this GitHub repository.

Abdominopelvic MR to CT registration using a synthetic CT intermediate.

Accurate coregistration of computed tomography (CT) and magnetic resonance (MR) imaging can provide clinically relevant and complementary information and can serve to facilitate multiple clinical tasks including surgical and radiation treatment planning, and generating a virtual Positron Emission Tomography (PET)/MR for the sites that do not have a PET/MR system available. Despite the long‐standing interest in multimodality co‐registration, a robust, routine clinical solution remains an unmet need. Part of the challenge may be the use of mutual information (MI) maximization and local phase difference (LPD) as similarity metrics, which have limited robustness, efficiency, and are difficult to optimize. Accordingly, we propose registering MR to CT by mapping the MR to a synthetic CT intermediate (sCT) and further using it in a sCT‐CT deformable image registration (DIR) that minimizes the sum of squared differences. The resultant deformation field of a sCT‐CT DIR is applied to the MRI to register it with the CT. Twenty‐five sets of abdominopelvic imaging data are used for evaluation. The proposed method is compared to standard MI‐ and LPD‐based methods, and the multimodality DIR provided by a state of the art, commercially available FDA‐cleared clinical software package. The results are compared using global similarity metrics, Modified Hausdorff Distance, and Dice Similarity Index on six structures. Further, four physicians visually assessed and scored registered images for their registration accuracy. As evident from both quantitative and qualitative evaluation, the proposed method achieved registration accuracy superior to LPD‐ and MI‐based methods and can refine the results of the commercial package DIR when using its results as a starting point. Supported by these, this manuscript concludes the proposed registration method is more robust, accurate, and efficient than the MI‐ and LPD‐based methods.

Methodological considerations for the use of mutual information: Examining the role of context in collocation research

There is a vast literature on collocation, and mutual information scores are frequently used to justify the listing, targeting or indexing of collocations within studies. However, despite methodologically focused papers that discuss the limitations of mutual information scores and the importance of uncertainty estimates in corpus linguistic data, very few applied papers consider the implications of these observations for their analyses. This paper provides a methodologically focused, empirical case study of mutual information scores to examine the extent to which such concerns have implications for study design and the interpretation of results. The paper reports mutual information scores for 100 node words in general American English, and then compares the lists of strong collocates identified with those identified in four corpora representing narrower domains of American English. In so doing, it highlights the range of collocations observed and the relatively poor overlap between conventionally defined strong collocates in general English and those that are identified in narrow domains. The results are discussed in relation to their implications for research involving listing collocates, pedagogical targeting of collocates, and the use of collocation as a correlate of broader constructs such as proficiency. Suggestions are made for calculations that could be used in a test-retest approach to address the issues identified.

The injury severity score: an operations perspective

BackgroundThe statistical evaluation of aggregation functions for trauma grades, such as the Injury Severity Score (ISS), is largely based on measurements of their Pearson product-moment correlation with mortality. However, correlation analysis makes assumptions about the nature of the involved random variables (cardinality) and their relationship (linearity) that may not be applicable to ordinal scores such as the ISS. Moreover, using correlation as a sole evaluation criterion neglects the dynamic properties of these aggregation functions scores.MethodsWe analyze the domain and ordinal properties of the ISS comparatively to arbitrary linear and cubic aggregation functions. Moreover, we investigate the axiomatic properties of the ISS as a multicriteria aggregation procedure. Finally, we use a queuing simulation with various empirical distributions of Abbreviated Injury Scale (AIS) grades reported in the literature, to evaluate the queuing performance of the three aggregation functions.ResultsWe show that the assumptions required for the computation of Pearson’s product-moment correlation coefficients are not applicable to the analysis of the association between the ISS and mortality. We suggest the use of Mutual Information, a information-theoretic statistic that is able to assess general dependence rather than a specialized, linear view based on curve-fitting. Using this metric on the same data set as the seminal study that introduced the ISS, we show that the sum of cubes conveys more information on mortality than the ISS. Moreover, we highlight some unintended, undesirable axiomatic properties of the ISS that can lead to bias in its use as a patient triage criterion. Lastly, our queuing simulation highlights the sensitivity of the queuing performance of different aggregation procedures to the underlying distribution of AIS grades among patients.ConclusionsViewing the ISS, and other possible aggregation functions for multiple AIS scores, as mere operational indicators of the priority of care, rather than cardinal measures of the response of the human body to multiple injuries (as was conjectured in the seminal study introducing the ISS) offers a perspective for their construction and evaluation on more robust grounds than the correlation coefficient. In this regard, Mutual Information appears as a more appropriate measure for the study of the association between injury severity and mortality, and queuing simulations as an actionable way to adapt the choice of an aggregation function to the underlying distribution of AIS scores.

Intensity-Mosaic: automatic panorama mosaicking of disordered images with insufficient features.

Purpose: Handling low-quality and few-feature medical images is a challenging task in automatic panorama mosaicking. Current mosaicking methods for disordered input images are based on feature point matching, whereas in this case intensity-based registration achieves better performance than feature-point registration methods. We propose a mosaicking method that enables the use of mutual information (MI) registration for mosaicking randomly ordered input images with insufficient features. Approach: Dimensionality reduction is used to map disordered input images into a low dimensional space. Based on the low dimensional representation, the image global correspondence can be recognized efficiently. For adjacent image pairs, we optimize the MI metric for registration. The panorama is then created after image blending. We demonstrate our method on relatively lower-cost handheld devices that acquire images from the retina in vivo, kidney ex vivo, and bladder phantom, all of which contain sparse features. Results: Our method is compared with three baselines: AutoStitch, "dimension reduction + SIFT," and "MI-Only." Our method compared to the first two feature-point based methods exhibits 1.25 (ex vivo microscope dataset) to two times (in vivo retina dataset) rate of mosaic completion, and MI-Only has the lowest complete rate among three datasets. When comparing the subsequent complete mosaics, our target registration errors can be 2.2 and 3.8 times reduced when using the microscopy and bladder phantom datasets. Conclusions: Using dimensional reduction increases the success rate of detecting adjacent images, which makes MI-based registration feasible and narrows the search range of MI optimization. To the best of our knowledge, this is the first mosaicking method that allows automatic stitching of disordered images with intensity-based alignment, which provides more robust and accurate results when there are insufficient features for classic mosaicking methods.

Characterization of Event-Based Sampling Encoders for Industrial Internet of Things Using Input–Output Mutual Information

The emergence of Industry 4.0 has resulted in a rapid increase in the demand for bandwidth due to the proliferation of the industrial Internet of Things (IIoT). Increased utilization of the network also gives rise to undesirable consequences such as high latency and increased likelihood of data loss through packet drops. Encoding inputs using event-based sampling is a potential solution for decreasing network traffic generated when continuous input variables are sampled. The problem addressed by this article is the fact that currently there is no effective method for comparing different encoders. The resulting contribution is the use of mutual information to compare the input and encoded output in terms of accuracy for the currently most efficient encoders that use either memory based event triggering (MBET) or deadband error modulation (DEM). This allows a practitioner to select a suitable encoder for a given input specification.

Minimum interpretation by autoencoder-based serial and enhanced mutual information production

The present paper aims to propose an information-theoretic method for interpreting the inference mechanism of neural networks. The new method aims to interpret the inference mechanism minimally by disentangling complex information into simpler and easily interpretable information. This disentanglement of complex information can be realized by maximizing mutual information between input patterns and the corresponding neurons. However, because the use of mutual information has faced difficulty in computation, we use the well-known autoencoder to increase mutual information by re-interpreting the sparsity constraint, which is considered a device to increase mutual information. The computational procedures to increase mutual information are decomposed into the serial operation of equal use of neurons and specific responses to input patterns. The specific responses are realized by enhancing the results by the equal use of neurons. The method was applied to three data sets: the glass, office equipment, and pulsar data sets. With all three data sets, we could observe that, when the number of neurons was forced to increase, mutual information could be increased. Then, collective weights, or average collectively treated weights, showed that the method could extract the simple and linear relations between inputs and targets, making it possible to interpret the inference mechanism minimally.

The Use of Mutual Information to Improve Value-at-Risk Forecasts for Exchange Rates

In this paper, we show a simple but novel approach in an attempt to improve value-at-risk forecasts. We use mutually dependent covariate returns to create exogenous break variables and jointly use the variables to augment GARCH models to account for time-variations and breaks in the unconditional volatility processes simultaneously. A study of hypothetical mutual dependencies between volatility and the covariates is first carried out to investigate the levels of the shared mutual information among the variables before using the augmented models to forecast 1% and 5% value-at-risks. The results provide evidence of some substantial exchange of information between volatility and the lagged exogenous covariates. In addition, the results show that the estimated augmented models have lower volatility persistence, reduced information leakages, and improved explanatory powers. Furthermore, there is evidence that our approach leads to fewer violations, improved 1% value-at-risk forecasts, and optimal daily capital requirements for all the models. There is, however evidence of relative superiority of the majority of the models for the 5% value-at-risks forecasts from our approach, although they have relatively higher failure rates. Based on these results, we recommend the incorporation of our approach to existing risk modeling frameworks. It is believed that such models may lead to fewer bank failures, expose banks to optimal market risks, and assist them in computing optimal regulatory capital requirements and minimize penalties from regulators.

A Innovative Topic Based Customer Complaints Sentiment classification System

Text mining helps to convert big text data into a small amount of most relevant data for a particular problem, and also helps providing knowledge provenance and interpreting patterns. Text mining has to do with further analysing the relevant data to discover the actionable knowledge that can be directly useful for decision making or many other tasks. In the previous work, authors have used conditional entropy with maxima/minima boundaries in identifying aspects from given text in identifying aspects. In the present work, we concentrated on mining word associations using mutual information in measuring co-occurrence of words. The objective of our research is to propose a novel probabilistic latent aspect rating regression (LARR) model. In which, natural language processing is used to represent the text, discovering and analysing topics using expectation maximisation (EM) algorithm. In addition to that, we compared the performance of the proposed model with multiple ordinal logistic regression classifier (MLRC) and fuzzy the rating obtained from the model. The finding in our research is the use of mutual information to mine synatagmatic relations helps in identifying the aspect at document level at greater extend, intern helps to multi level sentiment polarity prediction.

An innovative topic-based customer complaints sentiment classification system

Text mining helps to convert big text data into a small amount of most relevant data for a particular problem, and also helps providing knowledge provenance and interpreting patterns. Text mining has to do with further analysing the relevant data to discover the actionable knowledge that can be directly useful for decision making or many other tasks. In the previous work, authors have used conditional entropy with maxima/minima boundaries in identifying aspects from given text in identifying aspects. In the present work, we concentrated on mining word associations using mutual information in measuring co-occurrence of words. The objective of our research is to propose a novel probabilistic latent aspect rating regression (LARR) model. In which, natural language processing is used to represent the text, discovering and analysing topics using expectation maximisation (EM) algorithm. In addition to that, we compared the performance of the proposed model with multiple ordinal logistic regression classifier (MLRC) and fuzzy the rating obtained from the model. The finding in our research is the use of mutual information to mine synatagmatic relations helps in identifying the aspect at document level at greater extend, intern helps to multi level sentiment polarity prediction.

Measuring transient phase-amplitude coupling using local mutual information

Here we demonstrate the suitability of a local mutual information measure for estimating the temporal dynamics of cross-frequency coupling (CFC) in brain electrophysiological signals. In CFC, concurrent activity streams in different frequency ranges interact and transiently couple. A particular form of CFC, phase-amplitude coupling (PAC), has raised interest given the growing amount of evidence of its possible role in healthy and pathological brain information processing. Although several methods have been proposed for PAC estimation, only a few have addressed the estimation of the temporal evolution of PAC, and these typically require a large number of experimental trials to return a reliable estimate. Here we explore the use of mutual information to estimate a PAC measure (MIPAC) in both continuous and event-related multi-trial data. To validate these two applications of the proposed method, we first apply it to a set of simulated phase-amplitude modulated signals and show that MIPAC can successfully recover the temporal dynamics of the simulated coupling in either continuous or multi-trial data. Finally, to explore the use of MIPAC to analyze data from human event-related paradigms, we apply it to an actual event-related human electrocorticographic (ECoG) data set that exhibits strong PAC, demonstrating that the MIPAC estimator can be used to successfully characterize amplitude-modulation dynamics in electrophysiological data.

Research on Digital Forensics Based on Uyghur Web Text Classification

This paper mainly discusses the use of mutual information (MI) and Support Vector Machines (SVMs) for Uyghur Web text classification and digital forensics process of web text categorization: automatic classification and identification, conversion and pretreatment of plain text based on encoding features of various existing Uyghur Web documents etc., introduces the pre-paratory work for Uyghur Web text encoding. Focusing on the non-Uyghur characters and stop words in the web texts filtering, we put forward a Multi-feature Space Normalized Mutual Information (M-FNMI) algorithm and replace MI between single feature and category with mutual information (MI) between input feature combination and category so as to extract more accurate feature words; finally, we classify features with support vector machine (SVM) algorithm. The experimental result shows that this scheme has a high precision of classification and can provide criterion for digital forensics with specific purpose.

Use of Mutual Information and Transfer Entropy to Assess Interaction between Parasympathetic and Sympathetic Activities of Nervous System from HRV

Obstructive sleep apnea (OSA) is a common sleep disorder that often associates with reduced heart rate variability (HRV) indicating autonomic dysfunction. HRV is mainly composed of high frequency components attributed to parasympathetic activity and low frequency components attributed to sympathetic activity. Although, time domain and frequency domain features of HRV have been used to sleep studies, the complex interaction between nonlinear independent frequency components with OSA is less known. This study included 30 electrocardiogram recordings (20 OSA patient recording and 10 healthy subjects) with apnea or normal label in 1-min segment. All segments were divided into three groups: N-N group (normal segments of normal subjects), P-N group (normal segments of OSA subjects) and P-OSA group (apnea segments of OSA subjects). Frequency domain indices and interaction indices were extracted from segmented RR intervals. Frequency domain indices included nuLF, nuHF, and LF/HF ratio; interaction indices included mutual information (MI) and transfer entropy (TE (H→L) and TE (L→H)). Our results demonstrated that LF/HF ratio was significant higher in P-OSA group than N-N group and P-N group. MI was significantly larger in P-OSA group than P-N group. TE (H→L) and TE (L→H) showed a significant decrease in P-OSA group, compared to P-N group and N-N group. TE (H→L) were significantly negative correlation with LF/HF ratio in P-N group (r = −0.789, p = 0.000) and P-OSA group (r = −0.661, p = 0.002). Our results indicated that MI and TE is powerful tools to evaluate sympathovagal modulation in OSA. Moreover, sympathovagal modulation is more imbalance in OSA patients while suffering from apnea event compared to free event.

Mutual information for intrapartum fetal heart rate analysis.

The analysis of the temporal dynamics in intrapartum fetal heart rate (FHR), aiming at early detection of fetal acidosis, constitutes an intricate signal processing task, that continuously receives significant research efforts. Entropy and entropy rates, envisaged as measures of complexity, often computed via popular implementations referred to as Approximate Entropy (ApEn) or Sample Entropy (SampEn), have regularly been reported as significant features for intrapartum FHR analysis. The present contribution aims to show how mutual information enhances characterization of FHR temporal dynamics and improves fetal acidosis detection performance. To that end, mutual information is first connected to ApEn and SampEn both conceptually and with respect to estimation procedure. Second, mutual information, ApEn and SampEn are computed on a large (≃ 1000 subjects) and documented database of FHR data, collected in a French academic hospital. Reported results show that the use of mutual information permits to significantly outperform ApEn and SampEn for acidosis detection, during any stage of labor.

A Novel Information-Theoretic Approach for Variable Clustering and Predictive Modeling Using Dirichlet Process Mixtures

In the era of big data, there are increasing interests on clustering variables for the minimization of data redundancy and the maximization of variable relevancy. Existing clustering methods, however, depend on nontrivial assumptions about the data structure. Note that nonlinear interdependence among variables poses significant challenges on the traditional framework of predictive modeling. In the present work, we reformulate the problem of variable clustering from an information theoretic perspective that does not require the assumption of data structure for the identification of nonlinear interdependence among variables. Specifically, we propose the use of mutual information to characterize and measure nonlinear correlation structures among variables. Further, we develop Dirichlet process (DP) models to cluster variables based on the mutual-information measures among variables. Finally, orthonormalized variables in each cluster are integrated with group elastic-net model to improve the performance of predictive modeling. Both simulation and real-world case studies showed that the proposed methodology not only effectively reveals the nonlinear interdependence structures among variables but also outperforms traditional variable clustering algorithms such as hierarchical clustering.

Computing algebraic transfer entropy and coupling directions via transcripts.

Most random processes studied in nonlinear time series analysis take values on sets endowed with a group structure, e.g., the real and rational numbers, and the integers. This fact allows to associate with each pair of group elements a third element, called their transcript, which is defined as the product of the second element in the pair times the first one. The transfer entropy of two such processes is called algebraic transfer entropy. It measures the information transferred between two coupled processes whose values belong to a group. In this paper, we show that, subject to one constraint, the algebraic transfer entropy matches the (in general, conditional) mutual information of certain transcripts with one variable less. This property has interesting practical applications, especially to the analysis of short time series. We also derive weak conditions for the 3-dimensional algebraic transfer entropy to yield the same coupling direction as the corresponding mutual information of transcripts. A related issue concerns the use of mutual information of transcripts to determine coupling directions in cases where the conditions just mentioned are not fulfilled. We checked the latter possibility in the lowest dimensional case with numerical simulations and cardiovascular data, and obtained positive results.