Concept Of Mutual Information Research Articles

Hybrid similarity relation based mutual information for feature selection in intuitionistic fuzzy rough framework and its applications

Fuzzy rough entropy established in the notion of fuzzy rough set theory, which has been effectively and efficiently applied for feature selection to handle the uncertainty in real-valued datasets. Further, Fuzzy rough mutual information has been presented by integrating information entropy with fuzzy rough set to measure the importance of features. However, none of the methods till date can handle noise, uncertainty and vagueness simultaneously due to both judgement and identification, which lead to degrade the overall performances of the learning algorithms with the increment in the number of mixed valued conditional features. In the current study, these issues are tackled by presenting a novel intuitionistic fuzzy (IF) assisted mutual information concept along with IF granular structure. Initially, a hybrid IF similarity relation is introduced. Based on this relation, an IF granular structure is introduced. Then, IF rough conditional and joint entropies are established. Further, mutual information based on these concepts are discussed. Next, mathematical theorems are proved to demonstrate the validity of the given notions. Thereafter, significance of the features subset is computed by using this mutual information, and corresponding feature selection is suggested to delete the irrelevant and redundant features. The current approach effectively handles noise and subsequent uncertainty in both nominal and mixed data (including both nominal and category variables). Moreover, comprehensive experimental performances are evaluated on real-valued benchmark datasets to demonstrate the practical validation and effectiveness of the addressed technique. Finally, an application of the proposed method is exhibited to improve the prediction of phospholipidosis positive molecules. RF(h2o) produces the most effective results till date based on our proposed methodology with sensitivity, accuracy, specificity, MCC, and AUC of 86.7%, 90.1%, 93.0% , 0.808, and 0.922 respectively.

Distinguishing between intrinsic and instrumental sources of the value of choice

Considerable evidence suggests that people value the freedom to choose. However, it is unclear whether this preference for choice stems purely from choice's intrinsic value, or whether people prefer to choose because it tends to provide instrumental information about desirable outcomes. To address this question, participants completed a novel choice task in which they could freely choose to exert choice or not, manipulating the level of instrumental contingency between participants' choices and eventual outcomes, which we operationalized using the information-theoretic concept of mutual information. Across two experiments (N = 100 each), we demonstrate a marked preference for choice, but importantly found that participants' preference for free choice is weakened when actions are decoupled from outcomes. Taken together, our results demonstrate that a significant factor in people's preference for choice is an assumption about the instrumental value of choice, suggesting against a purely intrinsic value of choice.

Estimating global identifiability using conditional mutual information in a Bayesian framework

A novel information-theoretic approach is proposed to assess the global practical identifiability of Bayesian statistical models. Based on the concept of conditional mutual information, an estimate of information gained for each model parameter is used to quantify the identifiability with practical considerations. No assumptions are made about the structure of the statistical model or the prior distribution while constructing the estimator. The estimator has the following notable advantages: first, no controlled experiment or data is required to conduct the practical identifiability analysis; second, unlike popular variance-based global sensitivity analysis methods, different forms of uncertainties, such as model-form, parameter, or measurement can be taken into account; third, the identifiability analysis is global, and therefore independent of a realization of the parameters. If an individual parameter has low identifiability, it can belong to an identifiable subset such that parameters within the subset have a functional relationship and thus have a combined effect on the statistical model. The practical identifiability framework is extended to highlight the dependencies between parameter pairs that emerge a posteriori to find identifiable parameter subsets. The applicability of the proposed approach is demonstrated using a linear Gaussian model and a non-linear methane-air reduced kinetics model. It is shown that by examining the information gained for each model parameter along with its dependencies with other parameters, a subset of parameters that can be estimated with high posterior certainty can be found.

Survival Prediction of Cervical Cancer Patients using Genetic Algorithm-Based Data Value Metric and Recurrent Neural Network

Survival analysis and machine learning has been shown to be an indispensable aspect of disease management as it enables practitioners to understand and prioritize treatment mostly in terminal diseases. Cervical cancer is the most common malignant tumor of the female reproductive organ worldwide. Survival analysis which is a time –to –event analysis for survival prediction is therefore needed for cervical cancer patients. Data Value Metric (DVM) is an information theoretic measure which uses the concept of mutual information and has shown to be a good metric for quantifying the quality and utility of data as well as feature selection. This study proposed the hybrid of Genetic Algorithm and Data Value Metric for feature selection while Recurrent Neural Network and Cox Proportionality Hazard ratio was used to build the survival prediction model in managing cervical cancer patients. Dataset of 107 patients of cervical cancer patients were collected from University of Benin Teaching Hospital, Benin, Edo State and was used in building the proposed model (RNN+GA-DVM). The proposed system outperform the existing system as the existing system had accuracy of 70% and ROC score of 0.6041 while the proposed model gave an accuracy of 75.16% and ROC score of 0.7120 respectively. From this study, It was observed using the GA_DVM features selection that the variables highly associated with cervical cancer mortality are age_at_diagnosis, Chemotherapy, Chemoradiation, Histology, Comorbidity, Menopause, and MENO_Post. Thus, with early diagnosis and proper health management of cervical cancer, the age of survival of cervical cancer patients can be prolonged.

Squashed entanglement in one-dimensional quantum matter

Squashed entanglement and its universal upper bound, the quantum conditional mutual information, are faithful measures of bipartite quantum correlations defined in terms of multipartitions. As such, they are sensitive to the fine-grain structure of quantum systems. Building on this observation, we introduce the concept of quantum conditional mutual information between the edges of quantum many-body systems. We show that this quantity characterizes unambiguously one-dimensional topological insulators and superconductors, being equal to Bell-state entanglement in the former and to half Bell-state entanglement in the latter, mirroring the different statistics of the edge modes in the two systems. The edge-to-edge quantum conditional mutual information is robust in the presence of disorder or local perturbations, converges exponentially with the system size to a quantized topological invariant, even in the presence of interactions, and vanishes in the trivial phase. We thus conjecture that it coincides with the edge-to-edge squashed entanglement in the entire ground-state phase diagram of symmetry-protected topological systems, and we provide some analytical evidence supporting the claim. By comparing them with the entanglement negativity, we collect further indications that the quantum conditional mutual information and the squashed entanglement provide a very accurate characterization of nonlocal correlation patterns in one-dimensional quantum matter.

Parallel multi-head attention and term-weighted question embedding for medical visual question answering.

The goal of medical visual question answering (Med-VQA) is to correctly answer a clinical question posed by a medical image. Medical images are fundamentally different from images in the general domain. As a result, using general domain Visual Question Answering (VQA) models to the medical domain is impossible. Furthermore, the large-scale data required by VQA models is rarely available in the medical arena. Existing approaches of medical visual question answering often rely on transfer learning with external data to generate good image feature representation and use cross-modal fusion of visual and language features to acclimate to the lack of labelled data. This research provides a new parallel multi-head attention framework (MaMVQA) for dealing with Med-VQA without the use of external data. The proposed framework addresses image feature extraction using the unsupervised Denoising Auto-Encoder (DAE) and language feature extraction using term-weighted question embedding. In addition, we present qf-MI, a unique supervised term-weighting (STW) scheme based on the concept of mutual information (MI) between the word and the corresponding class label. Extensive experimental findings on the VQA-RAD public medical VQA benchmark show that the proposed methodology outperforms previous state-of-the-art methods in terms of accuracy while requiring no external data to train the model. Remarkably, the presented MaMVQA model achieved significantly increased accuracy in predicting answers to both close-ended (78.68%) and open-ended (55.31%) questions. Also, an extensive set of ablations are studied to demonstrate the significance of individual components of the system.

Fast computation of mutual information in the frequency domain with applications to global multimodal image alignment

Multimodal image alignment is the process of finding spatial correspondences between images formed by different imaging techniques or under different conditions, to facilitate heterogeneous data fusion and correlative analysis. The information-theoretic concept of mutual information (MI) is widely used as a similarity measure to guide multimodal alignment processes, where most works have focused on local maximization of MI that typically works well only for small displacements; this points to a need for global maximization of MI, which has previously been computationally infeasible due to the high run-time complexity of existing algorithms. We propose an efficient algorithm for computing MI for all discrete displacements (formalized as the cross-mutual information function (CMIF)), which is based on cross-correlation computed in the frequency domain. We show that the algorithm is equivalent to a direct method while asymptotically superior in terms of run-time. Furthermore, we propose a method for multimodal image alignment for transformation models with few degrees of freedom (e.g. rigid) based on the proposed CMIF-algorithm. We evaluate the efficacy of the proposed method on three distinct benchmark datasets, of aerial images, cytological images, and histological images, and we observe excellent success-rates (in recovering known rigid transformations), overall outperforming alternative methods, including local optimization of MI as well as several recent deep learning-based approaches. We also evaluate the run-times of a GPU implementation of the proposed algorithm and observe speed-ups from 100 to more than 10,000 times for realistic image sizes compared to a GPU implementation of a direct method. Code is shared as open-source at \url{github.com/MIDA-group/globalign}.

PKNN-MIFS: A Parallel KNN Classifier over an Optimal Subset of Features

Feature selection help select an optimal subset of features from a large feature space to achieve better classification performance. The performance of KNN classifier can be improved significantly using an appropriate subset of features from a large feature space. Recent development in General Purpose Graphics Processing Units (GPGPU) has provided us a low cost yet high performance computing support for wide range of applications. This paper presents a parallel KNN classifier powered by a mutual information based feature selection called PKNN-MIFS for effective classification of real life data. It selects an optimal subset of features from the original feature set by exploiting the mutual information concept for the estimation of feature-class and feature-feature relevance. It selects a non-redundant feature by giving higher priority on feature-class relevance. The performance of the proposed PKNN-MIFS has been evaluated over several datasets and has been found to be superior to its closed counterpart.

A computational theory of the subjective experience of flow

Flow is a subjective state characterized by immersion and engagement in one’s current activity. The benefits of flow for productivity and health are well-documented, but a rigorous description of the flow-generating process remains elusive. Here we develop and empirically test a theory of flow’s computational substrates: the informational theory of flow. Our theory draws on the concept of mutual information, a fundamental quantity in information theory that quantifies the strength of association between two variables. We propose that the mutual information between desired end states and means of attaining them — I({M;E}) — gives rise to flow. We support our theory across five experiments (four preregistered) by showing, across multiple activities, that increasing I({M;E}) increases flow and has important downstream benefits, including enhanced attention and enjoyment. We rule out alternative constructs including alternative metrics of associative strength, psychological constructs previously shown to predict flow, and various forms of instrumental value.

Diversity and interdisciplinarity: Should variety, balance and disparity be combined as a product or better as a sum? An information-theoretical and statistical estimation approach

Diversity is a central concept not only in ecology, but also in the social sciences and in bibliometrics. The discussion about an adequate measure of diversity is strongly driven by the work of Rao (Sankhyā Indian J Stat Series A 44:1-22, 1982) and Stirling (J R Soc Interface 4:707-719, 2007). It is to the credit of Leydesdorff (Scientometr 116:2113-2121, 2018) to have proposed a decisive improvement with regard to an inconsistency in the Rao-Sterling-diversity indicator that Rousseau (Scientometr 116:645-653, 2018) had pointed out. With recourse to Shannon's probabilistically based entropy concept, in this contribution the three components of diversity “variety”, “balance”, and “disparity” are to be reconceptualized as entropy masses that add up to an overall diversity indicator dive. Diversity can thus be interpreted as the degree of uncertainty or unpredictability. For "disparity", for example, the concept of mutual information is used. However, probabilities must be estimated statistically. A basic estimation strategy (cross tables) and a more sophisticated one (parametric statistical model) are presented. This overall probability-theoretical based concept is applied exemplarily to data on research output types of funded research projects in UK that were the subject of the Metric Tide Report (REF 2014) and ex-ante evaluation data of a research funding organization. As expected, research output types depend on the research area, with journal articles having the strongest individual balance among the output types, i.e., being represented in almost all research areas. For the ex-ante evaluation data of 1,221 funded projects the diversity components were statistically estimated. The overall diversity of the projects in terms of entropy is 55.5% of the maximal possible entropy.

Quantum Proton Effects from Density Matrix Renormalization Group Calculations.

We recently introduced [J. Chem. Phys. 2020, 152, 204103] the nuclear-electronic all-particle density matrix renormalization group (NEAP-DMRG) method to solve the molecular Schrödinger equation, based on a stochastically optimized orbital basis, without invoking the Born-Oppenheimer approximation. In this work, we combine the DMRG method with the nuclear-electronic Hartree-Fock (NEHF-DMRG) approach, treating nuclei and electrons on the same footing. Inter- and intraspecies correlations are described within the DMRG method without truncating the excitation degree of the full configuration interaction wave function. We extend the concept of orbital entanglement and mutual information to nuclear-electronic wave functions and demonstrate that they are reliable metrics to detect strong correlation effects. We apply the NEHF-DMRG method to the HeHHe+ molecular ion, to obtain accurate proton densities, ground-state total energies, and vibrational transition frequencies by comparison with state-of-the-art data obtained with grid-based approaches and modern configuration interaction methods. For HCN, we improve on the accuracy of the latter approaches with respect to both the ground-state absolute energy and proton density, which is a major challenge for multireference nuclear-electronic state-of-the-art methods.

Degree of polymerization mapping Raman feature extraction of oil-paper insulation aging based on quadratic mutual information.

To assess the aging of oil-paper insulation, an accelerated aging experiment is executed. Raman spectroscopy, a nondestructive detection method with access to component identification and fault diagnosis, is used to analyze the aging of oil-paper insulation. Raman feature of oil-paper insulation aging with a close relationship with the degree of polymerization is obtained based on the concept of quadratic mutual information. By analyzing the relationship between the extracted feature and the degree of polymerization of samples at different aging degrees, the feasibility of reflecting the aging degree of oil-paper insulation is determined. By load analysis, the chemical correlation between the extracted feature and oil-paper insulation aging is clarified. For eight test samples, the prediction error of degree of polymerization based on Raman features is less than 50. These results show that the features extracted in this paper are helpful to realizing the Raman spectrum diagnosis of oil-paper insulation aging.

An efficient feature selection framework based on information theory for high dimensional data

Feature selection plays a vital role in many fields, particularly in pattern recognition and bioinformatics, for selecting informative and relevant features from high dimensional datasets. The increase in dimensionality of data along with the existence of redundant and irrelevant features leads to challenging performance issues when processing and analysing the data. In this paper, an effective feature selection technique called mutual information and Monte Carlo based feature selection (MIMCFS) is proposed. It comprises of two stages. The first stage aims to select predominant features from the high dimensional data. The second stage involves elimination of redundant features that were selected in the first stage. For the purpose of implementing the first stage, a new feature selection strategy based on the approximate Markov blanket and the concept of mutual information is proposed to find out irrelevant and redundant features. In second stage, to avoid misjudgement of redundant features as relevant features, a new strategy based on Monte Carlo tree search technique is proposed in order to completely eradicate redundant features and to improve feature interaction. For experimental evaluation, eight benchmark microarray datasets including imbalanced ones pertaining to cancer analysis are used. Further, in order to compare and justify the performance of the proposed feature selection method, seven state-of-art feature selection techniques namely CFS, Relief, DISR, JMI, CMIM and CMI are employed. The outputs from these feature selection techniques are provided to three standard classifiers namely Naive Bayes, SVM and C4.5 in order to assess the significance of the selected features in building classification models. 10-fold cross validation is adopted to evaluate the classifiers. Accuracy, precision, recall, f-measure, standard deviation, statistical significance metrics are measured to quantify the classifier performance. Experimental results demonstrate the outstanding performance of the proposed algorithm when compared to that of the standard existing methods.

An Artificial Neural Network for Predicting the Near-fault Directivity-pulse Period

ABSTRACT The velocity pulses produced by forward-directivity effects in the near-fault regions can have destructive effects on structures. Proper estimation of the duration of such velocity pulses is an essential step in the near-fault seismic hazard analysis and mitigating potential damage. In this study, the effects of different source, path, source-to-site geometry, and local site parameters on the duration of directivity pulse (T p) are investigated based on the mutual information (MI) concept. A dataset of near-fault pulse-like ground motions from the NGA-West2 database including 135 observations from 17 strike-slip events and 14 non-strike-slip events is utilized for the purpose of this study. The selected ground motion variables are the magnitude, hypocentral distance, depth, D and VS30 that are further applied in an artificial neural network (ANN) to predict T p. The ANN estimates are verified by support vector regression (SVR) as one of the most efficient machine learning algorithms. High correlation between observations and predictions and low error functions reveals the good predictive ability of both ANN and SVR for estimating directivity pulse period. The predictions made by ANN and SVR are further compared with those provided by the empirical and physical models.

Gloria Corpas Pastor and Jean-Pierre Colson (eds). 2020. Computational Phraseology

Computational Phraseology is a compilation of a number of articles and essays dealing with recent research on phraseology, the study of multiword expressions ranging from collocations to formulas and proverbs. The articles mainly deal with the computational aspects of phraseology, in particular the automatic extraction of collocations, which has been a hot topic in linguistics and in lexicography for over 30 years, going back to Church and Hanks’ seminal 1990 paper centred on the concept of mutual information, which is used to identify relations between words which co-occur more often than chance. Over 10 years ago, I noted elsewhere that ‘statistical tools to extract collocations have become such a hot topic that several workshops and conferences have been organised over the last few years to address the role of collocations in lexicography, in applied and computational linguistics and to compare the various approaches that have been proposed’ (in Fontenelle 2008: 12). This is still true today and the range of statistical techniques and practical tools made available to researchers keeps expanding, allowing the latter to identify, extract, formalize, and represent a whole range of ‘prefabricated units’ for a variety of applications (NLP systems, machine translation, dictionaries…).

Change Detection in Electricity Consumption Patterns Utilizing Adaptive Information Theoretic Algorithms

The high-resolution data on electricity consumption, recorded by smart meters at customers' premises, are valuable sources of operational information and consumption patterns. In addition, customers' characterization plays an undeniable role in the implementation of demand response (DR) programs, as any changes in the consumption patterns could affect DR programs. Therefore, an accurate algorithm for detecting changes in the consumption patterns is very useful in not only the implementation of DR but also in other fields, such as load forecasting and peak shaving. This article proposes a reliable procedure for detecting changes in customers' consumption patterns. For this reason, an adaptive algorithm is introduced to improve the clustering quality of customers' consumption patterns by determining the optimum number of clusters, using a locally weighted entropy-based segmentation. Moreover, considering the customers' consumption records in different time slots as the features, another adaptive algorithm is introduced for feature selection based on the mutual information concept. The proposed method is evaluated by applying a real dataset provided by the Irish Social Science Data Archive. The results corroborate the efficiency of the proposed procedure.

A Novel CBIR-Based System using Texture Fused LBP Variants and GLCM Features

This paper presents Multiple-features extraction and reduction for Content-Based Image Retrieval (CBIR). At first, discrete wavelet transform (DWT) is applied on the R, G and B channels to get multi-level decomposition of the image in which approximation and detailed coefficients are extracted. Approximation coefficients contain the main content of the image while detailed coefficients provide the local noise variation in the image. Further computationally efficient and rotation- invariant Dominant-Rotated Local Binary Pattern called a texture descriptor is applied on all approximation and detailed coefficients. By calculating the descriptor relative to reference in a local neighbor patch, a rotation invariance feature image is obtained. The proposed methodology contains the complete structural information extracted by the Local Binary Patterns and also extract the extra information using the information of magnitude, thus attaining extra discriminative power. Then, by getting the Dominant Rotated Local Binary Pattern image, concept of GLCM has been used to extract the statistical characteristics for the classification of texture images. GLCM directly works with the intensity of the images and also provides the spatial relationship of the pixels in the image by calculating frequency of occurring of similar patterns in different directions which makes it useful for the extraction of texture characteristics. GLCM has been improvised into a generalized co-occurrence matrix that extracts significant spatial properties from the distribution of local maxima. Further Median Robust Extended Local Binary Pattern is extracted out of the approximation and detailed coefficients and also the histogram is calculated out of them. Unlike the traditional LBP method and several variants of the LBP, Median Robust Extended Local Binary Pattern compares the local median of images instead of intensities of the raw images. It is a multiscale LBP-type descriptor that proficiently compares the image medians with novel sampling schemes, capable of capturing both microstructure and macrostructure. Further feature concatenation is applied in which GLCM features of DRLBP and histogram of MRELBP are combined for getting a large feature vector. Further, we applied a mutual information concept to sort out the most differentiable features for all categories of CORAL dataset which are further fed to particle swarm optimization-based feature selector which reduced the number of features that can be used in the classification phase. PSO uses the SVM classifier in evaluating its objective function which is average precision value for the selected features. PSO tries to increase precision value for all the categories and provides a feature vector with a large precision value when classify by SVM. Further three classifiers are trained ad tested named SVM, KNN and decision tree in which SVM gives high accuracy and precision rates of classification. Experimental results show above 94% accuracy and .80 to .90 precision values for most of the categories of CORAL dataset.

Mutual Information Input Selector and Probabilistic Machine Learning Utilisation for Air Pollution Proxies

An air pollutant proxy is a mathematical model that estimates an unobserved air pollutant using other measured variables. The proxy is advantageous to fill missing data in a research campaign or to substitute a real measurement for minimising the cost as well as the operators involved (i.e., virtual sensor). In this paper, we present a generic concept of pollutant proxy development based on an optimised data-driven approach. We propose a mutual information concept to determine the interdependence of different variables and thus select the most correlated inputs. The most relevant variables are selected to be the best proxy inputs, where several metrics and data loss are also involved for guidance. The input selection method determines the used data for training pollutant proxies based on a probabilistic machine learning method. In particular, we use a Bayesian neural network that naturally prevents overfitting and provides confidence intervals around its output prediction. In this way, the prediction uncertainty could be assessed and evaluated. In order to demonstrate the effectiveness of our approach, we test it on an extensive air pollution database to estimate ozone concentration.

Accurate estimation of information transfer rate based on symbol occurrence probability in brain-computer interfaces

ObjectiveThe common criteria for evaluating the performance of the brain–computer interface (BCI) are classification accuracy and information transfer rate (ITR). Due to the fact that the BCI system has direct interaction with patients, the accuracy of estimated ITR is very influential. Although the most popular method for ITR estimation is the Wolpaw’s definition, the estimated ITR by this definition is often inaccurate in online applications. One of the existing limitations of it is that all symbols supposed to have the same occurrence probabilities, but symbols do not share the same probability in most real-world applications. MethodsIn this paper, a comprehensive ITR formula is proposed based on symbol probabilities, using the general concept of mutual information. ResultsThe Wolpaw’s definition leads to a strong ITR over-estimation compared to considering the symbol real occurrence probabilities. This estimation error increases with increasing classification accuracy and the number of symbols. For shorter required time to select one symbol, the ITR estimation error is also greater. ConclusionThe proposed Method estimates the ITR more accurately in online applications. SignificanceThe presented formulas provide simplified ITR definition based on symbol probabilities corresponding to a variety of BCI hierarchical structures.

Impact of Monetary Policy on Financial Markets Efficiency under Speculative Bubbles: a Non-Normal and Non-Linear Entropy-based Approach

This paper examines, through the concept of mutual information based on Shannon’s entropy, the impact of monetary policy on the loss of efficiency in the financial markets due to speculative bubbles. The proposed information measure is useful to quantify the efficiency with which stock markets respond to the implementation of monetary policy. The findings show that an increase in both money supply and credit growth, as well as declining interest rates, lead to strong market inefficiencies during the initial periods of formation of a bubble. Moreover, empirical evidence suggests that when a loose monetary policy (money supply is expanded and is accessible to agents to encourage economic growth) generates inefficiencies, its instruments are not effective to realign the performance of financial markets. JEL Classification: E5, G14, D84, C60.