Mutual Information Distance Research Articles

Fault Diagnosis for Large-Scale Processes Based on Robust Multiblock Global Orthogonal Projections to Latent Structures

Effective fault diagnosis can be obtained by using multiblock global orthogonal projections to latent structures (MBGOPLS), but there exist certain limitations in this method in terms of block division intelligence and model robustness to outliers. Although these issues can be addressed by developing correlation analysis, such as mutual information and copula-correlation, the complex coupling relationship between variables has not been fully examined. In this study, a robust MBGOPLS is proposed to intelligently diagnose faults using a relatively stable model. First, a double hierarchical clustering method is established, in which internal hierarchical clustering is performed based on Euclidean distance to discretize the samples. Subsequently, external hierarchical clustering is adopted based on mutual information distance to divide variables into different blocks, which results in intelligent block division while suppressing the influence of outliers. Second, a robust block regression coefficient matrix (BRCM) is obtained by employing joint <inline-formula> <tex-math notation="LaTeX">$\ell _{2,1}$</tex-math> </inline-formula>-norm minimization on the BRCM and prediction error of the correlation between the block input and output. Furthermore, BRCM is integrated into the MBGOPLS framework. Therefore, the proposed method is robust to outliers while retaining the diagnostic properties of MBGOPLS. Finally, the proposed method is applied in a numerical case and an actual thermal power plant. The results verify the method&#x2019;s applicability and superiority. <i>Note to Practitioners</i>&#x2014;With increasing large-scale, complex and intelligent to achieve anticipant performance, thermal power plant is prone to faults that can lead to unplanned outages. Meanwhile, complex operation mechanism and environment, and diverse data acquisition sensors make the coupling relationship between variables complex, and collected data often contain numerous outliers, which happens frequently in modern industrial process. Therefore, fault diagnosis is critical to modern industrial process, and the diagnosis accuracy and robustness are the main challenges. This forces us to ensure the block division accuracy and model robustness when using multiblock-based fault diagnosis technology. This paper proposes a robust MBGOPLS to intelligently diagnose faults of thermal power plant using a relatively stable model. Additionally, the proposed method can be extended to fault diagnosis for other large-scale processes.

Prediction of the Critical Temperature of Superconductors Based on Two-Layer Feature Selection and the Optuna-Stacking Ensemble Learning Model.

The study of superconductors' critical temperature (T c) has been a matter of interest. A method combining a two-layer feature selection (TL) and Optuna-Stacking ensemble learning model is proposed in the study for predicting T c from physicochemical components. Since most machine-learning models require a large amount of prior knowledge to construct the feature vectors associated with T c manually, they may contain redundant or invalid features that adversely affect the analysis and prediction of T c. The TL model combines the advantages of filtered and packed feature selection. In the first layer, feature importance is ranked by "SHapley Additive explain (SHAP)" in combination with CatBoost, followed by maximum mutual information coefficient (MIC) and distance correlation coefficient (DCC) for initial feature selection in terms of feature importance ranking. The second layer uses a cross-validation-based genetic algorithm (cv-GA) to eliminate the remaining redundant/invalid features. The selected features are fed into the Stacking integrated learning model to achieve prediction of Tc, and the multidimensional hyperparametric optimization of the metamodel is achieved by Optuna, an improved Bayesian hyperparametric optimization framework based on the Tree-structured Parzen Estimator (TPE) and pruning strategy. The model has obvious advantages and generality in terms of prediction performance and feature reduction rate, and it also proves to be suitable for high-temperature superconductor T c prediction. It provides an efficient and cost-effective method for data-driven superconductor research.

Unsupervised Change Detection by Cross-Resolution Difference Learning

Change detection (CD) aims to identify the differences between multitemporal images acquired over the same geographical area at different times. With the advantages of requiring no cumbersome labeled change information, unsupervised CD has attracted extensive attention of researchers. Multitemporal images tend to have different resolutions as they are usually captured at different times with different sensor properties. It is difficult to directly obtain one pixelwise change map for two images with different resolutions, so current methods usually resize multitemporal images to a unified size. However, resizing operations change the original information of pixels, which limits the final CD performance. This article aims to detect changes from multitemporal images in the originally different resolutions without resizing operations. To achieve this, a cross-resolution difference learning method is proposed. Specifically, two cross-resolution pixelwise difference maps are generated for the two different resolution images and fused to produce the final change map. First, the two input images are segmented into individual homogeneous regions separately due to different resolutions. Second, each pixelwise difference map is produced according to two measure distances, the mutual information distance and the deep feature distance, between image regions in which the pixel lies. Third, the final binary change map is generated by fusing and binarizing the two cross-resolution difference maps. Extensive experiments on four datasets demonstrate the effectiveness of the proposed method for detecting changes from different resolution images.

Regression and Generalized Additive Model to Enhance the Performance of Photovoltaic Power Ensemble Predictors

Photovoltaic (PV) power prediction has a constantly evolving solutions landscape with a myriad of data-driven techniques. Each technique leverages a self-adaptive algorithm that must retrain in intervals, be it each day, week, or season, to avoid the model generalizing poorly because of overfitting, underfitting, or concept drift. This paper aims to improve the generalization capability of PV power predictors such as autoencoders used widely in the industry by introducing feature-enhanced ensemble learning (FEEL) after the feature selection step. This framework uses a combination of nonparametric regression and generalized additive models, and an ensemble of weak regularized multilayer perceptron models. Once trained, the framework can reliably generalize on test data across long time periods without any significant degradation in performance. The proposed framework was validated against the baseline autoencoder-based feature enhancement model on a real PV system from a smart neighborhood in Alabama for September 2019. The FEEL framework performed three times better than the baseline, but when applied to the baseline, its performance improved by two times on average. Furthermore, the framework generalized consistently better than five other feature enhancement strategies. Despite fluctuations in weather, the FEEL framework’s R-square score had a range of 8.1%, whereas that of the baseline was 48.3%. The mutual information and Minkowski distance scores attempted to quantify concept and model drift, respectively. These scores show that the FEEL framework generalized the ensemble learning models at least two times better than the baseline across the different test days. These results form the first step toward decentralized intelligence for smart grid applications that could free up resources for other expensive analytics in the field.

Feature Selection Based on Distance Measurement

Every day we receive a large amount of information through different social media and software, and this data and information can be realized with the advent of data mining methods. In the process of data mining, to solve some high-dimensional problems, feature selection is carried out in limited training samples, and effective features are selected. This paper focuses on two Relief feature selection algorithms: Relief and ReliefF algorithm. The differences between them and their respective applicable scopes are analyzed. Based on Relief algorithm, the high weight feature subset is obtained, and the correlation between features is calculated according to the mutual information distance measure, and the high redundant features are removed to obtain the feature subset with higher quality. Experimental results on six datasets show the effectiveness of our method.

Biomarker Prioritisation and Power Estimation Using Ensemble Gene Regulatory Network Inference.

Inferring the topology of a gene regulatory network (GRN) from gene expression data is a challenging but important undertaking for gaining a better understanding of gene regulation. Key challenges include working with noisy data and dealing with a higher number of genes than samples. Although a number of different methods have been proposed to infer the structure of a GRN, there are large discrepancies among the different inference algorithms they adopt, rendering their meaningful comparison challenging. In this study, we used two methods, namely the MIDER (Mutual Information Distance and Entropy Reduction) and the PLSNET (Partial least square based feature selection) methods, to infer the structure of a GRN directly from data and computationally validated our results. Both methods were applied to different gene expression datasets resulting from inflammatory bowel disease (IBD), pancreatic ductal adenocarcinoma (PDAC), and acute myeloid leukaemia (AML) studies. For each case, gene regulators were successfully identified. For example, for the case of the IBD dataset, the UGT1A family genes were identified as key regulators while upon analysing the PDAC dataset, the SULF1 and THBS2 genes were depicted. We further demonstrate that an ensemble-based approach, that combines the output of the MIDER and PLSNET algorithms, can infer the structure of a GRN from data with higher accuracy. We have also estimated the number of the samples required for potential future validation studies. Here, we presented our proposed analysis framework that caters not only to candidate regulator genes prediction for potential validation experiments but also an estimation of the number of samples required for these experiments.

Dynamics of topic formation and quantitative analysis of hot trends in physical science

Successful research in the face of increasing complexity of modern scientific knowledge together with diversity and depth of the studied problems requires an understanding of the structure and evolution of trends in science. Available digital records open wide possibilities for statistical analysis of scientific publications and related metadata for topic modeling and evolution, knowledge mapping, citation indexing, etc. We investigate dynamical properties of the physical topics using analysis of temporal evolution of proximity measure for word pairs related to the mutual information. We use full-text conceptualization of content of scientific documents provided by the ScienceWISE platform for topic mapping, trend analysis and detection of hot topics together with relevant papers retrieval. We found that time evolution of relative mutual information distance reveals a hidden topic structure and could be used for quantitative analysis of current trends in scientific research.

An Effective Stock Classification Method Via MDS Based on Modified Mutual Information Distance

This paper proposes a multidimensional scaling (MDS) method based on modified mutual information distance (M-MDS) to analyze stock market data. To better describe the relativity of financial data, it is worthwhile to point out that the commonly used proximity matrix in MDS is replaced with modified mutual information distance (M-MI-D) matrix. Refer to M-MI-D, a higher dissimilarity leads to a larger distance. In order to demonstrate the stability and accuracy of M-MDS, logistic time series are used in simulation experiments. In addition, a comparison of this new M-MDS method with classical MDS is given using the stock market data. It is noted that the new M-MDS method shows better stability than that of classical MDS method. Moreover, not only the stocks in the same US stock block, but also the stocks in different blocks have been discussed to illustrate the efficiency of M-MDS method.

Vowel Articulation Dynamic Stability Related to Parkinson's Disease Rating Features: Male Dataset.

Neurodegenerative pathologies as Parkinson's Disease (PD) show important distortions in speech, affecting fluency, prosody, articulation and phonation. Classically, measurements based on articulation gestures altering formant positions, as the Vocal Space Area (VSA) or the Formant Centralization Ratio (FCR) have been proposed to measure speech distortion, but these markers are based mainly on static positions of sustained vowels. The present study introduces a measurement based on the mutual information distance among probability density functions of kinematic correlates derived from formant dynamics. An absolute kinematic velocity associated to the position of the jaw and tongue articulation gestures is estimated and modeled statistically. The distribution of this feature may differentiate PD patients from normative speakers during sustained vowel emission. The study is based on a limited database of 53 male PD patients, contrasted to a very selected and stable set of eight normative speakers. In this sense, distances based on Kullback-Leibler divergence seem to be sensitive to PD articulation instability. Correlation studies show statistically relevant relationship between information contents based on articulation instability to certain motor and nonmotor clinical scores, such as freezing of gait, or sleep disorders. Remarkably, one of the statistically relevant correlations point out to the time interval passed since the first diagnostic. These results stress the need of defining scoring scales specifically designed for speech disability estimation and monitoring methodologies in degenerative diseases of neuromotor origin.

The mutual information based minimum spanning tree to detect and evaluate dependencies between aero-engine gas path system variables

There is a great interest in studying statistical dependence characteristics of aero-engine gas path system time series. The mutual information is effective, mainly in quantifying the dependency of time series. By applying the mutual information and average mutual information method to aero-engine gas path system, the statistical dependence between two data steams from a finite number of samples are established. To better understand dependency of gas path system time series, we define the mutual information distance and propose the mutual information based minimum spanning tree to investigate the performance parameters and their interaction of gas path system. By examining the minimum spanning tree, we find that the exhaust gas temperature (EGT) and the low-spool rotor speed (N1) are confirmed as the predominant variables in fourteen gas path parameters. The results show that the proposed method is effective to detect the statistical dependence of gas path system parameters and has more valuable information.

Appliance of effective clustering technique for gene expression datasets using GPU

The study of medical datasets for analytical purpose is made possible by the innovation of different data mining techniques. Microarrays make simultaneous monitoring of genes under several conditions. Finding out co-expressed genes and coherent patterns is the main goal in bioinformatics research. Cluster analysis of gene expression data has been proven to be a valuable tool for finding biologically groups of genes. The mutual information criteria of the algorithm try to measure the dependency among gene variables. Simulated annealing is applied to solve the local minima problem of K-means algorithm. The improvements in the algorithm utilized further enhances with the use of parallelization techniques. The computational tasks in data mining can be effectively performed by graphics processing units (GPU). An optimized K-means implementation on the GPU using compute unified device architecture (CUDA) of NVIDIA is used as the programming environment. Importance is given on optimizations directly working on data parallel architecture to best use the computational capabilities available. The algorithm is performed in a hybrid manner, parallelizing simulated annealing K-means based on mutual information criteria (MIK). A performance study on medical dataset is performed, demonstrating a maximum 7$$\times $$ speed increase. Experimental analysis shows that the proposed method performs well on gene expression data. The performances of the new clustering methods are compared with those of some existing methods. It is seen that the clustering algorithm based on a combined metric of mutual information and Euclidean distance metric achieves the best performance.

Security Formalizations and Their Relationships for Encryption and Key Agreement in Information-Theoretic Cryptography

This paper revisits formalizations of information-theoretic security for symmetric-key encryption and key agreement protocols which are very fundamental primitives in cryptography. In general, we can formalize information-theoretic security in various ways: some of them can be formalized as stand-alone security by extending (or relaxing) Shannon's perfect secrecy or by other ways such as semantic security; some of them can be done based on composable security. Then, a natural question about this is: what is the gap between the formalizations? To answer the question, we investigate relationships between several formalizations of information-theoretic security for symmetric-key encryption and key agreement protocols. Specifically, for symmetric-key encryption protocols in a general setting including the case where there exist decryption-errors, we deal with the following formalizations of security: formalizations extended (or relaxed) from Shannon's perfect secrecy by using mutual information and statistical distance; information-theoretic analogues of indistinguishability and semantic security by Goldwasser and Micali; and composable security by Maurer et al. and Canetti. Then, we explicitly show the equivalence and non-equivalence between those formalizations. Under the model, we also derive lower bounds on the adversary's (or distinguisher's) advantage and the size of secret-keys required under all of the above formalizations. Although some of them may be already known, we can explicitly derive them all at once through our relationships between the formalizations. In addition, we briefly observe impossibility results which easily follow from the lower bounds. The similar results are also shown for key agreement protocols in a general setting including the case where there exist agreement-errors in the protocols.

Population model of bladder motion and deformation based on dominant eigenmodes and mixed-effects models in prostate cancer radiotherapy.

In radiotherapy for prostate cancer irradiation of neighboring organs at risk may lead to undesirable side-effects. Given this setting, the bladder presents the largest inter-fraction shape variations hampering the computation of the actual delivered dose vs. planned dose. This paper proposes a population model, based on longitudinal data, able to estimate the probability of bladder presence during treatment, using only the planning computed tomography (CT) scan as input information. As in previously-proposed principal component analysis (PCA) population-based models, we have used the data to obtain the dominant eigenmodes that describe bladder geometric variations between fractions. However, we have used a longitudinal analysis along each mode in order to properly characterize patient's variance from the total population variance. We have proposed is a mixed-effects (ME) model in order to separate intra- and inter-patient variability, in an effort to control confounding cohort effects. Other than using PCA, bladder shapes are represented by using spherical harmonics (SPHARM) that additionally enables data compression without information lost. Based on training data from repeated CT scans, the ME model was thus implemented following dimensionality reduction by means of SPHARM and PCA. We have evaluated the model in a leave-one-out cross validation framework on the training data but also using independent data. Probability maps (PMs) were thus generated with several draws from the learnt model as predicted regions where the bladder will likely move and deform. These PMs were compared with the actual regions using metrics based on mutual information distance and misestimated voxels. The prediction was also compared with two previous population PCA-based models. The proposed model was able to reduce the uncertainties in the estimation of the probable region of bladder motion and deformation. This model can thus be used for tailoring radiotherapy treatments.

Enabling network inference methods to handle missing data and outliers.

BackgroundThe inference of complex networks from data is a challenging problem in biological sciences, as well as in a wide range of disciplines such as chemistry, technology, economics, or sociology. The quantity and quality of the data greatly affect the results. While many methodologies have been developed for this task, they seldom take into account issues such as missing data or outlier detection and correction, which need to be properly addressed before network inference.ResultsHere we present an approach to (i) handle missing data and (ii) detect and correct outliers based on multivariate projection to latent structures. The method, called trimmed scores regression (TSR), enables network inference methods to analyse incomplete datasets by imputing the missing values coherently with the latent data structure. Furthermore, it substitutes the faulty values in a dataset by proper estimations. We provide an implementation of this approach, and show how it can be integrated with any network inference method as a preliminary data curation step. This functionality is demonstrated with a state of the art network inference method based on mutual information distance and entropy reduction, MIDER.ConclusionThe methodology presented here enables network inference methods to analyse a large number of incomplete and faulty datasets that could not be reliably analysed so far. Our comparative studies show the superiority of TSR over other missing data approaches used by practitioners. Furthermore, the method allows for outlier detection and correction.Electronic supplementary materialThe online version of this article (doi:10.1186/s12859-015-0717-7) contains supplementary material, which is available to authorized users.

Automated clustering of independent components for discontinuous sounds thoracic imaging.

Discontinuous lung sounds (DLS), also known as crackles, are abnormal sounds produced by different pulmonary pathologies (PP) whose thoracic spatial distribution and prevalence are relevant for diagnosis purpose. Recently, DLS imaging has been proposed to help diagnose and follow-up PP where automated recognition of DLS is meaningful. The present study focuses on the automated selection of independent components (ICs) associated with DLS. Extraction of ICs information for clustering by k-means is achieved in two ways: (1) forming features vectors (FVs) containing the kurtosis, entropy and sparsity of each IC or (2) by applying mutual information (MI) or Euclidean distance (ED) to all ICs. Next, silhouette index is computed to estimate the number of necessary clusters (C). Afterward, to detect just the clusters containing ICs of DLS a selection index is proposed. Finally, to estimate the number of DLS per ICs in each selected cluster a time-variant AR modeling is applied; the estimated number is shown in conjunction with the 2D-ICs spatial distribution. The methodology is applied to simulated and real cases; DLS imaging results are also compared against clinical auscultation. The results showed that the automated selection via FVs is promising to imaging DLS.

MIDER: network inference with mutual information distance and entropy reduction.

The prediction of links among variables from a given dataset is a task referred to as network inference or reverse engineering. It is an open problem in bioinformatics and systems biology, as well as in other areas of science. Information theory, which uses concepts such as mutual information, provides a rigorous framework for addressing it. While a number of information-theoretic methods are already available, most of them focus on a particular type of problem, introducing assumptions that limit their generality. Furthermore, many of these methods lack a publicly available implementation. Here we present MIDER, a method for inferring network structures with information theoretic concepts. It consists of two steps: first, it provides a representation of the network in which the distance among nodes indicates their statistical closeness. Second, it refines the prediction of the existing links to distinguish between direct and indirect interactions and to assign directionality. The method accepts as input time-series data related to some quantitative features of the network nodes (such as e.g. concentrations, if the nodes are chemical species). It takes into account time delays between variables, and allows choosing among several definitions and normalizations of mutual information. It is general purpose: it may be applied to any type of network, cellular or otherwise. A Matlab implementation including source code and data is freely available (http://www.iim.csic.es/~gingproc/mider.html). The performance of MIDER has been evaluated on seven different benchmark problems that cover the main types of cellular networks, including metabolic, gene regulatory, and signaling. Comparisons with state of the art information–theoretic methods have demonstrated the competitive performance of MIDER, as well as its versatility. Its use does not demand any a priori knowledge from the user; the default settings and the adaptive nature of the method provide good results for a wide range of problems without requiring tuning.

A methodology to assess the intrinsic discriminative ability of a distance function and its interplay with clustering algorithms for microarray data analysis

BackgroundClustering is one of the most well known activities in scientific investigation and the object of research in many disciplines, ranging from statistics to computer science. Following Handl et al., it can be summarized as a three step process: (1) choice of a distance function; (2) choice of a clustering algorithm; (3) choice of a validation method. Although such a purist approach to clustering is hardly seen in many areas of science, genomic data require that level of attention, if inferences made from cluster analysis have to be of some relevance to biomedical research.ResultsA procedure is proposed for the assessment of the discriminative ability of a distance function. That is, the evaluation of the ability of a distance function to capture structure in a dataset. It is based on the introduction of a new external validation index, referred to as Balanced Misclassification Index (BMI, for short) and of a nontrivial modification of the well known Receiver Operating Curve (ROC, for short), which we refer to as Corrected ROC (CROC, for short). The main results are: (a) a quantitative and qualitative method to describe the intrinsic separation ability of a distance; (b) a quantitative method to assess the performance of a clustering algorithm in conjunction with the intrinsic separation ability of a distance function. The proposed procedure is more informative than the ones available in the literature due to the adopted tools. Indeed, the first one allows to map distances and clustering solutions as graphical objects on a plane, and gives information about the bias of the clustering algorithm with respect to a distance. The second tool is a new external validity index which shows similar performances with respect to the state of the art, but with more flexibility, allowing for a broader spectrum of applications. In fact, it allows not only to quantify the merit of each clustering solution but also to quantify the agglomerative or divisive errors due to the algorithm.ConclusionsThe new methodology has been used to experimentally study three popular distance functions, namely, Euclidean distance d2, Pearson correlation dr and mutual information dMI. Based on the results of the experiments, we have that the Euclidean and Pearson correlation distances have a good intrinsic discrimination ability. Conversely, the mutual information distance does not seem to offer the same flexibility and versatility as the other two distances. Apparently, that is due to well known problems in its estimation. since it requires that a dataset must have a substantial number of features to be reliable. Nevertheless, taking into account such a fact, together with results presented in Priness et al., one receives an indication that dMI may be superior to the other distances considered in this study only in conjunction with clustering algorithms specifically designed for its use. In addition, it results that K-means, Average Link, and Complete link clustering algorithms are in most cases able to improve the discriminative ability of the distances considered in this study with respect to clustering. The methodology has a range of applicability that goes well beyond microarray data since it is independent of the nature of the input data. The only requirement is that the input data must have the same format of a "feature matrix". In particular it can be used to cluster ChIP-seq data.

Analysis and evolution of air quality monitoring networks using combined statistical information indexes

In this work, we present combined statistical indexes for evaluating air quality monitoring networks based on concepts derived from the information theory and Kullback–Liebler divergence. More precisely, we introduce: (1) the standard measure of complementary mutual information or ‘specificity’ index; (2) a new measure of information gain or ‘representativity’ index; (3) the information gaps associated with the evolution of a network and (4) the normalised information distance used in clustering analysis. All these information concepts are illustrated by applying them to 14 yr of data collected by the air quality monitoring network in Santiago de Chile (33.5 S, 70.5 W, 500 m a.s.l.). We find that downtown stations, located in a relatively flat area of the Santiago basin, generally show high ‘representativity’ and low ‘specificity’, whereas the contrary is found for a station located in a canyon to the east of the basin, consistently with known emission and circulation patterns of Santiago. We also show interesting applications of information gain to the analysis of the evolution of a network, where the choice of background information is also discussed, and of mutual information distance to the classifications of stations. Our analyses show that information as those presented here should of course be used in a complementary way when addressing the analysis of an air quality network for planning and evaluation purposes.

On the systematic reduction of data complexity in multimodel atmospheric dispersion ensemble modeling

The aim of this work is to explore the effectiveness of theoretical information approaches for the reduction of data complexity in multimodel ensemble systems. We first exploit a weak form of independence, i.e. uncorrelation, as a mechanism for detecting linear relationships. Then, stronger and more general forms of independence measure, such as mutual information, are used to investigate dependence structures for model selection. A distance matrix, measuring the interdependence between data, is derived for the investigated measures, with the scope of clustering correlated/dependent models together. Redundant information is discarded by selecting a few representative models from each cluster. We apply the clustering analysis in the context of atmospheric dispersion modeling, by using the ETEX‐1 data set. We show how the selection of a small subset of models, according to uncorrelation or mutual information distance criteria, usually suffices to achieve a statistical performance comparable to, or even better than, that achieved from the whole ensemble data set, thus providing a simpler description of ensemble results without sacrificing accuracy.

A GPU-based Implementation of Multimodal Deformable Image Registration Based on Mutual Information or Bhattacharyya Distance

This paper describes a multimodal deformable image registration method on the GPU. It is a CUDA-based implementation of a paper by E. D’Agostino et. al, ‘’A viscous fluid model for multimodal non-rigid image registration using mutual information’’. In addition, we incorporate an alternative metric as opposed to mutual information, called Bhattacharyya Distance, in the recent work of Lou and Tannenbaum. This paper is accompanied with the source code, input data, parameters and output data that the authors used for validating the algorithm.