Multivariate Mutual Information Research Articles

Research on rotor fault diagnosis technology of three-phase asynchronous motor based on NA-MEMD mutual information and SVM

Aiming at the problem of mode aliasing in the adaptive decomposition of nonlinear and non-stationary current signals generated by three-phase asynchronous motor faults, and the fault features contained in signals collected by a single sensor can not be accurately and comprehensively extracted and characterized when early rotor bar breakage and air gap eccentricity faults occur, A fault diagnosis method for three-phase asynchronous motor based on noise assisted multivariate empirical mode decomposition (NA-MEMD) and mutual information is proposed. Firstly, the NA-MEMD algorithm is used to decompose the three-phase stator current signal of the asynchronous motor to obtain multi-scale intrinsic mode functions (IMFs). Then, the correlation algorithm is used to screen the IMFs containing useful information. Then, the filtered IMF components are reconstructed into new signals and their features are extracted, Finally, support vector machines (SVM) are used to identify the rotor broken bars and air gap eccentric faults of the three-phase asynchronous motor. The experimental results show that the NA-MEMD method has a higher recognition rate than the traditional empirical mode decomposition (EMD) and the ensemble empirical mode decomposition (EEMD) methods.

Higher-order neurodynamical equation for simplex prediction

It is demonstrated that higher-order patterns beyond pairwise relations can significantly enhance the learning capability of existing graph-based models, and simplex is one of the primary form for graphically representing higher-order patterns. Predicting unknown (disappeared) simplices in real-world complex networks can provide us with deeper insights, thereby assisting us in making better decisions. Nevertheless, previous efforts to predict simplices suffer from two issues: (i) they mainly focus on 2- or 3-simplices, and there are few models available for predicting simplices of arbitrary orders, and (ii) they lack the ability to analyze and learn the features of simplices from the perspective of dynamics. In this paper, we present a Higher-order Neurodynamical Equation for Simplex Prediction of arbitrary order (HNESP), which is a framework that combines neural networks and neurodynamics. Specifically, HNESP simulates the dynamical coupling process of nodes in simplicial complexes through different relations (i.e., strong pairwise relation, weak pairwise relation, and simplex) to learn node-level representations, while explaining the learning mechanism of neural networks from neurodynamics. To enrich the higher-order information contained in simplices, we exploit the entropy and normalized multivariate mutual information of different sub-structures of simplices to acquire simplex-level representations. Furthermore, simplex-level representations and multi-layer perceptron are used to quantify the existence probability of simplices. The effectiveness of HNESP is demonstrated by extensive simulations on seven higher-order benchmarks. Experimental results show that HNESP improves the AUC values of the state-of-the-art baselines by an average of 8.32%. Our implementations will be publicly available at: https://github.com/jianruichen/HNESP.

Abstract A107: MAPK signaling is a conserved mechanism explaining phenotypic heterogeneity across diverse drivers of epithelial-mesenchymal transition in pancreas cancer

Abstract Epithelial-mesenchymal transition (EMT) is a developmental process that is aberrantly activated in cancers such as pancreatic ductal adenocarcinoma (PDAC) to promote disease progression and chemoresistance. EMT occurs heterogeneously among PDAC ductal cells, which frustrates efforts to identify the underlying signaling regulatory mechanisms. We hypothesize that cell-to-cell variation in the activities of specific signaling pathways explains EMT heterogeneity. To test this, we developed a workflow integrating: 1) iterative indirect immunofluorescence imaging (4i) to quantify the activities of up to seven purported EMT-regulating pathways with single-cell resolution and 2) a multivariate mutual information computational model to predict the pathways that are most informative of the EMT phenotype. In cultured PDAC cells, EMT was induced by treating cells with growth factors present in the tumor microenvironment (hepatocyte growth factor, epidermal growth factor, transforming growth factor β1, or two combinations) or with chemotherapeutics (gemcitabine or 5-fluorouracil). EMT phenotypic markers and signaling pathway proxies were assessed daily for up to 96 hr. A mutual information analysis of the aggregated data identified ERK as the most important signaling node explaining EMT heterogeneity across the diverse set of agonists investigated. Kinase inhibition and siRNA studies confirmed the implied role of ERK but also revealed a compensatory, EMT-driving JNK activation in the context of MEK inhibition that explains a cooperative effect of simultaneous MEK and JNK inhibition on EMT antagonism. Importantly, a model based on cell population-level measurements nominated JNK as more informative than ERK for determining EMT state in response to growth factors. We are implementing this approach in vivo using pancreas cancer tissue sections to determine generalizability of these findings in different microenvironmental contexts. Overall, our results demonstrate the pervasive importance of ERK for diverse drivers of EMT and the need to account for the effects of heterogeneous signaling pathway activation in the determination of EMT. Citation Format: Michelle C. Barbeau, Matthew J. Lazzara. MAPK signaling is a conserved mechanism explaining phenotypic heterogeneity across diverse drivers of epithelial-mesenchymal transition in pancreas cancer [abstract]. In: Proceedings of the AACR Special Conference in Cancer Research: Pancreatic Cancer; 2023 Sep 27-30; Boston, Massachusetts. Philadelphia (PA): AACR; Cancer Res 2024;84(2 Suppl):Abstract nr A107.

Path Planning for Adaptive CSI Map Construction With A3C in Dynamic Environments

The fingerprint localization based Channel State Information (CSI) plays a vital role given the popularity of Location-Based Service. Since its easy implementation, low device cost and CSI provides fine-grained information which can achieve adequate accuracy. However, the main drawback is that the approach has to construct the fingerprint map manually during the off-line stage, which is tedious and time-consuming. In this paper, we propose a novel data collection strategy for path planning based on reinforcement learning, namely Asynchronous Advantage Actor-Critic (A3C). Given the limited exploration step length, it needs to maximize the informative CSI data for reducing manual cost. We collect a small amount of real data in advance to predict the rewards of all sampling points by multivariate Gaussian process and mutual information. Then the optimization problem is transformed to a sequential decision process, which can exploit the informative path by A3C. We complete the proposed algorithm in two real-world dynamic environments and extensive experiments verify its performance. Compared to coverage path planning, our system not only can achieve similar indoor localization accuracy, but also reduce the CSI collection task.

JoMIC: A joint MI-based filter feature selection method

Feature selection (FS) is a common preprocessing step of machine learning that selects informative subset of features which fuels a model to perform better during prediction or classification. It helps in the design of an intelligent and expert system used in computer vision, image processing, gene expression data analysis, intrusion detection and natural language processing. In this paper, we introduce an effective filter method called Joint Mutual Information with Class relevance (JoMIC) using multivariate Joint Mutual Information (JMI) and Mutual Information (MI). Our method considers both JMI and MI of a non selected feature with selected ones w.r.t a given class to select a feature that is highly relevant to the class but non redundant to other selected features. We compare our method with seven other filter-based methods using the machine learning classifiers viz., Logistic Regression, Support Vector Machine, K-nearest Neighbor (KNN), Decision Tree, Random Forest, Naïve Bayes, and Stochastic Gradient Descent on various datasets. Experimental results reveal that our method yields better performance in terms of accuracy, Matthew’s Correlation Coefficient (MCC) and F1-score over 16 benchmark datasets, as compared to other competent methods. The superiority of our proposed method is that it uses an effective objective function that combines both JMI and MI to choose the relevant and non redundant features.

Prediction of protein-protein interactions based on ensemble residual convolutional neural network

With the development of bio-medical big data, the prediction of protein-protein interactions (PPIs) with the help of deep learning (DL) has attracted much attention for the study of intermolecular mechanism, drug design, human disease treatment. Given that the experiment-based methods can be difficult, reliable and DL-based approaches are needed. In this paper, we develop the EResCNN, an effective predictor to predict PPIs based on ensemble residual convolutional neural network. First, the fused feature representation is captured by concatenating the vectors obtained via pseudo amino acid composition (PseAAC), auto covariance descriptor (AC), pseudo position-specific scoring matrix (PsePSSM), encoding based on grouped weight (EBGW), multivariate mutual information (MMI) and conjoint triad (CT). Then the high-level information can be obtained using the convolution and pooling of the residual convolutional neural network (RCNN) via layer-by-layer learning. At last, we ensemble RCNN, XGBoost, random forest, LightGBM and extremely randomized trees to build the EResCNN model. The predictive results indicate that EResCNN achieves better performance with the ACC values of 95.34%, 87.89% and 98.61% on S. cerevisiae, H. pylori and Human-Y. pestis datasets, respectively. We also apply EResCNN to the datasets of H. sapiens, M. musculus, C. elegans and E. coli for cross-species prediction. Especially, we find EResCNN can infer significant PPIs network on one-core network, Wnt-related signal pathway network, cancer-specific network and multi-core network, which could provide some references for signal pathway research, disease-related gene mining, and interaction network topology.

MARPPI: boosting prediction of protein-protein interactions with multi-scale architecture residual network.

Protein-protein interactions (PPIs) are a major component of the cellular biochemical reaction network. Rich sequence information and machine learning techniques reduce the dependence of exploring PPIs on wet experiments, which are costly and time-consuming. This paper proposes a PPI prediction model, multi-scale architecture residual network for PPIs (MARPPI), based on dual-channel and multi-feature. Multi-feature leverages Res2vec to obtain the association information between residues, and utilizes pseudo amino acid composition, autocorrelation descriptors and multivariate mutual information to achieve the amino acid composition and order information, physicochemical properties and information entropy, respectively. Dual channel utilizes multi-scale architecture improved ResNet network which extracts protein sequence features to reduce protein feature loss. Compared with other advanced methods, MARPPI achieves 96.03%, 99.01% and 91.80% accuracy in the intraspecific datasets of Saccharomyces cerevisiae, Human and Helicobacter pylori, respectively. The accuracy on the two interspecific datasets of Human-Bacillus anthracis and Human-Yersinia pestis is 97.29%, and 95.30%, respectively. In addition, results on specific datasets of disease (neurodegenerative and metabolic disorders) demonstrate the ability to detect hidden interactions. To better illustrate the performance of MARPPI, evaluations on independent datasets and PPIs network suggest that MARPPI can be used to predict cross-species interactions. The above shows that MARPPI can be regarded as a concise, efficient and accurate tool for PPI datasets.

Interaction networks from discrete event data by Poisson multivariate mutual information estimation and information flow with applications from gene expression data

In this work, we introduce a new methodology for inferring the interaction structure of discrete valued time series which are Poisson distributed. While most related methods are premised on continuous state stochastic processes, in fact, discrete and counting event oriented stochastic process are natural and common, so called time-point processes. An important application that we focus on here is gene expression, where it is often assumed that the data is generated from a multivariate Poisson distribution. Nonparameteric methods such as the popular k-nearest neighbors are slow converging for discrete processes, and thus data hungry. Now, with the new multi-variate Poisson estimator developed here as the core computational engine, the causation entropy (CSE) principle, together with the associated greedy search algorithm optimal CSE (oCSE) allows us to efficiently infer the true network structure for this class of stochastic processes that were previously not practical. We illustrate the power of our method, first in benchmarking with synthetic datum, and then by inferring the genetic factors network from a breast cancer micro-ribonucleic acid sequence count data set. We show the Poisson oCSE gives the best performance among the tested methods and discovers previously known interactions on the breast cancer data set.

Day-ahead inflow forecasting using causal empirical decomposition

It is essential to have accurate and reliable daily-inflow forecasting to improve short-term hydropower scheduling. This paper proposes a Causal multivariate Empirical mode Decomposition (CED) framework as a complementary pre-processing step for a day-ahead inflow forecasting problem. The idea behind CED is combining physics-based causal inference with signal processing-based decomposition to get the most relevant features among multiple time-series to the inflow values. The CED framework is validated for two areas in Norway with different meteorological and hydrological conditions. The validation results show that using CED as a pre-processing step significantly enhances (up to 70%) the forecasting accuracy for various state-of-the-art forecasting methods. • Pre-processing step is essential for daily inflow forecasting. • A combination of causal-based inference network with decomposition technique is proposed as CED framework. • Multivariate empirical mode decomposition, transfer entropy and conditional mutual information were used in CED framework. • CED framework improved daily inflow forecasting by 70% compared with no preprocessing scenario.

DEEPStack-RBP: Accurate identification of RNA-binding proteins based on autoencoder feature selection and deep stacking ensemble classifier

RNA-binding proteins (RBPs) are involved in a number of biological processes such as RNA synthesis, protein folding, alternative splicing, etc. Predicting RBPs can facilitate the discovery and treatment of human diseases, such as muscle atrophy, nervous system diseases, and cancer. However, there are still various challenges in identifying RBPs using experimental methods. Computational methods, and in particular Deep Learning, are being deployed to alleviate some of these challenges and provide new avenues of investigation in the field of RBPs prediction. Here, we propose DEEPStack-RBP, a novel RBPs prediction tool based on deep learning and ensemble learning. First, conjoint triad (CT), local descriptors (LD), pseudo amino acid composition (PseAAC), multivariate mutual information (MMI) and position specific scoring matrix-transition probability composition (PSSM-TPC) are applied to extract multiple features from the proteins. Subsequently, autoencoder (AE) is used to eliminate redundancy in features, and SMOTE-ENN is employed to balance the samples by minimizing the number difference between positive and negative cases. Finally, the stacked ensemble classifier composed of bidirectional long short-term memory (BiLSTM), gated recurrent unit (GRU), and support vector machine (SVM) is used for prediction. On the training dataset RBP9873, the ACC value of DEEPStack-RBP reaches 98.76% with a MCC value of 0.9508. For the three independent test datasets of Human, S. cerevisiae and A. thaliana, the accuracy of the model is 97.16%, 97.67% and 99.57% respectively, and the MCC is 0.9405, 0.9499 and 0.9906 respectively. These results show that DEEPStack-RBP can be used as a powerful tool for RBPs prediction.

Performance of multivariate mutual information and autocorrelation encoding methods for the prediction of protein-protein interactions

<span lang="EN-US">Protein interactions play an essential role in the study of how an organism can be infected with a disease and also its effects. One of the challenges in computational methods in the prediction of protein-protein interactions is how to represent a sequence of amino acids in a vector so that it can be used in machine learning to create a model that can predict whether or not an interaction occurs in a protein pair. This paper examined the qualitative feature encoding methods of amino acid sequence, namely, multivariate mutual information (MMI), and the quantitative feature encoding methods, namely, autocorrelation. We develop the new design for MMI and autocorrelation feature encoding methods which give better results than the previous research. There are four ways to build the MMI method and six ways to build the autocorrelation method that we tested. We also built four types of MMI-autocorrelation (mixed) method and look for the best form of each type of MMI, autocorrelation, and mixed-method. We combine these feature encoding methods with support vector machine (SVM) as machine learning methods. We also test the encoding methods we propose to several machine learning classifier methods, such as random forest (RF), k-nearest neighbor (KNN), and gradient boosting.</span>

Multivariate Gaussian Copula Mutual Information to Estimate Functional Connectivity with Less Random Architecture.

Recognition of a brain region’s interaction is an important field in neuroscience. Most studies use the Pearson correlation to find the interaction between the regions. According to the experimental evidence, there is a nonlinear dependence between the activities of different brain regions that is ignored by Pearson correlation as a linear measure. Typically, the average activity of each region is used as input because it is a univariate measure. This dimensional reduction, i.e., averaging, leads to a loss of spatial information across voxels within the region. In this study, we propose using an information-theoretic measure, multivariate mutual information (mvMI), as a nonlinear dependence to find the interaction between regions. This measure, which has been recently proposed, simplifies the mutual information calculation complexity using the Gaussian copula. Using simulated data, we show that the using this measure overcomes the mentioned limitations. Additionally using the real resting-state fMRI data, we compare the level of significance and randomness of graphs constructed using different methods. Our results indicate that the proposed method estimates the functional connectivity more significantly and leads to a smaller number of random connections than the common measure, Pearson correlation. Moreover, we find that the similarity of the estimated functional networks of the individuals is higher when the proposed method is used.

PET Imaging of Tau Pathology and Amyloid-β, and MRI for Alzheimer's Disease Feature Fusion and Multimodal Classification.

Machine learning is a promising tool for biomarker-based diagnosis of Alzheimer's disease (AD). Performing multimodal feature selection and studying the interaction between biological and clinical AD can help to improve the performance of the diagnosis models. This study aims to formulate a feature ranking metric based on the mutual information index to assess the relevance and redundancy of regional biomarkers and improve the AD classification accuracy. From the Alzheimer's Disease Neuroimaging Initiative (ADNI), 722 participants with three modalities, including florbetapir-PET, flortaucipir-PET, and MRI, were studied. The multivariate mutual information metric was utilized to capture the redundancy and complementarity of the predictors and develop a feature ranking approach. This was followed by evaluating the capability of single-modal and multimodal biomarkers in predicting the cognitive stage. Although amyloid-β deposition is an earlier event in the disease trajectory, tau PET with feature selection yielded a higher early-stage classification F1-score (65.4%) compared to amyloid-β PET (63.3%) and MRI (63.2%). The SVC multimodal scenario with feature selection improved the F1-score to 70.0% and 71.8% for the early and late-stage, respectively. When age and risk factors were included, the scores improved by 2 to 4%. The Amyloid-Tau-Neurodegeneration [AT(N)] framework helped to interpret the classification results for different biomarker categories. The results underscore the utility of a novel feature selection approach to reduce the dimensionality of multimodal datasets and enhance model performance. The AT(N) biomarker framework can help to explore the misclassified cases by revealing the relationship between neuropathological biomarkers and cognition.

Alleviating the independence assumptions of averaged one-dependence estimators by model weighting

Of numerous proposals to refine naive Bayes by weakening its attribute independence assumption, averaged one-dependence estimators (AODE) has been shown to be able to achieve significantly higher classification accuracy at a moderate cost in classification efficiency. However, all one-dependence estimators (ODEs) in AODE have the same weights and are treated equally. To address this issue, model weighting, which assigns discriminate weights to ODEs and then linearly combine their probability estimates, has been proved to be an efficient and effective approach. Most information-theoretic weighting metrics, including mutual information, Kullback-Leibler measure and the information gain, place more emphasis on the correlation between root attribute (value) and class variable. We argue that the topology of each ODE can be divided into a set of local directed acyclic graphs (DAGs) based on the independence assumption, and multivariate mutual information is introduced to measure the extent to which the DAGs fit data. Based on this premise, in this study we propose a novel weighted AODE algorithm, called AWODE, that adaptively selects weights to alleviate the independence assumption and make the learned probability distribution fit the instance. The proposed approach is validated on 40 benchmark datasets from UCI machine learning repository. The experimental results reveal that, AWODE achieves bias-variance trade-off and is a competitive alternative to single-model Bayesian learners (such as TAN and KDB) and other weighted AODEs (such as WAODE).

Explainable fault diagnosis of gas-liquid separator based on fully convolutional neural network

The diagnosis of slug flow is extremely important for the efficient operation of the gas-liquid separator. Traditional fault diagnosis based on the convolutional neural network has not involved the explainability of the convolutional neural network, which makes the model difficult to interpret from the perspective of physical meaning. An explainable diagnostic method based on a fully convolutional neural network is proposed. The class activation mapping, multivariate mutual information, global average pooling and t-distributed stochastic neighbor embedding are combined to analyze the diagnostic process of the network. The experimental results based on simulation data showed that the proposed method can be utilized to interpret the correlation degree between different operating conditions, the importance of each period in the measurement variable, and the engineering significance of the convolutional kernels of the last layer, which provides information supplement for fault reasoning of human experts.

Bits and pieces: understanding information decomposition from part-whole relationships and formal logic.

Partial information decomposition (PID) seeks to decompose the multivariate mutual information that a set of source variables contains about a target variable into basic pieces, the so-called ‘atoms of information’. Each atom describes a distinct way in which the sources may contain information about the target. For instance, some information may be contained uniquely in a particular source, some information may be shared by multiple sources and some information may only become accessible synergistically if multiple sources are combined. In this paper, we show that the entire theory of PID can be derived, firstly, from considerations of part-whole relationships between information atoms and mutual information terms, and secondly, based on a hierarchy of logical constraints describing how a given information atom can be accessed. In this way, the idea of a PID is developed on the basis of two of the most elementary relationships in nature: the part-whole relationship and the relation of logical implication. This unifying perspective provides insights into pressing questions in the field such as the possibility of constructing a PID based on concepts other than redundant information in the general n-sources case. Additionally, it admits of a particularly accessible exposition of PID theory.

Learning Continuous High-Dimensional Models using Mutual Information and Copula Bayesian Networks

We propose a new framework to learn non-parametric graphical models from continuous observational data. Our method is based on concepts from information theory in order to discover independences and causality between variables: the conditional and multivariate mutual information (such as \cite{verny2017learning} for discrete models). To estimate these quantities, we propose non-parametric estimators relying on the Bernstein copula and that are constructed by exploiting the relation between the mutual information and the copula entropy \cite{ma2011mutual, belalia2017testing}. To our knowledge, this relation is only documented for the bivariate case and, for the need of our algorithms, is here extended to the conditional and multivariate mutual information. This framework leads to a new algorithm to learn continuous non-parametric Bayesian network. Moreover, we use this estimator to speed up the BIC algorithm proposed in \cite{elidan2010copula} by taking advantage of the decomposition of the likelihood function in a sum of mutual information \cite{koller2009probabilistic}. Finally, our method is compared in terms of performances and complexity with other state of the art techniques to learn Copula Bayesian Networks and shows superior results. In particular, it needs less data to recover the true structure and generalizes better on data that are not sampled from Gaussian distributions.

Introducing a differentiable measure of pointwise shared information.

Partial information decomposition of the multivariate mutual information describes the distinct ways in which a set of source variables contains information about a target variable. The groundbreaking work of Williams and Beer has shown that this decomposition cannot be determined from classic information theory without making additional assumptions, and several candidate measures have been proposed, often drawing on principles from related fields such as decision theory. None of these measures is differentiable with respect to the underlying probability mass function. We here present a measure that satisfies this property, emerges solely from information-theoretic principles, and has the form of a local mutual information. We show how the measure can be understood from the perspective of exclusions of probability mass, a principle that is foundational to the original definition of mutual information by Fano. Since our measure is well defined for individual realizations of random variables it lends itself, for example, to local learning in artificial neural networks. We also show that it has a meaningful Möbius inversion on a redundancy lattice and obeys a target chain rule. We give an operational interpretation of the measure based on the decisions that an agent should take if given only the shared information.

Prediction of protein–protein interactions based on elastic net and deep forest

Prediction of protein–protein interactions (PPIs) helps to grasp molecular roots of disease. However, web-lab experiments to predict PPIs are limited and costly. Using machine-learning-based frameworks can not only automatically identify PPIs, but also provide new ideas for drug research and development from a promising alternative. We present a novel deep-forest-based method for PPIs prediction. Firstly, pseudo amino acid composition (PAAC), autocorrelation descriptor (Auto), multivariate mutual information (MMI), composition-transition-distribution (CTD), amino acid composition position-specific scoring matrix (AAC-PSSM), and dipeptide composition PSSM (DPC-PSSM) are adopted to extract and construct the pattern of PPIs. Secondly, elastic net is utilized to optimize the initial feature vectors and boost the predictive performance. Finally, we ensemble XGBoost, random forest, and extremely randomized trees to construct deep forest model via cascade architecture for PPIs prediction (GcForest-PPI). Benchmark experiments reveal that the proposed approach outperforms other state-of-the-art predictors on Saccharomyces cerevisiae and Helicobacter pylori. We also apply GcForest-PPI on independent test sets, CD9-core network, crossover network, and cancer-specific network. The evaluation shows that GcForest-PPI can boost the prediction accuracy, complement experiments and improve drug discovery.

Ensemble Linear Subspace Analysis of High-Dimensional Data.

Regression models provide prediction frameworks for multivariate mutual information analysis that uses information concepts when choosing covariates (also called features) that are important for analysis and prediction. We consider a high dimensional regression framework where the number of covariates (p) exceed the sample size (n). Recent work in high dimensional regression analysis has embraced an ensemble subspace approach that consists of selecting random subsets of covariates with fewer than p covariates, doing statistical analysis on each subset, and then merging the results from the subsets. We examine conditions under which penalty methods such as Lasso perform better when used in the ensemble approach by computing mean squared prediction errors for simulations and a real data example. Linear models with both random and fixed designs are considered. We examine two versions of penalty methods: one where the tuning parameter is selected by cross-validation; and one where the final predictor is a trimmed average of individual predictors corresponding to the members of a set of fixed tuning parameters. We find that the ensemble approach improves on penalty methods for several important real data and model scenarios. The improvement occurs when covariates are strongly associated with the response, when the complexity of the model is high. In such cases, the trimmed average version of ensemble Lasso is often the best predictor.