Mutual Information Research Articles

Bivariate Pareto–Feller Distribution Based on Appell Hypergeometric Function

The Pareto–Feller distribution has been widely used across various disciplines to model “heavy-tailed” phenomena, where extreme events such as high incomes or large losses are of interest. In this paper, we present a new bivariate distribution based on the Appell hypergeometric function with marginal Pareto–Feller distributions obtained from two independent gamma random variables. The proposed distribution has the beta prime marginal distributions as special case, which were obtained using a Kibble-type bivariate gamma distribution, and the stochastic representation was obtained by the quotient of a scale mixture of two gamma random variables. This result can be viewed as a generalization of the standard bivariate beta I (or inverted bivariate beta distribution). Moreover, the obtained bivariate density is based on two confluent hypergeometric functions. Then, we derive the probability distribution function, the cumulative distribution function, the moment-generating function, the characteristic function, the approximated differential entropy, and the approximated mutual information index. Based on numerical examples, the exact and approximated expressions are shown.

ASCL: Accelerating semi‐supervised learning via contrastive learning

SummarySSL (semi‐supervised learning) is widely used in machine learning, which leverages labeled and unlabeled data to improve model performance. SSL aims to optimize class mutual information, but noisy pseudo‐labels introduce false class information due to the scarcity of labels. Therefore, these algorithms often need significant training time to refine pseudo‐labels for performance improvement iteratively. To tackle this challenge, we propose a novel plug‐and‐play method named Accelerating semi‐supervised learning via contrastive learning (ASCL). This method combines contrastive learning with uncertainty‐based selection for performance improvement and accelerates the convergence of SSL algorithms. Contrastive learning initially emphasizes the mutual information between samples as a means to decrease dependence on pseudo‐labels. Subsequently, it gradually turns to maximizing the mutual information between classes, aligning with the objective of semi‐supervised learning. Uncertainty‐based selection provides a robust mechanism for acquiring pseudo‐labels. The combination of the contrastive learning module and the uncertainty‐based selection module forms a virtuous cycle to improve the performance of the proposed model. Extensive experiments demonstrate that ASCL outperforms state‐of‐the‐art methods in terms of both convergence efficiency and performance. In the experimental scenario where only one label is assigned per class in the CIFAR‐10 dataset, the application of ASCL to Pseudo‐label, UDA (unsupervised data augmentation for consistency training), and Fixmatch benefits substantial improvements in classification accuracy. Specifically, the results demonstrate notable improvements in respect of 16.32%, 6.9%, and 24.43% when compared to the original outcomes. Moreover, the required training time is reduced by almost 50%.

Recurrence resonance - noise-enhanced dynamics in recurrent neural networks

Understanding how neural networks process information is a fundamental challenge in neuroscience and artificial intelligence. A pivotal question in this context is how external stimuli, particularly noise, influence the dynamics and information flow within these networks. Traditionally, noise is perceived as a hindrance to information processing, introducing randomness and diminishing the fidelity of neural signals. However, distinguishing noise from structured input uncovers a paradoxical insight: under specific conditions, noise can actually enhance information processing. This intriguing possibility prompts a deeper investigation into the nuanced role of noise within neural networks. In specific motifs of three recurrently connected neurons with probabilistic response, the spontaneous information flux, defined as the mutual information between subsequent states, has been shown to increase by adding ongoing white noise of some optimal strength to each of the neurons. However, the precise conditions for and mechanisms of this phenomenon called ‘recurrence resonance’ (RR) remain largely unexplored. Using Boltzmann machines of different sizes and with various types of weight matrices, we show that RR can generally occur when a system has multiple dynamical attractors, but is trapped in one or a few of them. In probabilistic networks, the phenomenon is bound to a suitable observation time scale, as the system could autonomously access its entire attractor landscape even without the help of external noise, given enough time. Yet, even in large systems, where time scales for observing RR in the full network become too long, the resonance can still be detected in small subsets of neurons. Finally, we show that short noise pulses can be used to transfer recurrent neural networks, both probabilistic and deterministic, between their dynamical attractors. Our results are relevant to the fields of reservoir computing and neuroscience, where controlled noise may turn out a key factor for efficient information processing leading to more robust and adaptable systems.

Improving Alzheimer's Disease Prediction with Different Machine Learning Approaches and Feature Selection Techniques.

Machine learning (ML) has increasingly been utilized in healthcare to facilitate disease diagnosis and prediction. This study focuses on predicting Alzheimer's disease (AD) through the development and comparison of ML models using Support Vector Machine (SVM), Random Forest (RF), and Logistic Regression (LR) algorithms. Additionally, feature selection techniques including Minimum Redundancy Maximum Relevance (mRMR) and Mutual Information (MI) were employed to enhance the model performance. The research methodology involved training and testing these models on the OASIS-2 dataset, evaluating their predictive accuracies. Notably, LR combined with mRMR achieved the highest accuracy of 99.08% in predicting AD. These findings underscore the efficacy of ML algorithms in AD prediction and highlight the utility of the feature selection methods in improving the model performance. This study contributes to the ongoing efforts to leverage ML for more accurate disease prognosis and underscores the potential of these techniques in advancing clinical decision-making.

Transforming Motor Imagery Analysis: A Novel EEG Classification Framework Using AtSiftNet Method

This paper presents an innovative approach for the Feature Extraction method using Self-Attention, incorporating various Feature Selection techniques known as the AtSiftNet method to enhance the classification performance of motor imaginary activities using electrophotography (EEG) signals. Initially, the EEG signals were sorted and then denoised using multiscale principal component analysis to obtain clean EEG signals. However, we also conducted a non-denoised experiment. Subsequently, the clean EEG signals underwent the Self-Attention feature extraction method to compute the features of each trial (i.e., 350×18). The best 1 or 15 features were then extracted through eight different feature selection techniques. Finally, five different machine learning and neural network classification models were employed to calculate the accuracy, sensitivity, and specificity of this approach. The BCI competition III dataset IV-a was utilized for all experiments, encompassing the datasets of five volunteers who participated in the competition. The experiment findings reveal that the average accuracy of classification is highest among ReliefF (i.e., 99.946%), Mutual Information (i.e., 98.902%), Independent Component Analysis (i.e., 99.62%), and Principal Component Analysis (i.e., 98.884%) for both 1 and 15 best-selected features from each trial. These accuracies were obtained for motor imagery using a Support Vector Machine (SVM) as a classifier. In addition, five-fold validation was performed in this paper to assess the fair performance estimation and robustness of the model. The average accuracy obtained through five-fold validation is 99.89%. The experiments’ findings indicate that the suggested framework provides a resilient biomarker with minimal computational complexity, making it a suitable choice for advancing Motor Imagery Brain–Computer Interfaces (BCI).

A proposed technique for predicting heart disease using machine learning algorithms and an explainable AI method

One of the critical issues in medical data analysis is accurately predicting a patient’s risk of heart disease, which is vital for early intervention and reducing mortality rates. Early detection allows for timely treatment and continuous monitoring by healthcare providers, which is essential but often limited by the inability of medical professionals to provide constant patient supervision. Early detection of cardiac problems and continuous patient monitoring by physicians can help reduce death rates. Doctors cannot constantly have contact with patients, and heart disease detection is not always accurate. By offering a more solid foundation for prediction and decision-making based on data provided by healthcare sectors worldwide, machine learning (ML) could help physicians with the prediction and detection of HD. This study aims to use different feature selection strategies to produce an accurate ML algorithm for early heart disease prediction. We have chosen features using chi-square, ANOVA, and mutual information methods. The three feature groups chosen were SF-1, SF-2, and SF-3. The study employed ten machine learning algorithms to determine the most accurate technique and feature subset fit. The classification algorithms used include support vector machines (SVM), XGBoost, bagging, decision trees (DT), and random forests (RF). We evaluated the proposed heart disease prediction technique using a private dataset, a public dataset, and different cross-validation methods. We used the Synthetic Minority Oversampling Technique (SMOTE) to eliminate inconsistent data and discover the machine learning algorithm that achieves the most accurate heart disease predictions. Healthcare providers might identify early-stage heart disease quickly and cheaply with the proposed method. We have used the most effective ML algorithm to create a mobile app that instantly predicts heart disease based on the input symptoms. The experimental results demonstrated that the XGBoost algorithm performed optimally when applied to the combined datasets and the SF-2 feature subset. It had 97.57% accuracy, 96.61% sensitivity, 90.48% specificity, 95.00% precision, a 92.68% F1 score, and a 98% AUC. We have developed an explainable AI method based on SHAP approaches to understand how the system makes its final predictions.

The art of selecting the ECG input in neural networks to classify heart diseases: a dual focus on maximizing information and reducing redundancy.

Accurate diagnosis of cardiovascular diseases often relies on the electrocardiogram (ECG). Since the cardiac vector is located within a three-dimensional space and the standard ECG comprises 12 projections or leads derived from it, redundant information is inherently present. This study aims to quantify this redundancy and its impact on classification tasks using Convolutional Neural Networks (CNNs) in cardiovascular diseases. We employed signal theory and mutual information to introduce a novel redundancy metric and explored techniques for redundancy augmentation and reduction. This involved lead selection and transformation to evaluate the effects on neural network performance. Our findings indicate that optimizing input configurations through redundancy reduction techniques can enhance the performance of deep learning models in cardiovascular diagnostics, provided that the information is preserved and minimally distorted. For the first time, this research has quantified the redundancy present in the input by validating various redundancy reduction techniques using a CNN. This discovery paves the way for advancing biomedical signal processing research, simplifying model complexity, and enhancing diagnostic performance in cardiovascular medicine within reduced lead systems, such as Holter monitors or wearables.

Robustness estimation for state-of-charge of a lithium-ion battery based on feature fusion

State of charge (SOC) of the lithium-ion battery stands as one of the crucial parameters of the battery management system. To enhance the robustness and accuracy of data-driven methods for estimating SOC, in this study, a novel framework based on feature selection and closed-loop estimation framework is proposed for estimating the SOC of a battery. Firstly, the maximum mutual information coefficient is used to correlate the data collected from the battery experiments. Secondly, an improved least squares support vector machine model (LSSVM) is proposed for SOC estimation using the sine cosine optimization algorithm for parameter optimization of LSSVM. Furthermore, based on the Sage-Husa adaptive Kalman filter, this study constructed a closed-loop estimation model. Finally, we conduct experiments on batteries with variable temperatures and working conditions, to verify its robustness and generalization. The results indicate that the proposed method has great SOC estimation accuracy compared to the control group. Even under simulated measurement noise conditions, the proposed method achieves the root mean square error 1.28 % and 1.12 % respectively, demonstrating strong robustness.

Sinkhorn divergence-based contrast domain adaptation for remaining useful life prediction of rolling bearings under multiple operating conditions

Under multiple operating conditions, the degradation characteristics of rolling bearings show diverse distributions. Domain adaptation (DA) achieves effective alignment between source and target domains by extracting domain-invariant features. However, in the prediction of remaining useful life (RUL) for bearings, numerous DA methods overlook mutual information from target-specific data and encounter potential challenges such as the vanishing gradient problem during the alignment of data distributions, leading to limited performance. To address these challenges, a novel method called Sinkhorn Divergence-based Contrast Domain Adaptation (SD_CDA) is proposed to predict RUL under multiple operating conditions. Firstly, an adversarial training framework is constructed to initially extract domain-invariant features. Subsequently, the cross-domain temporal mixup strategy is proposed for the data augment, which obtains positive samples to serve contrastive learning. Then self-supervised momentum contrast (MoCo) is employed to extract mutual information from target-specific data, preserving its specificity. Finally, Sinkhorn divergence is introduced to further align the fine-grained structure of the source domain and target domain, and enhance the transfer ability of the model. The experimental results demonstrate the superiority and effectiveness of the proposed method under multiple operating conditions.

Graph Convolution Network Combining Spatiotemporal Self-Attention and Mutual Information for Pedestrian-Trajectory Prediction

Accurate pedestrian-trajectory prediction is significant and challenging for intelligent driving. There are two challenges in this field: most existing methods ignore pedestrian–vehicle interaction in the multiple pedestrian and vehicle scenario; and research on interaction modes and objects during the interaction process remains relatively limited in depth and scope. To tackle these challenges, a novel graph convolution network combining spatiotemporal self-attention and mutual information (STSAMI-GCN) is designed for predicting pedestrian trajectories. First, to model the spatiotemporal interaction between pedestrians and vehicles, we build a spatiotemporal graph interaction module that integrates graph convolutional networks and self-attention mechanisms. Second, to filter redundant interactions and optimize feature extraction in the spatiotemporal graph interaction module, a mutual information graph network module is designed as an auxiliary network, which can reduce information redundancy and simplify graph complexity. In addition, to address the problem of propagation and accumulation of prediction errors, we design a trajectory decoder to generate future trajectories. Finally, a suitable dataset for our research scenario is built to validate the proposed model. We also evaluate it on two publicly available datasets. Experimental results show that, in the multiple pedestrian and vehicle scenario, STSAMI-GCN can be used to predict the future trajectories of multiple pedestrians, and to predict socially acceptable future trajectories. Specifically, STSAMI-GCN achieves 0.25 and 0.27 average and final deviation error, respectively, and outperforms various mainstream methods. Furthermore, ablation studies demonstrate the effectiveness of the spatiotemporal dual self-attention mechanism and mutual information graph network in improving pedestrian-trajectory prediction accuracy.

Scrambling power of soft photons

Observable scattering processes entail emission-absorption of soft photons. As these degrees of freedom go undetected, some information is lost. Whether some of this information can be recovered in the observation of the hard particles, depends of the actual pattern of the scrambling of information. We compute the information scrambling of electron and soft photon scattering by the tripartite mutual information in terms of the 2-Renyi entropy, and find a finite amount of scrambling is present. We show that scrambling is a byproduct of decoherence achieved by the scattering system in its interaction with the environment, due to the emission-absorption of soft photons in fully unitary processes.

Incremental robust predictive control for anti-pitching in catamarans with appendage constraints

To solve the problem that high-speed catamarans have excessive amplitude of vertical motion in rough sea conditions, an anti-pitching method based on incremental robust predictive control of catamarans is proposed, which takes into account the model uncertainty and input rate of appendage requirements. Thus, a high-speed catamaran vertical control model is established with T-foils and flaps serving as anti-pitching appendages, and the hydrodynamic coefficient uncertainties is analyzed, which is converted into a state space model with norm-bounded uncertainty to facilitate control system design; moreover, transforming the norm-bounded uncertainty model of a high-speed catamaran into an incremental model, which can increase the integral action to improve the anti-pitching performance. On this basis, the hybrid H2/H∞ robust predictive control method is proposed to achieve wave disturbance resistance capability and robust stability of the catamaran system, taking the rate of input of the anti-pitching appendages as the input constraints, and the linear matrix inequality (LMI) receding-horizon optimization is used to solve the incremental control input of appendages. Ultimately, the effectiveness of the proposed algorithm is verified by digital simulation.

Machine learning based classification of spontaneous intracranial hemorrhages using radiomics features.

To assess the efficacy of radiomics features extracted from non-contrast computed tomography (NCCT) scans in differentiating multiple etiologies of spontaneous intracerebral hemorrhage (ICH). CT images and clinical data from 141 ICH patients from 2010 to 2022 were collected. The cohort comprised primary (n = 57), tumorous (n = 46), and vascular malformation-related ICH (n = 38). Radiomics features were extracted from the initial brain NCCT scans and identified potential features using mutual information. A hierarchical classification with AdaBoost classifiers was employed to classify the multiple etiologies of ICH. Age of the patient and ICH's location were examined alongside radiomics features. The accuracy, area under the curve (AUC), sensitivity, and specificity were used to evaluate classification performance. The proposed method achieved an accuracy of 0.79. For identifying primary ICH, the model achieved a sensitivity of 0.86 and specificity of 0.87. Meanwhile, the sensitivity and specificity for identifying tumoral causes were 0.78 and 0.93, respectively. For vascular malformation, the model reached a sensitivity and specificity of 0.72 and 0.89, respectively. The AUCs for primary, tumorous, and vascular malformation were 0.86, 0.85, and 0.82, respectively. The findings further highlight the importance of texture-based variables in ICH classification. The age and location of the ICH can enhance the classification performance. The use of a machine learning model with radiomics features has the potential in classifying the three types of non-traumatic ICH. It may help the radiologist decide on an appropriate further examination plan to arrive at a correct diagnosis.

Prediction on seismic performance levels of reinforced concrete beams by considering crack development

The plastic rotation angle or deflection is typically used as the indicator to evaluate the earthquake-induced damage and classify the seismic performance levels of reinforced concrete (RC) beams. Herein, one of the most important issues is to determine the seismic performance level limits of RC beams. Whereas, different countries provided their specific method to determine the limit values by considering the loading-carrying capacity of RC beams, which could not be used to describe the earthquake-induced seepage of structures, especially for underground structures. Therefore, in this study, the seismic performance level limits of RC beams were predicted by using the machine learning methods and considering the development of cracks. Firstly, the seismic performance level limits of RC beams were presented after discussing the methods in different codes and the development of cracks. Then an earthquake performance test database of RC beams was established after collecting 452 test results of RC beams, and Pearson correlation analysis was conducted for feature selection to determine the input mechanical parameters and dimensional parameters for machine learning. Meanwhile, the correlation between the inputs and limit values was analyzed using the mutual information method. Regression models of seven machine learning methods were then established to predict the performance level limits of RC beams, and the hyperparameters of the machine learning models were optimized with the TPE optimization algorithm and cross-validation. The generalization ability of the prediction models was evaluated and the accuracy of predicted results by different methods was analyzed. Finally, the predicted seismic performance level limits of RC beams could be used to evaluate the earthquake-induced damage of RC beams by combining them with the seismic behavior of RC beams.

NetSci: A Library for High Performance Biomolecular Simulation Network Analysis Computation.

We present the NetSci program-an open-source scientific software package designed for estimating mutual information (MI) between data sets using GPU acceleration and a k-nearest-neighbor algorithm. This approach significantly enhances calculation speed, achieving improvements of several orders of magnitude over traditional CPU-based methods, with data set size limits dictated only by available hardware. To validate NetSci, we accurately compute MI for an analytically verifiable two-dimensional Gaussian distribution and replicate the generalized correlation (GC) analysis previously conducted on the B1 domain of protein G. We also apply NetSci to molecular dynamics simulations of the Sarcoendoplasmic Reticulum Calcium-ATPase (SERCA) pump, exploring the allosteric mechanisms and pathways influenced by ATP and 2'-deoxy-ATP (dATP) binding. Our analysis reveals distinct allosteric effects induced by ATP compared to dATP, with predicted information pathways from the bound nucleotide to the calcium-binding domain differing based on the nucleotide involved. NetSci proves to be a valuable tool for estimating MI and GC in various data sets and is particularly effective for analyzing intraprotein communication and information transfer.

Texture-Based Classification of Fetal Growth Restriction From Intrauterine Neurosonographic Image.

Fetal growth restriction (FGR) is a condition where fetuses fail to reach their genetic potential for growth, posing a significant health challenge for newborns. The aim of this research was to explore the efficacy of texture-based analysis of neurosonographic images in identifying FGR in fetuses, which may provide a promising tool for early assessment of FGR. A retrospective analysis collected 100 intrauterine neurosonographic images from 50 FGR and 50 gestational age-appropriate fetuses. Using MaZda software, approximately 300 texture features were extracted from occipital white matter (OWM) and cerebellum of intrauterine neurosonographic images, respectively. Then 10 optimal features were separately selected by 3 algorithms, including the Fisher coefficient method, the method of minimizing classification error probability and average correlation coefficients, and the mutual information coefficient method. Further, the 10 statistically most significant features were selected from these sets to form the mixed feature set. After nonlinear discriminant analysis was performed to reduce feature dimensionality, the artificial neural network (ANN) classifier was conducted, respectively. For OWM and cerebellum, a total of 11 and 14 statistically significant features were selected. When the mixed feature sets of OWM and cerebellum were applied to ANN classifier, classification accuracy were 90.00% (κ = 0.800; P < .001) and 93.00% (κ = 0.860; P < .001), and the receiver operating characteristic curve for identifying FGR showed an area under the curve of 0.82 and 0.87. Texture analysis of fetal intrauterine neurosonographic images is a feasible and noninvasive strategy for evaluating FGR fetuses.

Linear fractional transportation problem in bipolar fuzzy environment

This article proposes a solution methodology for the Linear Fractional Transportation Problem (LFTP) by incorporating bipolar fuzzy sets (BFSs) to accommodate both positive and negative judgmental perspectives. The approach explores Zimmermann's extension within bipolar fuzzy environment to compare outcomes. In this context, the cost function and constraint coefficients are depicted using interval-valued trapezoidal bipolar fuzzy numbers (IVTrBFNs) and defuzzified by (s, t)-cut. The initial approach employs the simplex method and fuzzy optimization method, renowned for its effectiveness in obtaining optimal solutions. In the alternative method, Bipolar fuzzy programming approach (BFPA) is utilized for better outcome. In this method the LFTP is altered to a Multi-Objective Transportation Problem (MOTP), and BFPA extends Zimmermann's technique under suitable positive and negative membership functions, converting MOTP to a one-objective transportation problem (TP) and solved using LINGO software. Supporting this proposed method some theorems are formulated to demonstrate that the most effective solution of the single-objective TP is a Pareto optimal solution for the corresponding MOTP. A quantitative example is provided for better understanding of the proposed BFPA method alongside two other approaches.

Information theoretic measures for Lifshitz system

In this work, we have studied various mixed state information theoretic quantities for an excited state of Lifshitz spacetime in 3 + 1-dimensions. This geometry is the gravity dual to a class of 2 + 1-dimensional quantum field theories having Lifshitz symmetry. We have holographically calculated mutual information, entanglement wedge cross section, entanglement negativity and mutual complexity for strip like subsystems at the boundary. For this we have used the results of holographic entanglement entropy and complexity present in the literature. We first calculate all of these mentioned quantities for the pure state of Lifshitz spacetime. Then we have moved on to calculate all these quantities for excited state of the Lifshitz spacetime. The gravity dual of excited state of Lifshitz systems in field theory can be obtained by applying constant perturbations along the boundary direction. Further, we would like to mention that for the simplicity of calculation we are only considering results up to the first order in perturbation. The change in the obtained holographic information theoretic quantities are then related to entanglement entropy, entanglement pressure, entanglement chemical potential and charge using the stress tensor complex. These relations are analogous to the first law of entanglement thermodynamics given earlier in the literature. All the calculations are carried out for both values of dynamical scaling exponent (z) present in the Lifshitz field theory.

Independence test via mutual information in the presence of measurement errors

Independence test via mutual information in the presence of measurement errors

Emotion recognition using cross-modal attention from EEG and facial expression

The fusion of facial expression and Electroencephalogram (EEG) signals in a multi-modal framework is a comprehensive and accurate method for emotion recognition. However, current methods often tend to directly concatenate two modalities, thereby overlooking the interplay between modalities. Furthermore, the selection of single static facial images for extracting facial features neglected the dynamic changes in facial expressions. These limitations have been identified as factors that constrain the accuracy and stability of the model. In this study, an innovative multimodal emotion recognition network was introduced, which using continuous facial expressions and EEG signals. It incorporated the cross-modal attention fusion mechanism to establish robust correlations between modal feature vectors, thereby generating amalgamated vectors with mutual information. Additionally, a Self-Attention Convolutional Long Short-Term Memory (SA-ConvLSTM) was employed to capture spatiotemporal information from facial expression images. The model proposed in this paper was experimentally evaluated on the DEAP and MAHNOB-HCI datasets. Its recognition accuracy was higher than the existing advanced research methods. At the same time, it still performed well in the Leave-One-Subject-Out (LOSO) experiment based on DEAP dataset. The experimental results showed the effectiveness of the proposed model in the task of multimodal emotion recognition.