Mutual Information Values Research Articles

Zero-shot learning by mutual information estimation and maximization

The key of zero-shot learning is to use the visual-semantic embedding to transfer the knowledge from seen classes to unseen classes. In this paper, we propose to build the visual-semantic embedding by maximizing the mutual information between visual features and corresponding attributes. Then, the mutual information between visual and semantic features can be utilized to guide the knowledge transfer from seen domain to unseen domain. Since we are primarily interested in maximizing mutual information, we introduce the noise-contrastive estimation to calculate lower-bound value of mutual information. Through the noise-contrastive estimation, we reformulate zero-shot learning as a binary classification problem, i.e., classifying the matching visual-semantic pairs (positive samples) and mismatching visual-semantic pairs (negative/noise samples). Experiments conducted on five datasets demonstrate that the proposed mutual information estimators outperforms current state-of-the-art methods both in conventional and generalized zero-shot learning settings.

User Embedding for Rating Prediction in SVD++-Based Collaborative Filtering

The collaborative filtering algorithm based on the singular value decomposition plus plus (SVD++) model employs the linear interactions between the latent features of users and items to predict the rating in the recommendation systems. Aiming to further enrich the user model with explicit feedback, this paper proposes a user embedding model for rating prediction in SVD++-based collaborative filtering, named UE-SVD++. We exploit the user potential explicit feedback from the rating data and construct the user embedding matrix by the proposed user-wise mutual information values. In addition, the user embedding matrix is added to the existing user bias and implicit parameters in the SVD++ to increase the accuracy of the user modeling. Through extensive studies on four different datasets, we found that the rating prediction performance of the UE-SVD++ model is improved compared with other models, and the proposed model’s evaluation indicators root-mean-square error (RMSE) and mean absolute error (MAE) are decreased by 1.002–2.110% and 1.182–1.742%, respectively.

Correlated activity favors synergistic processing in local cortical networks in vitro at synaptically relevant timescales.

Neural information processing is widely understood to depend on correlations in neuronal activity. However, whether correlation is favorable or not is contentious. Here, we sought to determine how correlated activity and information processing are related in cortical circuits. Using recordings of hundreds of spiking neurons in organotypic cultures of mouse neocortex, we asked whether mutual information between neurons that feed into a common third neuron increased synergistic information processing by the receiving neuron. We found that mutual information and synergistic processing were positively related at synaptic timescales (0.05–14 ms), where mutual information values were low. This effect was mediated by the increase in information transmission—of which synergistic processing is a component—that resulted as mutual information grew. However, at extrasynaptic windows (up to 3,000 ms), where mutual information values were high, the relationship between mutual information and synergistic processing became negative. In this regime, greater mutual information resulted in a disproportionate increase in redundancy relative to information transmission. These results indicate that the emergence of synergistic processing from correlated activity differs according to timescale and correlation regime. In a low-correlation regime, synergistic processing increases with greater correlation, and in a high-correlation regime, synergistic processing decreases with greater correlation.

A New Method for Automatic Electrooculogram and Eye Blink Artifacts Correction of EEG Signals using CCA and NAPCT

Eye movements during electroencephalogram (EEG) recordings are the major sources of artifacts. These artifacts tend to mask the EEG signals. So, to obtain good quality EEG signals, these artifacts must be removed without deteriorating the underlying EEG activity. In this paper, a new algorithm is proposed that combines canonical correlation analysis (CCA) and noise adjusted principal component transform (NAPCT) to efficiently remove the electrooculogram (EOG) and blink artifacts in a considerably fast manner. CCA-NAPCT is implemented after the preliminary outlier thresholding of EEG data. CCA is used to estimate the noise covariance matrix while NAPCT is implemented for noise removal. The results of this algorithm on EOG affected BCI competition III dataset IVb and blink contaminated EEG data of four subjects showed the efficacy of the proposed algorithm in effective removal of noise. The algorithm provides an average signal to noise ratio and root mean square error values of 3.616 & 42.456 with artifactual EEG data respectively. Moreover, the average correlation coefficients (0.8839) and mutual information (1.1546) values also verify the efficacy of algorithm more firmly as supported by comparison with the state-of-the-art technique. The proposed algorithm successfully removed the artifactual components with no manual intervention.

Hybrid Segmentation Algorithm for Medical Image Segmentation Based on Generating Adversarial Networks, Mutual Information and Multi-Scale Information

This paper proposes 3D-MedGAN, MLU-Net and Info-Max-Net models for overcoming the lack of labeled data and extracting the multi-level feature of images in medical image segmentation. 3D-MedGAN is aimed at dealing with the lack of labeled data in medical images. It uses a generative adversarial network to simulate data and then draws newly generated samples from the distribution learned by the model. Training the segmentation model by mixing generated samples with real samples can effectively improve the effect of the segmentation model. MLU-Net uses multiple layers of different levels of convolutional angles to extract feature information from multiple angles in medical images. By adopting the attention mechanism to fuse the multi-level feature information, MLU-Net is able to improve the feature expressions and segmentation effect. Info-Max-Net is aimed at handling the noise problem in medical images. When the information in the images is complex, it is difficult to extract features. Using mutual information to measure the dependency between the image and the extracted features can effectively reduce noise in the image and improve the effect of segmentation. At the same time, for solving the problem that the high dimension of the image makes it difficult to measure mutual information, this paper uses a lower bound BL-estimator to measure the mutual information between the optimized image and the extracted features. Therefore, the model can maintain a high convergence speed as it approaches the true value of mutual information. Considering that the quality of images generated by 3D-MedGAN are not as good as the original images, we combine the 3D-MedGAN, MLU-Net, and Info-Max-Net to improve the sensitivity and the power of feature extraction of the hybird model. The effectiveness of our model is verified through experiments of the 3D-MedGAN, MLU-Net, Info-Max-Net, and the hybrid model over the LIVER100 dataset.

Multisource Latent Feature Selective Ensemble Modeling Approach for Small-Sample High-Dimensional Process Data in Applications

Several difficult-to-measure production qualities or environment pollution indices of industrial process must be measured using offline laboratory instruments. Soft measurement method is often used to perform online prediction of such parameters. Only small-sample modeling data with high-dimensional input features can be obtained due to the limitations and complex characteristics of the measurement device and process, respectively. Therefore, a new multisource latent feature selective ensemble (SEN) modeling approach is proposed in this study. First, input features are divided into different subgroups according to the characteristics of the modeling data. Second, the extracted multisource latent features evolve from the multi-layered selection algorithms, which are specified by feature reduction ratio, feature contribution ratio and mutual information value orderly for each subgroup. Finally, in order to construct candidate sub-models, an adaptive hyper-parameter selection algorithm based on the multi-step grid search is employed in terms of the reduced features. Sequentially, the optimized ensemble submodels with their weighting strategies are adaptively determined to build the final SEN model. The proposed method is verified by using benchmark near-infrared data, high dimensional mechanical frequency spectrum data and industrial dioxin emission concentration data.

A bag of constrained informative deep visual words for image retrieval

In this paper, we propose a bag of constrained informative deep visual words (BoCIDVW) model for image retrieval. Informative patches from each image are first obtained using patch entropy values. Each such patch is represented by deep features extracted through VGG16-Net. Two sets of constraints, namely, the must-link (ML) and the cannot-link (CL), are obtained for each deep informative patch in an unsupervised manner from its mutual information values (with other patches). The patches are then quantized using the Linear-time Constrained Vector Quantization Error (LCVQE), a fast yet accurate constrained K-means algorithm. The resulting clusters, which we term constrained informative deep visual words, are employed to label each patch. Finally, a bag (histogram) of constrained informative visual words is developed for image retrieval. Experiments on three different publicly available datasets demonstrate the merit of the proposed formulation.

A Feature Selection Algorithm Based on Equal Interval Division and Minimal-Redundancy–Maximal-Relevance

Minimal-redundancy–maximal-relevance (mRMR) algorithm is a typical feature selection algorithm. To select the feature which has minimal redundancy with the selected features and maximal relevance with the class label, the objective function of mRMR subtracts the average value of mutual information between features from mutual information between features and the class label, and selects the feature with the maximum difference. However, the problem is that the feature with the maximum difference is not always the feature with minimal redundancy maximal relevance. To solve the problem, the objective function of mRMR is first analyzed and a constraint condition that determines whether the objective function can guarantee the effectiveness of the selected features is achieved. Then, for the case where the objective function is not accurate, an idea of equal interval division is proposed and combined with ranking to process the interval of mutual information between features and the class label, and that of the average value of mutual information between features. Finally, a feature selection algorithm based on equal interval division and minimal-redundancy–maximal-relevance (EID–mRMR) is proposed. To validate the performance of EID–mRMR, we compare it with several incremental feature selection algorithms based on mutual information and other feature selection algorithms. Experimental results demonstrate that the EID–mRMR algorithm can achieve better feature selection performance.

Evaluation Method for the Optimization of 3D Rigid Image Registration on Multimodal Image Datasets

Optimization based three dimensional (3D) rigid image registration (RIR) is one of the most commonly used methods of image registration in radiotherapy. Interpolator and similarity metric plays a crucial role in optimization image registration process. In this paper, the efficiency of image registration algorithm is analyzed by using various combinations of interpolators and similarity metric in terms of quantitative measures and is compared with commercially available image registration algorithm in radiotherapy. Computed Tomography (CT) and Cone Beam Computed Tomography (CBCT) image datasets were registered by image registration algorithm written in python language using simple image tool kit (SITK). Different combinations of similarity metric and interpolator such as mean square difference (MSD), mutual information (MI), demons and nearest neighbor (NN), linear, B- spline respectively were used in this study. The efficiency of the algorithm was quantified in terms of mean square error (MSE), structural similarity index (SSI), normalized cross correlation (NCC) and mutual information (MI). The image registration algorithm with most efficient combination of similarity metric and interpolator was selected for comparison with the commercially available image registration algorithm. The algorithm for multimodal (CT-CBCT) 3D image registration with NN interpolator and MI similarity metric showed the highest values of SSI, NCC and MI as 0.865, 0.933, 1.223 respectively among other combination of interpolator and similarity metric. Further this algorithm when compared and statistically analyzed with commercially available image registration algorithm of Treatment Planning System (TPS. most commonly used for radiotherapy treatment) resulted in no significant difference (F value NCC-3.18, MI-4.010, SSI-2.776) in their quantitative measures. The present study is limited to 3D RIR and can be extended for deformable image registration.

Information theoretic waveform design for OFDM radar‐communication coexistence in Gaussian mixture interference

In this study, mutual information (MI)-based waveform design methods for the coexistence between orthogonal frequency division multiplexing (OFDM) radar system and communication system corrupted by Gaussian mixture interference are investigated for the first time. Firstly, the received signal is formulated as a Bayesian linear model. Subsequently, from a statistical perspective, the approximate differential entropy of received signal is derived, followed by three different approximate MIs and corresponding theoretic upper bounds. After that, two adaptive waveform design methods are proposed by maximising MI and minimising radar transmitted power, respectively. The numerical results show that exploiting the reflected communication signals helps to obtain higher MI value and reduce radar transmitted power. Besides, by maximising the MIs, the optimised radar waveforms remarkably outperform the fixed ones, which benefits the estimation performance.

Visible/near Infrared Reflection Spectrometer and Electronic Nose Data Fusion as an Accuracy Improvement Method for Portable Total Soluble Solid Content Detection of Orange

The visible/near infrared (VIS/NIR) spectrometer and electronic nose (E-nose) are two commonly used portable and nondestructive detection apparatuses which have a promising application for the quick acquisition of fruit’s internal quality in both the orchard and market. However, the accuracy of these instruments is sometimes unsatisfactory, especially for thick peeled fruit like the ‘Aiyuan 38’ orange, which was investigated in this research. The objective of this research was to find a method to improve the accuracy for the detection of an orange’s total soluble solid content (TSS) using a VIS/NIR spectrometer and E-nose. Different spectrum detection positions and conventional feature extraction methods are compared to get the optimal data fusion parameters. The detection model was then built up based on the obtained fusion data under the optimal parameters. Partial least squares regression (PLSR) and mutual information theory (MIT) were applied for feature extraction, and PLSR and principal component analysis (PCA)-back propagation neural network (BPNN) were applied for modeling and detection. PLSR results showed that the sampling reflection spectrum at the position of the calyx results in a better orange TSS detection than other sampling positions. For VIS/NIR reflection spectrum feature extraction, PLSR and MIT results showed that the optimal data process + feature extraction method is Savitzky-Golay + 763 features, when their mutual information values between the feature and TSS value were larger than 0.74. For E-nose feature extraction, PLSR and MIT results showed that the combined feature (combination of 75 s value, average value, average of differential value, integral value, and maximum value) is the optimal feature extraction method, and all features are retained for modeling. The PLSR detection ability of orange TSS based on fusion data is better than the single detection method, with the detection ability of the single detection methods being unsatisfactory. PCA-BPNN has better orange TSS detection ability than PLSR. The R2, RMSE, and slope from the calibration set for PCA-BPNN detection were 0.9695, 0.1895, and 0.9665, respectively, and from the validation set for PCA-BPNN detection were 0.8872, 0.4709, and 1.0871, respectively, indicating that this method can detect orange TSS efficiently.

A kernel-induced weighted object-cluster association-based ensemble method for educational data clustering

ABSTRACTExploiting single models for a better ensemble one is popular for a classification or clustering task. For educational data clustering, our work proposes a kernel-induced weighted object-cluster association ensemble method, named kWOCA, to discover the inherent structure of the data and return the student clusters of higher quality. kWOCA is an advanced version that can automatically determine the number of desired clusters based on Bayesian Information Criterion. It also conducts consensus clustering in the feature space for non-linearly separated clusters where the discrimination between the objects is captured from the base clusterings. Besides, it encodes the differences between the clusters in each base clustering and those between the base clusterings in the ensemble when making a synthesis of the base clusterings. A kernel-induced weighted object-cluster association matrix is defined to store such rich information. Using this matrix, kWOCA outperforms the existing clustering ensemble methods. Experimental results on the real educational data sets and the benchmark Iris data set show the better effectiveness of kWOCA with higher Normalized Mutual Information values. As a result, groups of the most similar students based on their study performance can be discovered better. These resulting student clusters can be then further analyzed for academic affairs.

Signature of quantum chaos in operator entanglement in 2d CFTs

We study operator entanglement measures of the unitary evolution operators of (1 + 1)-dimensional conformal field theories (CFTs), aiming to uncover their scrambling and chaotic behaviors. In particular, we compute the bi-partite and tri-partite mutual information for various configurations of input and output subsystems, and as a function of time. We contrast three different CFTs: the free fermion theory, compactified free boson theory at various radii, and CFTs with holographic dual. We find that the bi-partite mutual information exhibits distinct behaviors for these different CFTs, reflecting the different information scrambling capabilities of these unitary operators; while a quasi-particle picture can describe the case of the free fermion and free boson CFTs well, it completely fails for the case of holographic CFTs. Similarly, the tri-partite mutual information also distinguishes the unitary evolution operators of different CFTs. In particular, its late-time behaviors, when the output subsystems are semi-infinite, are quite distinct for these theories. We speculate that for holographic theories the late-time saturation value of the tri-partite mutual information takes the largest possible negative value and saturates the lower bound among quantum field theories.

Secure limitation analysis of public-key cryptography for smart card settings

Smart cards are widely used in high security applications due to their self-contained nature. At the same time, the security of smart card has become an urgent problem in the field of intelligent environment. Public-key Cryptography is the main means to solve the security problems based on smart card password authentication and identity authentication protocol. This paper reviews the security issues of public key cryptography used in smart cards from the perspective of information theory. By constructing a attackers channel, we model the Public-key Cryptography process in the way of an adversary to capture the attack ability in the Public-key Cryptography setting. Then, we convert the secure problems of Public-key Cryptography into the attack channels capacity of adversaries that the maximum value of the average mutual information is the secure limitations of a Public-key Cryptography scheme, which is a reachable theoretic limitation of secure communication parties. Finally, we give the bounds of insecure for public-key encryption and signature in different secure levels, and analyze and discuss the secure limitation.

A Pilot Study of Chaos Criteria with Hilbert Transform and Mutual Information

Chaos feature criteria tree has many fruits with geometric maps and calculated values. To easily understand how to study chaos identification, we propose one new learning combination solution based on Hilbert transform (HT) and mutual information (MI). The most important coding probe (x-axis) used the HT of locked uniformly distributed random number (LUSRN), so far the y-axis in contrast standard map points to LUDRN. The measure between contrast standard map and unknown map (normalized test data, HT of LUSRN) used MI values. The test cases cover five chaos equations and eighteen quasi-period functions. The results show in statistics that our new maps and corresponding MI values can classify three kinds of signals with periodic, chaotic and random states. And this work may promote rapid growth of an apple embedded in novel chaos criteria.

Increased electroencephalography connectivity precedes epileptic spasm onset in infants with tuberous sclerosis complex.

To identify whether abnormal electroencephalography (EEG) connectivity is present before the onset of epileptic spasms (ES) in infants with tuberous sclerosis complex (TSC). Scalp EEG recordings were collected prospectively in infants diagnosed with TSC in the first year of life. This study compared the earliest recorded EEG from infants prior to ES onset (n=16) and from infants who did not develop ES (n=28). Five minutes of stage II or quiet sleep was clipped and filtered into canonical EEG frequency bands. Mutual information values between each pair of EEG channels were compared directly and used as a weighted graph to calculate graph measures of global efficiency, characteristic path length, average clustering coefficient, and modularity. At the group level, infants who later developed ES had increased EEG connectivity in sleep. They had higher mutual information values between most EEG channels in all frequency bands adjusted for age. Infants who later developed ES had higher global efficiency and average clustering coefficients, shorter characteristic path lengths, and lower modularity across most frequency bands adjusted for age. This suggests that infants who went on to develop ES had increased local and long-range EEG connectivity with less segregation of graph regions into distinct modules. This study suggests that increased neural connectivity precedes clinical ES onset in a cohort of infants with TSC. Overconnectivity may reflect progressive pathologic network synchronization culminating in generalized ES. Further research is needed before scalp EEG connectivity measures can be used as a potential biomarker of ES risk and treatment response in pre-symptomatic infants with TSC.

Mutual Information Analysis of Brain-Body Interactions during different Levels of Mental stress.

In this work, we analyze brain-heart interactions during different mental states computing mutual information (MI) between the dynamic activity of different physiological systems. In 18 healthy subjects monitored in a relaxed resting state and during a mental arithmetic and a serious game task, multichannel EEG, one lead ECG, respiration and blood volume pulse were collected via wireless non-invasive biosensors. From these signals, synchronous 300-second time series were extracted measuring brain activity via the δ, θ, α, and β EEG power, and activity of the body district via the ECG R-R interval η, the respiratory amplitude ρ and the pulse arrival time π. MI was computed using a linear estimator: (i) between {η,ρ,π} and {δ,θ,α,β}, to measure overall brain-body interactions; (ii) between each time series and the others of the same district, to measure information shared within a district; and (iii) between each time series of a district and all series of the other district, to evaluate individual contributions to the information shared between brain and body. Results document the existence of statistically significant brain-body interactions, with high MI values involving mainly the η body dynamics and the δ and β brain dynamics. State-dependent variations were mostly relevant to the MI of the brain system involving δ, θ, α during mental arithmetic, and α and β during serious game. Thus, MI can be useful to detect correlated activity within and between brain and body systems monitored simultaneously during different mental states.

Two-Stage Hybrid Gene Selection Using Mutual Information and Genetic Algorithm for Cancer Data Classification.

Cancer is a deadly disease which requires a very complex and costly treatment. Microarray data classification plays an important role in cancer treatment. An efficient gene selection technique to select the more promising genes is necessary for cancer classification. Here, we propose a Two-stage MI-GA Gene Selection algorithm for selecting informative genes in cancer data classification. In the first stage, Mutual Information based gene selection is applied which selects only the genes that have high information related to the cancer. The genes which have high mutual information value are given as input to the second stage. The Genetic Algorithm based gene selection is applied in the second stage to identify and select the optimal set of genes required for accurate classification. For classification, Support Vector Machine (SVM) is used. The proposed MI-GA gene selection approach is applied to Colon, Lung and Ovarian cancer datasets and the results show that the proposed gene selection approach results in higher classification accuracy compared to the existing methods.

A Coherence Algorithm for 3-D Seismic Data Analysis Based on the Mutual Information

Coherence algorithm is widely used to describe geological discontinuity and subtle features of seismic data. Traditional coherence algorithms often use the linear correlation measurement to measure the relationship between two seismic traces. It does not work well because seismic data do not obey the normal distribution. To describe the coherence measurement of seismic data, we propose an improved coherence algorithm by combining the mutual information (MI) and third-generation coherence (C3) algorithm. The MI is a measurement of the general dependence and used to describe nonlinear relationships between two variables. Note that, the MI does not require that variables obey specific distribution (e.g., the normal distribution). Note that, we calculate precise MI values using the copula function. In addition, we introduce the information divergence to save calculation time by replacing the eigenvalue decomposition of the C3 algorithm. To demonstrate the effectiveness of the proposed algorithm, we apply it to field data. Field data experiments demonstrate the effectiveness of the proposed algorithm to describe geological discontinuity and heterogeneity, such as channels with different thicknesses.

Lexicon-based emotion analysis in Turkish

In this paper, we proposed a lexicon for emotion analysis in Turkish for six emotional categories happiness, fear, anger, sadness, disgust, and surprise. Besides, we also investigated the effects of a lemmatizer and a stemmer, two term-weighting schemes, four lexicon enrichment methods, and a term selection approach for lexicon construction. To do this, we generated Turkish emotion lexicon based on a dataset, TREMO, containing 25,989 documents. We then preprocessed the documents to obtain dictionary and stem forms of each term using a lemmatizer and a stemmer. Afterwards, we proposed two different weighting schemes where term frequency, term-class frequency and mutual information (MI) values for six emotion categories are taken into consideration. We then enriched the lexicon by using bigram and concept hierarchy methods, and performed term selection for efficiency issues. Then, we compared the performance of lexicon-based approach with machine learning based approach by using our proposed lexicon. The experiments showed that the use of the proposed lexicon efficiently produces comparable results in emotion analysis in Turkish text.