Maximum Information Entropy Research Articles

Integrating information theory and adversarial learning for cross-modal retrieval

Accurately matching visual and textual data in cross-modal retrieval has been widely studied in the multimedia community. To address these challenges posited by the heterogeneity gap and the semantic gap, we propose integrating Shannon information theory and adversarial learning. In terms of the heterogeneity gap, we integrate modality classification and information entropy maximization adversarially. For this purpose, a modality classifier (as a discriminator) is built to distinguish the text and image modalities according to their different statistical properties. This discriminator uses its output probabilities to compute Shannon information entropy, which measures the uncertainty of the modality classification it performs. Moreover, feature encoders (as a generator) project uni-modal features into a commonly shared space and attempt to fool the discriminator by maximizing its output information entropy. Thus, maximizing information entropy gradually reduces the distribution discrepancy of cross-modal features, thereby achieving a domain confusion state where the discriminator cannot classify two modalities confidently. To reduce the semantic gap, Kullback-Leibler (KL) divergence and bi-directional triplet loss are used to associate the intra- and inter-modality similarity between features in the shared space. Furthermore, a regularization term based on KL-divergence with temperature scaling is used to calibrate the biased label classifier caused by the data imbalance issue. Extensive experiments with four deep models on four benchmarks are conducted to demonstrate the effectiveness of the proposed approach.

Self-consistent equations governing the dynamics of non-equilibrium binary colloidal systems

Most of the existing non-equilibrium theories are developed based on local-equilibrium assumptions and therefore encounter difficulty in addressing dynamical processes far-from-equilibrium. Herein, we present a set of dynamical equations to describe the dynamics of non-equilibrium binary colloidal system, which is derived by combining the Kramers equation with the maximum path information entropy principle. These equations, involving the local density, local momentum and local kinetic energy, are coupled with each other and eventually depend on the two-body probability distribution function, whose least-biased prediction is given by applying the maximum path information entropy principle. We show that the proposed dynamical governing equations are self-consistent, and can recover to the existing relevant theories upon various local equilibrium assumptions. The simplified forms of these equations are also discussed for several types of systems with geometrical symmetries. This work provides a theoretical framework at molecular level for investigating dynamical behaviors of multi-component systems far-from-equilibrium.

Constructing the Electronic Countermeasures System Architecture Based on Case-Based Reasoning and Bayesian Inference

In order to promote the quality of electronic countermeasures system architecture, this paper present a methodology for constructing the electronic countermeasures system architecture based on case-based reasoning and Bayesian inference. The methodology described history cases by CPN model, established the largest similarity case and feasibility space by combining the CPN color set and k-means algorithm, and then proposed the optimizing orderings of variables of Bayesian by the LFS (Legal Firing Sequence) of CPN model and maximum information entropy, learned the Bayesian networks structure by K2 algorithm using the result of orderings of variables. At last, this paper concluded the electronic countermeasures system architecture tactics and discussed the way of reasoning for constructing the electronic countermeasures system architecture.

Maximal Entropy Approach for Quantum State Tomography

Quantum computation has been growing rapidly in both theory and experiments. In particular, quantum computing devices with a large number of qubits have been developed by IBM, Google, IonQ, and others. The current quantum computing devices are noisy intermediate-scale quantum $($NISQ$)$ devices, and so approaches to validate quantum processing on these quantum devices are needed. One of the most common ways of validation for an n-qubit quantum system is quantum tomography, which tries to reconstruct a quantum system's density matrix by a complete set of observables. However, the inherent noise in the quantum systems and the intrinsic limitations poses a critical challenge to precisely know the actual measurement operators which make quantum tomography impractical in experiments. Here, we propose an alternative approach to quantum tomography, based on the maximal information entropy, that can predict the values of unknown observables based on the available mean measurement data. This can then be used to reconstruct the density matrix with high fidelity even though the results for some observables are missing. Of additional contexts, a practical approach to the inference of the quantum mechanical state using only partial information is also needed.

On an enhancement of RNA probing data using information theory

Identifying the secondary structure of an RNA is crucial for understanding its diverse regulatory functions. This paper focuses on how to enhance target identification in a Boltzmann ensemble of structures via chemical probing data. We employ an information-theoretic approach to solve the problem, via considering a variant of the Rényi-Ulam game. Our framework is centered around the ensemble tree, a hierarchical bi-partition of the input ensemble, that is constructed by recursively querying about whether or not a base pair of maximum information entropy is contained in the target. These queries are answered via relating local with global probing data, employing the modularity in RNA secondary structures. We present that leaves of the tree are comprised of sub-samples exhibiting a distinguished structure with high probability. In particular, for a Boltzmann ensemble incorporating probing data, which is well established in the literature, the probability of our framework correctly identifying the target in the leaf is greater than 90%.

Feature Selection Method Based on Differential Correlation Information Entropy

Feature selection is one of the major aspects of pattern classification systems. In previous studies, Ding and Peng recognized the importance of feature selection and proposed a minimum redundancy feature selection method to minimize redundant features for sequential selection in microarray gene expression data. However, since the minimum redundancy feature selection method is used mainly to measure the dependency between random variables of mutual information, the results cannot be optimal without consideration of global feature selection. Therefore, based on the framework of minimum redundancy-maximum correlation, this paper introduces entropy to measure global feature selection and proposes a new feature subset evaluation method, differential correlation information entropy. In our function, different bivariate correlation metrics are selected. Then, the feature selection is completed through sequence forward search. Two different classification models are used on eleven standard data sets of the UCI machine learning knowledge base to compare various comparison algorithms, such as mRMR, reliefF and feature selection method with joint maximal information entropy, with our method. The experimental results show that feature selection based on our proposed method is obviously superior to that of other models.

A Trust-Game-Based Access Control Model for Cloud Service

In order to promote mutual trust and win-win cooperation between the users and the providers, we propose a trust-game-based access control model for cloud service in the paper. First, we construct a trust evaluation model based on multiple factors, such as the direct trust, feedback trust, reward punishment, and trust risk and further propose a weight method by maximum discrete degree and information entropy theory; second, we combine trust evaluation with the payoff matrix for game analysis and calculate the mixed Nash equilibrium strategy for the users and service providers; third, we give the game control condition based on trust level prediction and payment matrix to encourage participants to make honest strategy. Experimental results show that our research has good effect in terms of acceptance probability, deception probability, accuracy of trust evaluation, and cooperation rate in the cloud service.

An Adaptive GaoFen-3 SAR Wind Field Retrieval Algorithm Based on Information Entropy

Chinese GaoFen-3 (GF-3) synthetic aperture radar (SAR) imagery acquired in standard strip mode can capture streak features caused by wind over the sea surface, which are commonly known as wind streaks. To retrieve the sea surface wind field from a GF-3 SAR image, an adaptive wind retrieval algorithm based on information entropy is proposed in this paper. This algorithm can adaptively extract the optimal wavenumber range of each subimage based on maximum information entropy. After all subimages are processed for possible wind streak direction solutions, a circle median filter is used to determine the unique wind streak direction solution for each subimage. Twelve GF-3 standard strip SAR images are inverted to validate the proposed algorithm. The experiments show that when the GF-3 image data are compared to the European Centre for Medium-Range Weather Forecasts reanalysis wind field data, the root mean square error of wind speed and direction retrieved by the algorithm are 1.29 m/s and 15.33°, respectively, and when they are compared to the National Data Buoy Center buoy wind data, the mean absolute deviation errors of wind speed and direction are 0.57 m/s and 9.95°, respectively. These comparisons demonstrate that the proposed algorithm can effectively retrieve wind field information from GF-3 SAR images.

Information Entropy Multi-Decision Attribute Reduction Fuzzy Rough Set for Dust Particulate Imagery Characteristic Extraction

High-precision extraction of particulate characteristic modes is essential for dust explosion safety measurements, such as particulate concentration and size distribution. A new solution based on information entropy multi-decision attribute reduction fuzzy rough set is proposed to analyse the particulate morphology characteristics, which effectively avoids the shortcomings of traditional technology (low accuracy, stochasticity, etc.). The proposed approach consists of three stages: membership function modelling, attribute reduction, and maximum information entropy threshold segmentation. The membership coefficient was determined with a multi-segment function by developing the fuzzy degree of the membership model for dust image pixels. The fuzzy dependence of the conditional attribute was determined to extract the fuzzy attribute reduction. Finally, the model of coal dust particulates with information entropy was improved to extract the maximum segmentation threshold, which is significant for classification. The proposed methods were evaluated over a sequence of 30 image sets. The unclassified rate evaluation reached 0.978 for particle sizes $\ge200~\mu \text{m}$ , 0.958 for [75 $\mu \text{m}$ , 200 $\mu \text{m}$ ] and 0.950 for particle sizes $ . The proposed reduction approach offered a performance improvement in terms of more important attribute implementation. The paper demonstrated that the maximum information entropy reduction model can remove the redundant attributes without compromising the precision.

State-of-Health Prognosis for Lithium-Ion Batteries Considering the Limitations in Measurements via Maximal Information Entropy and Collective Sparse Variational Gaussian Process

Prognostics and health management (PHM) for electronic devices is intricate yet crucial in an era of electricity. The impending future of electric vehicles and clean energy requires more in-depth scrutiny and monitoring of their storage elements, lithium-ion batteries. State-of-health (SOH), as one of the most evident indicators for battery degradation quantifications, is a matter of vital importance that worths more attention. In this regard, this article proposes a novel SOH prognostics framework for lithium-ion batteries considering the limitations in the recorded measurements through the use of linear statistical k-nearest neighbors (LSKNN) data interpolation, maximal information entropy search (MIES), and collective sparse variational Gaussian process regression (CSVGPR). First, the incomplete charging measurements are processed by LSKNN to infer the missing data points and suppress the unanticipated noises in the extracted temporal features, which indicate the trend of degradation. Then, the MIES scheme is proposed to filter the features that are extraneous to the SOHs of the corresponding batteries and that greatly correlate to the other features in the feature set. Finally, the CSVGPR model, considering the uncertainties within each of the sparse variational Gaussian processes, is utilized to implement SOH prognosis. The proposed framework is verified by a subset of the repository from NASA. In the test, multiple prognostics comparisons of inner-battery tests, cross-battery tests, and tests with other statistical learning methods are presented. The experiment results lend support to the superiority and effectiveness of the work.

Identification of lung cancer gene markers through kernel maximum mean discrepancy and information entropy

BackgroundThe early diagnosis of lung cancer has been a critical problem in clinical practice for a long time and identifying differentially expressed gene as disease marker is a promising solution. However, the most existing gene differential expression analysis (DEA) methods have two main drawbacks: First, these methods are based on fixed statistical hypotheses and not always effective; Second, these methods can not identify a certain expression level boundary when there is no obvious expression level gap between control and experiment groups.MethodsThis paper proposed a novel approach to identify marker genes and gene expression level boundary for lung cancer. By calculating a kernel maximum mean discrepancy, our method can evaluate the expression differences between normal, normal adjacent to tumor (NAT) and tumor samples. For the potential marker genes, the expression level boundaries among different groups are defined with the information entropy method.ResultsCompared with two conventional methods t-test and fold change, the top average ranked genes selected by our method can achieve better performance under all metrics in the 10-fold cross-validation. Then GO and KEGG enrichment analysis are conducted to explore the biological function of the top 100 ranked genes. At last, we choose the top 10 average ranked genes as lung cancer markers and their expression boundaries are calculated and reported.ConclusionThe proposed approach is effective to identify gene markers for lung cancer diagnosis. It is not only more accurate than conventional DEA methods but also provides a reliable method to identify the gene expression level boundaries.

Effective and Efficient Blind Quality Evaluator for Contrast Distorted Images

This paper mainly focuses on developing a blind quality assessment method that can effectively and efficiently evaluate the quality of contrast distorted images without requiring reference information. Through experiments, we discover and validate that the global intensity change is the main characteristic of contrast distorted images and has a close relationship to the perceptual quality. With these observations, two elements are utilized to quantify this characteristic, i.e., the maximum information entropy of intensity values and the Kullback-Leibler (K-L) divergence between the test image's intensity histogram and the prior one based on the statistical experiment over a great number of high-quality images. To be specific, the entropy represents the valuable information of an image and the K-L divergence reflects the change degree of intensity distribution. In view of these, the proposed method is generated by combining these two elements linearly. Extensive experiments on three publicly available databases demonstrate the superiority of the proposed method. More specifically, it is more consistent with subjective evaluation results than the state-of-the-art image quality assessment methods and requires a lower computational complexity.

Feature subset selection combining maximal information entropy and maximal information coefficient

Feature subset selection is an efficient step to reduce the dimension of data, which remains an active research field in decades. In order to develop highly accurate and fast searching feature subset selection algorithms, a filter feature subset selection method combining maximal information entropy (MIE) and the maximal information coefficient (MIC) is proposed in this paper. First, a new metric mMIE-mMIC is defined to minimize the MIE among features while maximizing the MIC between the features and the class label. The mMIE-mMIC algorithm is designed to evaluate whether a candidate subset is valid for classification. Second, two searching strategies are adopted to identify a suitable solution in the candidate subset space, including the binary particle swarm optimization algorithm (BPSO) and sequential forward selection (SFS). Finally, classification is performed on UCI datasets to validate the performance of our work compared to 9 existing methods. Experimental results show that in most cases, the proposed method behaves equally or better than the other 9 methods in terms of classification accuracy and F1-score.

Derivations of the Core Functions of the Maximum Entropy Theory of Ecology.

The Maximum Entropy Theory of Ecology (METE), is a theoretical framework of macroecology that makes a variety of realistic ecological predictions about how species richness, abundance of species, metabolic rate distributions, and spatial aggregation of species interrelate in a given region. In the METE framework, “ecological state variables” (representing total area, total species richness, total abundance, and total metabolic energy) describe macroecological properties of an ecosystem. METE incorporates these state variables into constraints on underlying probability distributions. The method of Lagrange multipliers and maximization of information entropy (MaxEnt) lead to predicted functional forms of distributions of interest. We demonstrate how information entropy is maximized for the general case of a distribution, which has empirical information that provides constraints on the overall predictions. We then show how METE’s two core functions are derived. These functions, called the “Spatial Structure Function” and the “Ecosystem Structure Function” are the core pieces of the theory, from which all the predictions of METE follow (including the Species Area Relationship, the Species Abundance Distribution, and various metabolic distributions). Primarily, we consider the discrete distributions predicted by METE. We also explore the parameter space defined by the METE’s state variables and Lagrange multipliers. We aim to provide a comprehensive resource for ecologists who want to understand the derivations and assumptions of the basic mathematical structure of METE.

Estimation of swine movement network at farm level in the US from the Census of Agriculture data

Swine movement networks among farms/operations are an important source of information to understand and prevent the spread of diseases, nearly nonexistent in the United States. An understanding of the movement networks can help the policymakers in planning effective disease control measures. The objectives of this work are: (1) estimate swine movement probabilities at the county level from comprehensive anonymous inventory and sales data published by the United States Department of Agriculture - National Agriculture Statistics Service database, (2) develop a network based on those estimated probabilities, and (3) analyze that network using network science metrics. First, we use a probabilistic approach based on the maximum information entropy method to estimate the movement probabilities among different swine populations. Then, we create a swine movement network using the estimated probabilities for the counties of the central agricultural district of Iowa. The analysis of this network has found evidence of the small-world phenomenon. Our study suggests that the US swine industry may be vulnerable to infectious disease outbreaks because of the small-world structure of its movement network. Our system is easily adaptable to estimate movement networks for other sets of data, farm animal production systems, and geographic regions.

Molecular reconstruction of naphtha by maximum information entropy method

Naphtha is the feedstock for steam cracking and catalytic reforming of aromatics. Based on the molecular library and 42 groups of practical data including the properties of density, potential aromatic content, the weight fractions of PNA and carbon number distribution, a molecular reconstruction model for naphtha was established by maximum information entropy method. The content of 451 components for naphtha were estimated. The molecular composition generated by the simulation is in good agreement with the practical data. The model can provide the referential value for the steam cracking and catalytic reforming of aromatics processes at molecular level.

Unsupervised semantic deep hashing

In recent years, deep hashing methods have been proved to be effective since it employs convolutional neural network to learn features and hashing codes simultaneously. However, these methods are mostly supervised. In real-world applications, it is a time-consuming and overloaded task for annotating a large number of images. In this paper, we propose a novel unsupervised deep hashing method for large-scale image retrieval. Our method, namely unsupervised semantic deep hashing (USDH), uses semantic information preserved in the CNN feature layer to guide the training of network. We enforce four criteria on hashing codes learning based on VGG-19 model: 1) preserving relevant information of feature space in hashing space; 2) minimizing quantization loss between binary-like codes and hashing codes; 3) improving the usage of each bit in hashing codes by using maximum information entropy, and 4) invariant to image rotation. Extensive experiments on CIFAR-10, NUSWIDE have demonstrated that USDH outperforms several state-of-the-art unsupervised hashing methods for image retrieval. We also conduct experiments on Oxford 17 datasets for fine-grained classification to verify its efficiency for other computer vision tasks.

Динамический хаос в нелинейной системе с 1/f-спектром

In the system of two nonlinear differential equations, proposed to explain the physical nature of the 1/f spectra, chaotization of the trajectories is revealed under periodic external action on one of the equations. External noise effects lead to stochastic resonance and low-frequency 1/f behavior of power spectra. Stochastic resonance and 1/f behavior of power spectra corresponds to the maximum information entropy, which indicates the stability of a random process.

Detecting stability of conical spouted beds based on information entropy theory

Effects of particle size, particle density, gas inlet diameter and static bed height on the stability of operation in conical spouted beds were investigated through analyses of information entropy of pressure fluctuations. In this respect, the maximum information entropy of pressure fluctuations was used as a stability criterion. The results showed that stability of the bed increases with an increase in the maximum entropy. The maximum information entropy of pressure fluctuations increases with increasing particle size and bed height while decreases with increasing gas inlet diameter and particle density. A stability map was also prepared to present the effect of operating parameters on the maximum entropy. Moreover, a correlation for prediction of maximum information entropy was developed to determine the stable operation conditions of conical spouted beds operating with low as well as high density particles.

Stability of Random Processes with the 1/f α Spectrum

Extreme fluctuations have been modeled based on point nonpotential and potential systems of stochastic equations, in which white noise induces random processes with a power-law frequency dependence of the power spectra. The distribution of extreme fluctuations corresponds to the maximum of statistical entropy, which indicates their stability. The stability of fluctuation processes with 1/f α power spectra has been analyzed based on the principle of maximum information entropy.