Information-theoretic Metrics Research Articles

Optimizing parameter learning and calibration in an integrated hydrological model: Impact of observation length and information

Integrated hydrological modeling is gaining popularity due to its mechanistic representation of the surface and subsurface processes. However, estimating the parameters of such process-based models can be computationally expensive if careful consideration is not given to the length of streamflow observations used during model calibration. Here we evaluate the influence of the calibration period, the role of streamflow information content, and the gage location in parameter learning and calibration of a fully integrated hydrological model, the Advanced Terrestrial Simulator (ATS). We conducted the study at the Upper Neversink River Watershed within the Delaware River Basin, where streamflow observations are available at 11 gages with varying record lengths. We leveraged a recently proposed knowledge-informed deep learning technique for parameter estimation. To assess the impact of observation period and gage location, model parameters were learned on scenarios using different chunks of streamflow observations, including (1) using only one year or consecutive multiple years of streamflow observations at the watershed outlet and (2) using one shared year of observations at each of the sub-catchment gages, with the period from 1991-10-01 through 1999-09-30 as the overall calibration period. Using the estimated parameters, ATS was rerun for each scenario and evaluated on a subsequent period from 1999-10-01 through 2002-09-30. Results show that the basin outlet discharge prediction is mostly improved when using at least four years of observations for parameter estimation. Further, the performance of the calibrated ATS run correlates with the information content of the observed streamflows, suggesting that the information-theoretic metrics could be indicators for selecting the observation period for parameter estimation. Finally, we find that observations from an informative gage can be used in learning parameters to predict the streamflow at a nearby gage, which would potentially lower the computational expense by reducing the watershed domain used in calibration. Our success underscores the potential of using information theory to achieve robust model parameter estimation on a reduced computational budget.

Measuring Abstraction Levels of Sculptural Objects

Abstract Freestanding sculpture can be scored for abstraction using order scales such as the three-rank ‘realistic,’ ‘mixed,’ and ‘abstract.’ Subjective but invaluable, the ranks support ordering but not arithmetic. Numeric limitations can be addressed via information-theoretic metrics that measure collective uncertainty about a sculpture; to this end 60 participants view 25 images of sculpture via an internet questionnaire. They (i) score each depicted object and (ii) type a caption stating what impression the object evokes. Captions for the image are simplified to their English simple subject, sorted into classification categories and counted. A metric converts an image’s categories and counts into viewers’ uncertainty. The investigation examines four-rank and five-rank scales plus metrics CC (category count), H (entropy) and EC ≡ 2H (equalized category count). Medians of scores or uncertainties reduce variability as appropriate. Comparisons of scores vs uncertainties show the two generally correlate well. Results also highlight inherent design tradeoffs. Scale/metric pairings affect both ranks (which should be statistically distinct) and rank/uncertainty correlations. Two good combinations are [four-rank; median CC] and [five-rank; median EC]. Metric CC is easy to compute and correlates slightly better than EC, although the latter supports five evenly separated ranks. Shifting focus to image scatterplots, the pair [four-rank median score; CC] yields a very strong correlation. Once uncertainties are linked to ranks to ‘calibrate’ them, one gets a quantitative sense of the rank order scale. The augmented framework offers the speed and ease of scoring along with valid numeric estimates of uncertainty unavailable from ranks alone.

Viewpoint Selection for 3D-Games with f-Divergences.

In this paper, we present a novel approach for the optimal camera selection in video games. The new approach explores the use of information theoretic metrics f-divergences, to measure the correlation between the objects as viewed in camera frustum and the ideal or target view. The f-divergences considered are the Kullback-Leibler divergence or relative entropy, the total variation and the χ2 divergence. Shannon entropy is also used for comparison purposes. The visibility is measured using the differential form factors from the camera to objects and is computed by casting rays with importance sampling Monte Carlo. Our method allows a very fast dynamic selection of the best viewpoints, which can take into account changes in the scene, in the ideal or target view, and in the objectives of the game. Our prototype is implemented in Unity engine, and our results show an efficient selection of the camera and an improved visual quality. The most discriminating results are obtained with the use of Kullback-Leibler divergence.

Quantifying the efficacy of voltage protocols in characterising ion channel kinetics: A novel information-theoretic approach.

Voltage-clamp experiments are commonly utilised to characterise cellular ion channel kinetics. In these experiments, cells are stimulated using a known time-varying voltage, referred to as the voltage protocol, and the resulting cellular response, typically in the form of current, is measured. Parameters of models that describe ion channel kinetics are then estimated by solving an inverse problem which aims to minimise the discrepancy between the predicted response of the model and the actual measured cell response. In this paper, a novel framework to evaluate the information content of voltage-clamp protocols in relation to ion channel model parameters is presented. Additional quantitative information metrics that allow for comparisons among various voltage protocols are proposed. These metrics offer a foundation for future optimal design frameworks to devise novel, information-rich protocols. The efficacy of the proposed framework is evidenced through the analysis of seven voltage protocols from the literature. By comparing known numerical results for inverse problems using these protocols with the information-theoretic metrics, the proposed approach is validated. The essential steps of the framework are: (i) generate random samples of the parameters from chosen prior distributions; (ii) run the model to generate model output (current) for all samples; (iii) construct reduced-dimensional representations of the time-varying current output using proper orthogonal decomposition (POD); (iv) estimate information-theoretic metrics such as mutual information, entropy equivalent variance, and conditional mutual information using non-parametric methods; (v) interpret the metrics; for example, a higher mutual information between a parameter and the current output suggests the protocol yields greater information about that parameter, resulting in improved identifiability; and (vi) integrate the information-theoretic metrics into a single quantitative criterion, encapsulating the protocol's efficacy in estimating model parameters.

Language prediction in monolingual and bilingual speakers: an EEG study

Prediction of upcoming words is thought to be crucial for language comprehension. Here, we are asking whether bilingualism entails changes to the electrophysiological substrates of prediction. Prior findings leave it open whether monolingual and bilingual speakers predict upcoming words to the same extent and in the same manner. We address this issue with a naturalistic approach, employing an information-theoretic metric, surprisal, to predict and contrast the N400 brain potential in monolingual and bilingual speakers. We recruited 18 Iranian Azeri-Persian bilingual speakers and 22 Persian monolingual speakers. Subjects listened to a story in Persian while their electroencephalogram (EEG) was recorded. Bayesian item-level analysis was used. While in monolingual speakers N400 was sensitive to information-theoretic properties of both the current and previous words, in bilingual speakers N400 reflected the properties of the previous word only. Our findings show evidence for a processing delay in bilingual speakers which is consistent with prior research.

Bayesian estimation of information-theoretic metrics for sparsely sampled distributions

Estimating the Shannon entropy of a discrete distribution from which we have only observed a small sample is challenging. Estimating other information-theoretic metrics, such as the Kullback–Leibler divergence between two sparsely sampled discrete distributions, is even harder. Here, we propose a fast, semi-analytical estimator for sparsely sampled distributions. Its derivation is grounded in probabilistic considerations and uses a hierarchical Bayesian approach to extract as much information as possible from the few observations available. Our approach provides estimates of the Shannon entropy with precision at least comparable to the benchmarks we consider, and most often higher; it does so across diverse distributions with very different properties. Our method can also be used to obtain accurate estimates of other information-theoretic metrics, including the notoriously challenging Kullback–Leibler divergence. Here, again, our approach has less bias, overall, than the benchmark estimators we consider.

Stochastic Thermodynamics of Learning Parametric Probabilistic Models.

We have formulated a family of machine learning problems as the time evolution of parametric probabilistic models (PPMs), inherently rendering a thermodynamic process. Our primary motivation is to leverage the rich toolbox of thermodynamics of information to assess the information-theoretic content of learning a probabilistic model. We first introduce two information-theoretic metrics, memorized information (M-info) and learned information (L-info), which trace the flow of information during the learning process of PPMs. Then, we demonstrate that the accumulation of L-info during the learning process is associated with entropy production, and the parameters serve as a heat reservoir in this process, capturing learned information in the form of M-info.

Dynamic Probabilistic Pruning: A General Framework for Hardware-Constrained Pruning at Different Granularities.

Unstructured neural network pruning algorithms have achieved impressive compression ratios. However, the resulting-typically irregular-sparse matrices hamper efficient hardware implementations, leading to additional memory usage and complex control logic that diminishes the benefits of unstructured pruning. This has spurred structured coarse-grained pruning solutions that prune entire feature maps or even layers, enabling efficient implementation at the expense of reduced flexibility. Here, we propose a flexible new pruning mechanism that facilitates pruning at different granularities (weights, kernels, and feature maps) while retaining efficient memory organization (e.g., pruning exactly k -out-of- n weights for every output neuron or pruning exactly k -out-of- n kernels for every feature map). We refer to this algorithm as dynamic probabilistic pruning (DPP). DPP leverages the Gumbel-softmax relaxation for differentiable k -out-of- n sampling, facilitating end-to-end optimization. We show that DPP achieves competitive compression ratios and classification accuracy when pruning common deep learning models trained on different benchmark datasets for image classification. Relevantly, the dynamic masking of DPP facilitates for joint optimization of pruning and weight quantization in order to even further compress the network, which we show as well. Finally, we propose novel information-theoretic metrics that show the confidence and pruning diversity of pruning masks within a layer.

Calibration, validation, and selection of hydrostatic testing-based remaining useful life prediction models for polyethylene pipes

A novel methodology for model selection among competing models for remaining useful life (RUL) prediction is developed in this paper. Due to the long durability of polyethylene (PE) pipes under normal conditions, life data under normal operating conditions is not available for model validation and selection. Physics-based, regression-based, and hybrid models for RUL prediction are available in the literature based on experimental data under accelerated test conditions. These models need to be evaluated for prediction performance under normal conditions, but in the absence of data under normal conditions. A model consistency-based metric for selecting the best model in the absence of data under normal conditions is proposed in this paper. The consistency-based metric evaluates the predictive consistency of the various probabilistic RUL prediction models over the estimated or known probability distribution of the normal operating conditions. A two-step approach for model selection is proposed: first, validation evaluation using empirical data, and second, consistency evaluation under likely operating conditions. The proposed model selection methodology is demonstrated using five candidate models for RUL prediction of PE pipes based on accelerated hydrostatic testing data. Bayesian inference is used to calibrate these models with empirical data, and its benefit over the least squares approach recommended in ISO 9080 is demonstrated. Further, the proposed two-step model selection methodology is compared against traditional model selection methods based on goodness of fit, model complexity and information theoretic metrics. It is seen that the proposed additional consistency criterion is successful in selecting the best model compared to existing methods that are unable to distinguish between the different available models.

Measures of Information Leakage for Incomplete Statistical Information: Application to a Binary Privacy Mechanism

Information leakage is usually defined as the logarithmic increment in the adversary’s probability of correctly guessing the legitimate user’s private data or some arbitrary function of the private data when presented with the legitimate user’s publicly disclosed information. However, this definition of information leakage implicitly assumes that both the privacy mechanism and the prior probability of the original data are entirely known to the attacker. In reality, the assumption of complete knowledge of the privacy mechanism for an attacker is often impractical. The attacker can usually have access to only an approximate version of the correct privacy mechanism, computed from a limited set of the disclosed data, for which they can access the corresponding un-distorted data. In this scenario, the conventional definition of leakage no longer has an operational meaning. To address this problem, in this article, we propose novel meaningful information-theoretic metrics for information leakage when the attacker has incomplete information about the privacy mechanism—we call them average subjective leakage , average confidence boost , and average objective leakage , respectively. For the simplest, binary scenario, we demonstrate how to find an optimized privacy mechanism that minimizes the worst-case value of either of these leakages.

Deep discriminative clustering and structural constraint for cross-domain fault diagnosis of rotating machinery

With the rapid development of intelligent manufacturing, fault diagnostic methods based on deep learning have achieved impressive results. However, most methods require plentiful annotated samples and are based on the assumption that data from the source and target domains has the same distribution. These two conditions are difficult to satisfy in practical engineering. In light of these problems, an unsupervised domain adaptation approach named Deep Discriminative Clustering network with Structural Constraint (DDCSC) is proposed in this article. In our method, a Convolutional Neural Network (CNN) module is exploited for learning feature representations of raw data. Then a softmax module is employed to simultaneously predict class probabilities and cluster assignments of the source and target data, respectively. The learnable cluster centroids are introduced into the latent feature space to alleviate the data distribution discrepancy while better capturing the discriminative structure of the target data. In addition, geometric properties of the source data in a feature space are constrained to expand the scope of each category, which facilitates to improve prediction accuracy. An information-theoretic metric is considered as the objective function of discriminative clustering. Diagnostic experiments on a rolling bearing dataset demonstrate that our approach outperforms other popular intelligent approaches and confirms the effectiveness of discriminative clustering.

Task-based differences in brain state dynamics and their relation to cognitive ability

Transient patterns of interregional connectivity form and dissipate in response to varying cognitive demands. Yet, it is not clear how different cognitive demands influence brain state dynamics, and whether these dynamics relate to general cognitive ability. Here, using functional magnetic resonance imaging (fMRI) data, we characterised shared, recurrent, global brain states in 187 participants across the working memory, emotion, language, and relation tasks from the Human Connectome Project. Brain states were determined using Leading Eigenvector Dynamics Analysis (LEiDA). In addition to the LEiDA-based metrics of brain state lifetimes and probabilities, we also computed information-theoretic measures of Block Decomposition Method of complexity, Lempel-Ziv complexity and transition entropy. Information theoretic metrics are notable in their ability to compute relationships amongst sequences of states over time, compared to lifetime and probability, which capture the behaviour of each state in isolation. We then related task-based brain state metrics to fluid intelligence.We observed that brain states exhibited stable topology across a range of numbers of clusters (K = 2:15). Most metrics of brain state dynamics, including state lifetime, probability, and all information theoretic metrics, reliably differed between tasks. However, relationships between state dynamic metrics and cognitive abilities varied according to the task, the metric, and the value of K, indicating that there are contextual relationships between task-dependant state dynamics and trait cognitive ability.This study provides evidence that the brain reconfigures across time in response to cognitive demands, and that there are contextual, rather than generalisable, relationships amongst task, state dynamics, and cognitive ability.

Information Theory-based Evolution of Neural Networks for Side-channel Analysis

Profiled side-channel analysis (SCA) leverages leakage from cryptographic implementations to extract the secret key. When combined with advanced methods in neural networks (NNs), profiled SCA can successfully attack even those cryptocores assumed to be protected against SCA. Despite the rise in the number of studies devoted to NN-based SCA, a range of questions has remained unanswered, namely: how to choose an NN with an adequate configuration, how to tune the NN’s hyperparameters, when to stop the training, etc. Our proposed approach, “InfoNEAT,” tackles these issues in a natural way. InfoNEAT relies on the concept of neural structure search, enhanced by information-theoretic metrics to guide the evolution, halt it with novel stopping criteria, and improve time-complexity and memory footprint. The performance of InfoNEAT is evaluated by applying it to publicly available datasets composed of real side-channel measurements. In addition to the considerable advantages regarding the automated configuration of NNs, InfoNEAT demonstrates significant improvements over other approaches for effective key recovery in terms of the number of epochs (e.g.,x6 faster) and the number of attack traces compared to both MLPs and CNNs (e.g., up to 1000s fewer traces to break a device) as well as a reduction in the number of trainable parameters compared to MLPs (e.g., by the factor of up to 32). Furthermore, through experiments, it is demonstrated that InfoNEAT’s models are robust against noise and desynchronization in traces.

Accounting for the Effect of Noise in Satellite Soil Moisture Data on Estimates of Land–Atmosphere Coupling Using Information Theoretical Metrics

Abstract Land states can affect the atmosphere through their control of surface turbulent fluxes and the subsequent impact of those fluxes on boundary layer properties. Information theoretic (IT) metrics are ideal to study the strength and type of coupling between surface soil moisture (SM) and land surface heat fluxes (HFs) because they are nonparametric and thus appropriate for the analysis of highly complex Earth systems containing nonlinear cause-and-effect interactions that may have nonnormal distributions. Specifically, a methodology for the estimation of IT metrics from noisy time series is proposed, accounting for random errors in satellite-based SM data. Performance of the proposed method is demonstrated through synthetic tests. Efficacy of the method is greatest for estimates of entropy and mutual information involving SM; improvements to estimates of transfer entropy are significant but less stark. A global depiction of the information flow between SM and HFs is then constructed from observationally based gridded data. This is used as independent verification for two configurations of the ECMWF modeling system: unconstrained open-loop (retrospective forecasts) and constrained by data assimilation (ERA5). Compared to studies that only investigate the linear SM–HF relationships, extended regions of significant terrestrial coupling are found over the globe, as IT metrics enable detection of nonlinear dependencies. The magnitude and spatial variability of coupling strength and type from models show discrepancies with those from observations, highlighting the potential to improve SM and HF covariability within models. Although ERA5 did not perform better than the unconstrained model in very dry climates, its performance is generally superior to that of the unconstrained model across metrics.

Target-specific compound selectivity for multi-target drug discovery and repurposing.

Most drug molecules modulate multiple target proteins, leading either to therapeutic effects or unwanted side effects. Such target promiscuity partly contributes to high attrition rates and leads to wasted costs and time in the current drug discovery process, and makes the assessment of compound selectivity an important factor in drug development and repurposing efforts. Traditionally, selectivity of a compound is characterized in terms of its target activity profile (wide or narrow), which can be quantified using various statistical and information theoretic metrics. Even though the existing selectivity metrics are widely used for characterizing the overall selectivity of a compound, they fall short in quantifying how selective the compound is against a particular target protein (e.g., disease target of interest). We therefore extended the concept of compound selectivity towards target-specific selectivity, defined as the potency of a compound to bind to the particular protein in comparison to the other potential targets. We decompose the target-specific selectivity into two components: 1) the compound’s potency against the target of interest (absolute potency), and 2) the compound’s potency against the other targets (relative potency). The maximally selective compound-target pairs are then identified as a solution of a bi-objective optimization problem that simultaneously optimizes these two potency metrics. In computational experiments carried out using large-scale kinase inhibitor dataset, which represents a wide range of polypharmacological activities, we show how the optimization-based selectivity scoring offers a systematic approach to finding both potent and selective compounds against given kinase targets. Compared to the existing selectivity metrics, we show how the target-specific selectivity provides additional insights into the target selectivity and promiscuity of multi-targeting kinase inhibitors. Even though the selectivity score is shown to be relatively robust against both missing bioactivity values and the dataset size, we further developed a permutation-based procedure to calculate empirical p-values to assess the statistical significance of the observed selectivity of a compound-target pair in the given bioactivity dataset. We present several case studies that show how the target-specific selectivity can distinguish between highly selective and broadly-active kinase inhibitors, hence facilitating the discovery or repurposing of multi-targeting drugs.

Joint entropy analysis of anterior-posterior and medial-lateral body sway.

We studied the Shannon entropy of center of pressure (COP) trajectories under different sensory feedback conditions and analyzed the interrelations between medial-lateral (ML), anterior-posterior (AP) and joint (ML and AP) sway entropy. Static balance was assessed on a force platform under different visual and proprioceptive feedback conditions: Standing on firm support with eyes open (condition 1) or closed (condition 2) and standing on foam with eyes closed (condition 3). Postural sway was analyzed by means of linear and nonlinear, information theoretic metrics (entropy of ML and AP sway, joint ML and AP entropy). ML entropy, AP entropy and joint ML and AP entropy remained stable from condition 1 through condition 3. The values of ML and AP entropies were practically at their theoretical maximum in all conditions. On the other hand, joint ML and AP entropies were clearly submaximal. Decreasing the reliability of visual and proprioceptive input does not alter the Shannon entropy of body sway, although it does increase the magnitude of conventional linear sway metrics. Importantly, individual ML and AP sway entropies tend towards absolute randomness, whereas the joint, ML and AP, sway entropy exhibits a higher degree of regularity, suggesting its role as the actual controlled variable.

Physics-based modeling and information-theoretic sensor and settings selection for tool wear detection in precision machining

Precision machining of metals is an energy intensive process with applications and impacts across the manufacturing industry. The energy efficiency, product yield, and maintenance of the precision machine require a digital twin that can assist with prognostics and health management. Here, a physics-based model is developed and validated against face milling data, and then used for the timely and precise inference of machining faults that cannot be measured directly. Computer numerical control (CNC) measurements of power and force are used through this physics-based machining model to predict deviations of the outputs of power consumption and cutting forces during normal operation. A model-based fault detection and isolation methodology is applied to determine the optimal (traditional and available) sensor suite and the test settings (admissible input values) that improve the inference of tool wear in face milling. The optimal sensor suite and input test settings are obtained by solving a mixed integer non-linear program that optimizes information-theoretic metrics relevant to the detection and isolation of tool wear from steady-state or transient machining measurements. Dynamic time warping and k—NN classification are then used to validate the robustness of the optimal design for fault detection test design, including the optimal sensor suite.

A New Constraint on the Nuclear Equation of State from Statistical Distributions of Compact Remnants of Supernovae

Understanding how matter behaves at the highest densities and temperatures is a major open problem in both nuclear physics and relativistic astrophysics. Our understanding of such behavior is often encapsulated in the so-called high-temperature nuclear equation of state (EOS), which influences compact binary mergers, core-collapse supernovae, and other phenomena. Our focus is on the type (either black hole or neutron star) and mass of the remnant of the core collapse of a massive star. For each six candidates of equations of state, we use a very large suite of spherically symmetric supernova models to generate a sample of synthetic populations of such remnants. We then compare these synthetic populations to the observed remnant population. Our study provides a novel constraint on the high-temperature nuclear EOS and describes which EOS candidates are more or less favored by an information-theoretic metric.

Group-level inference of information-based measures for the analyses of cognitive brain networks from neurophysiological data

The reproducibility crisis in neuroimaging and in particular in the case of underpowered studies has introduced doubts on our ability to reproduce, replicate and generalize findings. As a response, we have seen the emergence of suggested guidelines and principles for neuroscientists known as Good Scientific Practice for conducting more reliable research. Still, every study remains almost unique in its combination of analytical and statistical approaches. While it is understandable considering the diversity of designs and brain data recording, it also represents a striking point against reproducibility. Here, we propose a non-parametric permutation-based statistical framework, primarily designed for neurophysiological data, in order to perform group-level inferences on non-negative measures of information encompassing metrics from information-theory, machine-learning or measures of distances. The framework supports both fixed- and random-effect models to adapt to inter-individuals and inter-sessions variability. Using numerical simulations, we compared the accuracy in ground-truth retrieving of both group models, such as test- and cluster-wise corrections for multiple comparisons. We then reproduced and extended existing results using both spatially uniform MEG and non-uniform intracranial neurophysiological data. We showed how the framework can be used to extract stereotypical task- and behavior-related effects across the population covering scales from the local level of brain regions, inter-areal functional connectivity to measures summarizing network properties. We also present an open-source Python toolbox called Frites1 that includes the proposed statistical pipeline using information-theoretic metrics such as single-trial functional connectivity estimations for the extraction of cognitive brain networks. Taken together, we believe that this framework deserves careful attention as its robustness and flexibility could be the starting point toward the uniformization of statistical approaches.

Alleviating the attribute conditional independence and I.I.D. assumptions of averaged one-dependence estimator by double weighting

Learning Bayesian network classifiers (BNCs) from data is NP-hard. Of numerous BNCs, averaged one-dependence estimator (AODE) performs extremely well against more sophisticated newcomers, and its trade-off between bias and variance can be attributed to the independence assumption and i.i.d. assumption, which respectively address the issues of structure complexity and data complexity. To alleviate these assumptions and improve AODE, we propose to apply double weighting, including attribute weighting and model weighting, to finely tune the estimates of conditional probability based on generative learning and joint probability based on discriminative learning, respectively. Instance weighting is introduced to define the information-theoretic metrics for identifying the variation in probability distributions for different data points. This highly scalable learning approach can establish a decision boundary that is specifically tailored to each instance. Our extensive experimental evaluation on 34 datasets from the UCI machine learning repository shows that, attribute weighting and model weighting are complementary although they can work separately. The proposed AODE applying double weighting schema, called DWAODE, is a competitive alternative to other weighting approaches. The experimental results show that DWAODE demonstrates significant advantage in terms of zero–one loss, bias–variance decomposition, RMSE (root mean squared error), Friedman and Nemenyi tests.