Conditional Entropy Measure Research Articles

Quantifying Source Location Privacy Routing Performance via Divergence and Information Loss

Source location Privacy (SLP) is an important property for security critical applications deployed over a wireless sensor network. This property specifies that the location of the source of messages needs to be kept secret from an eavesdropping adversary that is able to move around the network. Most previous work on SLP has focused on developing protocols to enhance the SLP imparted to the network under various attacker models and other conditions. Other works have focused on analysing the level of SLP being imparted by a specific protocol. In this paper, we introduce the notion of a routing matrix which captures when messages are <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">first</i> received. We then introduce a novel approach where an optimal SLP routing matrix is derived. In this approach, the attacker’s movement is modelled as a Markov chain where measures of conditional entropy and divergence are used to compare routing matrices and quantify if they provide high levels of SLP. We propose the notion of a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">properly competing paths</i> that causes an attacker to <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">divert</i> when moving towards the source. This concept provides the basis for developing a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">perturbation model</i> , similar to those used in privacy-preserving data mining. We formally prove that properly competing paths are both necessary and sufficient in ensuring the existence of an SLP-aware routing matrix and show their usage in developing an SLP-aware routing matrix. Further, we show how different SLP-aware routing matrices can be obtained through different instantiations of the framework. Those instantiations are obtained based on a notion of information loss achieved through the use of the perturbation model proposed.

Granular-conditional-entropy-based attribute reduction for partially labeled data with proxy labels

Attribute reduction is attracting considerable attention in the theory of rough sets, and thus many rough-set-based attribute reduction methods have been presented. However, most of them are specifically designed for either labeled or unlabeled data, whereas many real-world applications involve partial supervision. In this paper, we propose a rough-set-based semi-supervised attribute reduction method for partially labeled data. Specifically, using prior class-distribution information, we first develop a simple yet effective strategy to produce proxy labels for unlabeled data. Then, the concept of information granularity is integrated into an information-theoretic measure, based on which, a novel granular conditional entropy measure is proposed, and its monotonicity is theoretically proved. Furthermore, a fast heuristic algorithm is provided to generate the optimal reduct of partially labeled data, which could accelerate the process of attribute reduction by removing irrelevant examples and simultaneously excluding redundant attributes. Extensive experiments conducted on UCI data sets demonstrate that the proposed semi-supervised attribute reduction method is promising and, in terms of classification performance, it even compares favorably with supervised methods on labeled and unlabeled data with true labels (Our code and experimental data are released at Mendeley Data https://doi.org/10.17632/v3byhx2v8s.1).

Some Information Measures for Fuzzy Rough Sets

Information theory is a tool to measure uncertainty; these days, it is used to solve various challenging problems that involve hybridization of information theory with the fuzzy set, rough sets, vague sets, etc. In order to solve challenging problems in scientific data analysis and visualization recently, various authors are working on hybrid measures of information theory. In this paper, using the relation between information measures, some measures are proposed for the fuzzy rough set. Firstly, an entropy measure is derived using the fuzzy rough similarity measure, and then corresponding to this entropy measure, some other measures like mutual information measure, joint entropy measure, and conditional entropy measure are also proposed. Some properties of these measures are also studied. Later, the proposed measure is compared with some existing measures to prove its efficiency. Further, the proposed measures are applied to pattern recognition, medical diagnoses, and a real-life decision-making problem for incorporating software in the curriculum at the Department of Statistics.

A validity and reliability study of Conditional Entropy Measures of Pulse Rate Variability.

In this work, we present the feasibility to use a simpler methodological approach for the assessment of the short-term complexity of Heart Rate Variability (HRV). Specifically, we propose to exploit Pulse Rate Variability (PRV) recorded through photoplethysmography in place of HRV measured from the ECG, and to compute complexity via a linear Gaussian approximation in place of the standard model-free methods (e.g., nearest neighbor entropy estimates) usually applied to HRV. Linear PRV-based and model-free HRV-based complexity measures were compared via statistical tests, correlation analysis and Bland-Altman plots, demonstrating an overall good agreement. These results support the applicability of the simpler proposed approach, which is faster and easier-to-implement, making our approach eligible for portable/wearable devices and thus broadening the out-of-lab accessibility of autonomic indexes.

Optimized quantum f-divergences and data processing

The quantum relative entropy is a measure of the distinguishability of two quantum states, and it is a unifying concept in quantum information theory: many information measures such as entropy, conditional entropy, mutual information, and entanglement measures can be realized from it. As such, there has been broad interest in generalizing the notion to further understand its most basic properties, one of which is the data processing inequality. The quantum f-divergence of Petz is one generalization of the quantum relative entropy, and it also leads to other relative entropies, such as the Petz–Rényi relative entropies. In this paper, I introduce the optimized quantum f-divergence as a related generalization of quantum relative entropy. I prove that it satisfies the data processing inequality, and the method of proof relies upon the operator Jensen inequality, similar to Petz’s original approach. Interestingly, the sandwiched Rényi relative entropies are particular examples of the optimized f-divergence. Thus, one benefit of this paper is that there is now a single, unified approach for establishing the data processing inequality for both the Petz–Rényi and sandwiched Rényi relative entropies, for the full range of parameters for which it is known to hold. This paper discusses other aspects of the optimized f-divergence, such as the classical case, the classical-quantum case, and how to construct optimized f-information measures.

Introducing Information Measures via Inference [Lecture Notes

Information measures, such as the entropy and the Kullback-Leibler (KL) divergence, are typically introduced using an abstract viewpoint based on a notion of surprise. Accordingly, the entropy of a given random variable (rv) is larger if its realization, when revealed, is on average more surprising (see, e.g., [1]-[3]). The goal of this lecture note is to describe a principled and intuitive introduction to information measures that builds on inference, i.e., estimation and hypothesis testing. Specifically, entropy and conditional entropy measures are defined using variational characterizations that can be interpreted in terms of the minimum Bayes risk in an estimation problem. Divergence metrics are similarly described using variational expressions derived via mismatched estimation or binary hypothesis testing principles. The classical Shannon entropy and the KL divergence are recovered as special cases of more general families of information measures.

Univariate and multivariate conditional entropy measures for the characterization of short-term cardiovascular complexity under physiological stress

Objective: A defining feature of physiological systems under the neuroautonomic regulation is their dynamical complexity. The most common approach to assess physiological complexity from short-term recordings, i.e. to compute the rate of entropy generation of an individual system by means of measures of conditional entropy (CE), does not consider that complexity may change when the investigated system is part of a network of physiological interactions. This study aims at extending the concept of short-term complexity towards the perspective of network physiology, defining multivariate CE measures whereby multiple physiological processes are accounted for in the computation of entropy rates. Approach: Univariate and multivariate CE measures are computed using state-of-the-art methods for entropy estimation and applied to time series of heart period (H), systolic (S) and diastolic (D) arterial pressure, and respiration (R) variability measured in healthy subjects monitored in a resting state and during conditions of postural and mental stress. Main results: Compared with the traditional univariate metric of short-term complexity, multivariate measures provide additional information with plausible physiological interpretation, such as (i) the dampening of respiratory sinus arrhythmia and activation of the baroreflex control during postural stress; (ii) the increased complexity of heart period and blood pressure variability during mental stress, reflecting the effect of respiratory influences and upper cortical centers; (iii) the strong influence of D on S, mediated by left ventricular ejection fraction and vascular properties; (iv) the role of H in reducing the complexity of D, related to cardiac run-off effects; and (v) the unidirectional role of R in influencing cardiovascular variability. Significance: Our results document the importance of employing a network perspective in the evaluation of the short-term complexity of cardiovascular and respiratory dynamics across different physiological states.

Cardiovascular and respiratory variability during orthostatic and mental stress: A comparison of entropy estimators.

The aim of this study is to characterize cardiovascular and respiratory signals during orthostatic and mental stress as reflected in indices of entropy and complexity, providing a comparison between the performance of different estimators. To this end, the heart rate variability, systolic blood pressure, diastolic blood pressure and respiration time series were extracted from the recordings of 61 healthy volunteers undergoing a protocol consisting of supine rest, head-up tilt test and mental arithmetic task. The analysis was performed in the information domain using measures of entropy and conditional entropy, estimated through model-based (linear) and model-free (binning, nearest neighbor) approaches. Our results show that different types of stress elicited different responses in the employed indices. On one hand, entropy mainly reflected known changes in the variance of physiological time series. On the other hand, the information conveyed by conditional entropy allowed to characterize the complexity of the four time series during the two stress tasks: we found that cardiac and vascular dynamics underwent a reduction in complexity as a consequence of postural stress, while vascular and respiratory complexity increased as a result of mental stress. As for the performance of different estimators, we did not find substantial differences between model-based and model-free approaches, possibly indicating that significant non-linear dynamics did not appear in the studied conditions.

Topology, Cross-Frequency, and Same-Frequency Band Interactions Shape the Generation of Phase-Amplitude Coupling in a Neural Mass Model of a Cortical Column.

Phase-amplitude coupling (PAC), a type of cross-frequency coupling (CFC) where the phase of a low-frequency rhythm modulates the amplitude of a higher frequency, is becoming an important indicator of information transmission in the brain. However, the neurobiological mechanisms underlying its generation remain undetermined. A realistic, yet tractable computational model of the phenomenon is thus needed. Here we analyze a neural mass model of a cortical column, comprising fourteen neuronal populations distributed across four layers (L2/3, L4, L5 and L6). A control analysis showed that the conditional transfer entropy (cTE) measure is able to correctly estimate the flow of information between neuronal populations. Then, we computed cTE from the phases to the amplitudes of the oscillations generated in the cortical column. This approach provides information regarding directionality by distinguishing PAC from APC (amplitude-phase coupling), i.e. the information transfer from amplitudes to phases, and can be used to estimate other types of CFC such as amplitude-amplitude coupling (AAC) and phase-phase coupling (PPC). While experiments often only focus on one or two PAC combinations (e.g., theta-gamma or alpha-gamma), we found that a cortical column can simultaneously generate almost all possible PAC combinations, depending on connectivity parameters, time constants, and external inputs. PAC interactions with and without an anatomical equivalent (direct and indirect interactions, respectively) were analyzed. We found that the strength of PAC between two populations was strongly correlated with the strength of the effective connections between the populations and, on average, did not depend on whether the PAC connection was direct or indirect. When considering a cortical column circuit as a complex network, we found that neuronal populations making indirect PAC connections had, on average, higher local clustering coefficient, efficiency, and betweenness centrality than populations making direct connections and populations not involved in PAC connections. This suggests that their interactions were more effective when transmitting information. Since approximately 60% of the obtained interactions represented indirect connections, our results highlight the importance of the topology of cortical circuits for the generation of the PAC phenomenon. Finally, our results demonstrated that indirect PAC interactions can be explained by a cascade of direct CFC and same-frequency band interactions, suggesting that PAC analysis of experimental data should be accompanied by the estimation of other types of frequency interactions for an integrative understanding of the phenomenon.

Conditional Entropy and Location Error in Indoor Localization Using Probabilistic Wi-Fi Fingerprinting.

Localization systems are increasingly valuable, but their location estimates are only useful when the uncertainty of the estimate is known. This uncertainty is currently calculated as the location error given a ground truth, which is then used as a static measure in sometimes very different environments. In contrast, we propose the use of the conditional entropy of a posterior probability distribution as a complementary measure of uncertainty. This measure has the advantage of being dynamic, i.e., it can be calculated during localization based on individual sensor measurements, does not require a ground truth, and can be applied to discrete localization algorithms. Furthermore, for every consistent location estimation algorithm, both the location error and the conditional entropy measures must be related, i.e., a low entropy should always correspond with a small location error, while a high entropy can correspond with either a small or large location error. We validate this relationship experimentally by calculating both measures of uncertainty in three publicly available datasets using probabilistic Wi-Fi fingerprinting with eight different implementations of the sensor model. We show that the discrepancy between these measures, i.e., many location estimates having a high location error while simultaneously having a low conditional entropy, is largest for the least realistic implementations of the probabilistic sensor model. Based on the results presented in this paper, we conclude that conditional entropy, being dynamic, complementary to location error, and applicable to both continuous and discrete localization, provides an important extra means of characterizing a localization method.

On Sequence Learning Models: Open-loop Control Not Strictly Guided by Hick's Law.

According to the Hick’s law, reaction times increase linearly with the uncertainty of target stimuli. We tested the generality of this law by measuring reaction times in a human sequence learning protocol involving serial target locations which differed in transition probability and global entropy. Our results showed that sigmoid functions better describe the relationship between reaction times and uncertainty when compared to linear functions. Sequence predictability was estimated by distinct statistical predictors: conditional probability, conditional entropy, joint probability and joint entropy measures. Conditional predictors relate to closed-loop control models describing that performance is guided by on-line access to past sequence structure to predict next location. Differently, joint predictors relate to open-loop control models assuming global access of sequence structure, requiring no constant monitoring. We tested which of these predictors better describe performance on the sequence learning protocol. Results suggest that joint predictors are more accurate than conditional predictors to track performance. In conclusion, sequence learning is better described as an open-loop process which is not precisely predicted by Hick’s law.

Conditional entropy in variation-adjusted windows detects selection signatures associated with expression quantitative trait loci (eQTLs).

BackgroundOver the past 50,000 years, shifts in human-environmental or human-human interactions shaped genetic differences within and among human populations, including variants under positive selection. Shaped by environmental factors, such variants influence the genetics of modern health, disease, and treatment outcome. Because evolutionary processes tend to act on gene regulation, we test whether regulatory variants are under positive selection. We introduce a new approach to enhance detection of genetic markers undergoing positive selection, using conditional entropy to capture recent local selection signals. Results We use conditional logistic regression to compare our Adjusted Haplotype Conditional Entropy (H|H) measure of positive selection to existing positive selection measures. H|H and existing measures were applied to published regulatory variants acting in cis (cis-eQTLs), with conditional logistic regression testing whether regulatory variants undergo stronger positive selection than the surrounding gene.These cis-eQTLs were drawn from six independent studies of genotype and RNA expression. The conditional logistic regression shows that, overall, H|H is substantially more powerful than existing positive-selection methods in identifying cis-eQTLs against other Single Nucleotide Polymorphisms (SNPs) in the same genes. When broken down by Gene Ontology, H|H predictions are particularly strong in some biological process categories, where regulatory variants are under strong positive selection compared to the bulk of the gene, distinct from those GO categories under overall positive selection. . However, cis-eQTLs in a second group of genes lack positive selection signatures detectable by H|H, consistent with ancient short haplotypes compared to the surrounding gene (for example, in innate immunity GO:0042742); under such other modes of selection, H|H would not be expected to be a strong predictor.. These conditional logistic regression models are adjusted for Minor allele frequency(MAF); otherwise, ascertainment bias is a huge factor in all eQTL data sets. Relationships between Gene Ontology categories, positive selection and eQTL specificity were replicated with H|H in a single larger data set. Our measure, Adjusted Haplotype Conditional Entropy (H|H), was essential in generating all of the results above because it: 1) is a stronger overall predictor for eQTLs than comparable existing approaches, and 2) shows low sequential auto-correlation, overcoming problems with convergence of these conditional regression statistical models.ConclusionsOur new method, H|H, provides a consistently more robust signal associated with cis-eQTLs compared to existing methods. We interpret this to indicate that some cis-eQTLs are under positive selection compared to their surrounding genes. Conditional entropy indicative of a selective sweep is an especially strong predictor of eQTLs for genes in several biological processes of medical interest. Where conditional entropy is a weak or negative predictor of eQTLs, such as innate immune genes, this would be consistent with balancing selection acting on such eQTLs over long time periods. Different measures of selection may be needed for variant prioritization under other modes of evolutionary selection.

Investigating cardiac and respiratory determinants of heart rate variability in an information-theoretic framework.

This study was aimed at comparing two alternative information-theoretic approaches for the combined analysis of heart rate variability (HRV) and respiration variability (RV). The approaches decompose the predictive information about HRV in two terms, quantifying respectively the information stored into HRV and that transferred to HRV from RV. Storage and transfer were assessed by the popular self entropy (SE) and transfer entropy (TE) measures, as well as by the alternative conditional SE (cSE) and cross entropy (CE) measures. The comparison was performed at a theoretical level, computing the exact values of the four measures for simulated cardiorespiratory dynamics, and on real data, estimating the measures from RV and HRV time series taken from healthy subjects during head-up tilt and paced breathing protocols. Both analyses suggested that, for the study of cardiorespiratory interactions which are mostly unidirectional from RV to HRV, the decomposition evidencing cSE and CE is more suitable to describe respiratory sinus arrhythmia and its modifications related to changes in cardiorespiratory interactions.

Dimensional information-theoretic measurement of facial emotion expressions in schizophrenia.

Altered facial expressions of emotions are characteristic impairments in schizophrenia. Ratings of affect have traditionally been limited to clinical rating scales and facial muscle movement analysis, which require extensive training and have limitations based on methodology and ecological validity. To improve reliable assessment of dynamic facial expression changes, we have developed automated measurements of facial emotion expressions based on information-theoretic measures of expressivity of ambiguity and distinctiveness of facial expressions. These measures were examined in matched groups of persons with schizophrenia (n = 28) and healthy controls (n = 26) who underwent video acquisition to assess expressivity of basic emotions (happiness, sadness, anger, fear, and disgust) in evoked conditions. Persons with schizophrenia scored higher on ambiguity, the measure of conditional entropy within the expression of a single emotion, and they scored lower on distinctiveness, the measure of mutual information across expressions of different emotions. The automated measures compared favorably with observer-based ratings. This method can be applied for delineating dynamic emotional expressivity in healthy and clinical populations.

A measure of underdefined information

The article considers the difficulties connected with the introduction of the concepts of conditional entropy and information measure for underdefined data. These difficulties arise because in the general case of underdefined data it is impossible to identify the entropy addition rule H(X) + H(Y |X) = H(XY) with the interpretation of conditional entropy in terms of relative complexity. A possible compromise is discussed invoking an additional assumption and a weaker form of the entropy addition rule.

Impact of Assimilating Passive Microwave Observations on Root-Zone Soil Moisture Under Dynamic Vegetation Conditions

In this paper, L-band microwave observations were assimilated using the ensemble Kalman filter to improve root-zone soil moisture (RZSM) estimates from a coupled soil vegetation atmosphere transfer (SVAT)-vegetation model linked to a forward microwave model. Simultaneous state-parameter updates were performed by assimilating both synthetic and field observations during a growing season of sweet corn every three days, matching the temporal coverage of observations from the Soil Moisture and Ocean Salinity and Soil Moisture Active Passive missions. The sensitivities of parameters to the states were investigated using the information-theoretic measure of conditional entropy. Among the soil parameters, the pore-size index (λ) was the most sensitive to brightness temperatures (TB) during the early and midgrowth stages, while porosity (φ) was the most sensitive to TB during the reproductive stage. In the microwave model, the soil roughness parameters, root mean square (RMS) height (r), and correlation length (l) were the most sensitive during the early and mid stages, while the vegetation regression parameter (b) was the most sensitive during the reproductive stage. In the synthetic experiment, assimilation of TB provided RMS error reductions in RZSM estimates of 70% compared to open loop estimates. Minimal variations in performance were observed across different stages of the season during the synthetic experiment. However, when field observations of TB were assimilated, significant differences in RZSM estimates were observed during different growth stages. Maximum RMS difference (RMSD) reductions in RZSM estimates of 33.3% were observed compared to open loop estimates during the early stages, while improvements of 4.8% and 16.7% were observed in the mid- and reproductive stages, respectively. Further analyses of assimilation with field observations also suggest some improvements in the SVAT model are needed for moisture transport immediately following the precipitation/irrigation events. In the microwave model, the linear vegetation formulation for estimating canopy opacity, parameterized by b, was inadequate in capturing the complexities in TB during stages of high vegetative and reproductive growth rates.

Approaches to Knowledge Reduction of Decision Systems based on Conditional Rough Entropy

Knowledge reduction in rough set theory is an important feature selection method. Since it is an NP-hard problem, it is necessary to investigate fast and effective approximate algorithms. In this paper, to address this issue, by introducing rough entropy in information systems, the novel measures of conditional rough entropy with distinguishing consistent objects form inconsistent objects are presented for both consistent and inconsistent decision systems. Thus, many important propositions, properties, and conclusions for reduct are drawn, and by using decomposition, radix sorting, hash, and input sequence techniques, we construct a forward greedy algorithm for knowledge reduction. Finally, through analyzing the given example, compared with some standard UCI datasets and other knowledge reduction algorithms, the proposed technique is effective and suitable for both consistent and inconsistent decision systems. Thus, it establishes the theoretical basis for seeking efficient algorithm of knowledge acquisition in decision systems.

Entropy in general physical theories

Information plays an important role in our understanding of the physical world. Hence we propose an entropic measure of information for any physical theory that admits systems, states and measurements. In the quantum and classical worlds, our measure reduces to the von Neumann and Shannon entropies, respectively. It can even be used in a quantum or classical setting where we are only allowed to perform a limited set of operations. In a world that admits superstrong correlations in the form of non-local boxes, our measure can be used to analyze protocols such as superstrong random access encodings and the violation of ‘information causality’. However, we also show that in such a world no entropic measure can exhibit all the properties we commonly accept in a quantum setting. For example, there exists no ‘reasonable’ measure of conditional entropy that is subadditive. Finally, we prove a coding theorem for some theories that is analogous to the quantum and classical settings, providing us with an appealing operational interpretation.

Validating module network learning algorithms using simulated data

BackgroundIn recent years, several authors have used probabilistic graphical models to learn expression modules and their regulatory programs from gene expression data. Despite the demonstrated success of such algorithms in uncovering biologically relevant regulatory relations, further developments in the area are hampered by a lack of tools to compare the performance of alternative module network learning strategies. Here, we demonstrate the use of the synthetic data generator SynTReN for the purpose of testing and comparing module network learning algorithms. We introduce a software package for learning module networks, called LeMoNe, which incorporates a novel strategy for learning regulatory programs. Novelties include the use of a bottom-up Bayesian hierarchical clustering to construct the regulatory programs, and the use of a conditional entropy measure to assign regulators to the regulation program nodes. Using SynTReN data, we test the performance of LeMoNe in a completely controlled situation and assess the effect of the methodological changes we made with respect to an existing software package, namely Genomica. Additionally, we assess the effect of various parameters, such as the size of the data set and the amount of noise, on the inference performance.ResultsOverall, application of Genomica and LeMoNe to simulated data sets gave comparable results. However, LeMoNe offers some advantages, one of them being that the learning process is considerably faster for larger data sets. Additionally, we show that the location of the regulators in the LeMoNe regulation programs and their conditional entropy may be used to prioritize regulators for functional validation, and that the combination of the bottom-up clustering strategy with the conditional entropy-based assignment of regulators improves the handling of missing or hidden regulators.ConclusionWe show that data simulators such as SynTReN are very well suited for the purpose of developing, testing and improving module network algorithms. We used SynTReN data to develop and test an alternative module network learning strategy, which is incorporated in the software package LeMoNe, and we provide evidence that this alternative strategy has several advantages with respect to existing methods.

Testing variability in multivariate quality control: A conditional entropy measure approach

In this paper the concept of entropy is used to construct multivariate quality-control charts to monitor multivariate variability in a process. It is assumed that the data parent population is multivariate normal with covariance structure Σ and estimator S. The asymptotic distribution of the sample entropy is obtained. It is shown that the use of | S| as an overall measure of variability is not satisfactory and it is proposed a conditional entropy approach when the correlation matrix P 0 is known or fixed. Under these assumptions, two new overall variability measures are defined based on the sample variances and ranges on the variables under consideration. Control charts are developed using these concepts and are applied to a multivariate quality-control example.