Information-theoretic Techniques Research Articles

Communication Complexity of the Secret Key Agreement in Algorithmic Information Theory

It is known that the mutual information, in the sense of Kolmogorov complexity, of any pair of strings x and y is equal to the length of the longest shared secret key that two parties can establish via a probabilistic protocol with interaction on a public channel, assuming that the parties hold as their inputs x and y respectively. We determine the worst-case communication complexity of this problem for the setting where the parties can use private sources of random bits. We show that for some x , y the communication complexity of the secret key agreement does not decrease even if the parties have to agree on a secret key the size of which is much smaller than the mutual information between x and y . On the other hand, we discuss a natural class of x , y such that the communication complexity of the protocol declines gradually with the size of the derived secret key. The proof of the main result uses the spectral properties of appropriate graphs and the expander mixing lemma, as well as information-theoretic techniques, including constraint information inequalities and Muchnik’s conditional descriptions. A preliminary version of this paper was published in the proceedings of MFCS 2020. In the present version we give full proofs of all theorems and get rid of the assumption that the number of random bits used in the communication protocols is polynomial.

Generalized decomposition of multivariate information.

Since its introduction, the partial information decomposition (PID) has emerged as a powerful, information-theoretic technique useful for studying the structure of (potentially higher-order) interactions in complex systems. Despite its utility, the applicability of the PID is restricted by the need to assign elements as either "sources" or "targets", as well as the specific structure of the mutual information itself. Here, I introduce a generalized information decomposition that relaxes the source/target distinction while still satisfying the basic intuitions about information. This approach is based on the decomposition of the Kullback-Leibler divergence, and consequently allows for the analysis of any information gained when updating from an arbitrary prior to an arbitrary posterior. As a result, any information-theoretic measure that can be written as a linear combination of Kullback-Leibler divergences admits a decomposition in the style of Williams and Beer, including the total correlation, the negentropy, and the mutual information as special cases. This paper explores how the generalized information decomposition can reveal novel insights into existing measures, as well as the nature of higher-order synergies. We show that synergistic information is intimately related to the well-known Tononi-Sporns-Edelman (TSE) complexity, and that synergistic information requires a similar integration/segregation balance as a high TSE complexity. Finally, I end with a discussion of how this approach fits into other attempts to generalize the PID and the possibilities for empirical applications.

Map point selection for visual SLAM

Simultaneous localisation and mapping (SLAM) play a vital role in autonomous robotics. Robotic platforms are often resource-constrained, and this limitation motivates resource-efficient SLAM implementations. While sparse visual SLAM algorithms offer good accuracy for modest hardware requirements, even these more scalable sparse approaches face limitations when applied to large-scale and long-term scenarios. A contributing factor is that the point clouds resulting from SLAM are inefficient to use and contain significant redundancy.This paper proposes the use of subset selection algorithms to reduce the map produced by sparse visual SLAM algorithms. Information-theoretic techniques have been applied to simpler related problems before, but they do not scale if applied to the full visual SLAM problem. This paper proposes a number of novel information-theoretic utility functions for map point selection and optimises these functions using greedy algorithms. The reduced maps are evaluated using practical data alongside an existing visual SLAM implementation (ORB-SLAM 2). Approximate selection techniques proposed in this paper achieve trajectory accuracy comparable to an offline baseline while being suitable for online use. These techniques enable the practical reduction of maps for visual SLAM with competitive trajectory accuracy.Results also demonstrate that SLAM front-end performance can significantly impact the performance of map point selection. This shows the importance of testing map point selection with a front-end implementation. To exploit this, this paper proposes an approach that includes a model of the front-end in the utility function when additional information is available. This approach outperforms alternatives on applicable datasets and highlights future research directions.

Refining HIV envelope protein conformational data from different experimental sources using bias-resampling methods and information theory.

Human immunodeficiency virus (HIV) remains a global epidemic, and vaccine development has been challenging. This challenge is due in part to conflicting data regarding which conformational states of the envelope glycoprotein (Env) are effectively targeted by neutralizing antibodies. Env is known to undergo conformational change as part of its function. Identifying the conformations best bound by neutralizing antibodies will improve vaccine design. To this end, we use bias resampling ensemble refinement (BRER) to integrate and compare prior spectroscopic measurements of HIV Env conformation. By using both electron paramagnetic resonance and single-molecule FRET data as restraints in conformational ensemble refinement, we generate models for the sets of HIV Env conformations consistent with each set of measurements. Results show that the “hidden” conformational states identified via single-molecule FRET spectroscopy indeed show differences from reference cryo-EM structures. Further analysis shows that the “hidden” conformational states have different distance distributions for some residue pairs than those measured via DEER. We have developed information-theoretic techniques to identify DEER label sites that would best differentiate conformational states defined by single-molecule FRET. We hope that testing of these proposed DEER experiments will help resolve the discrepancy between observations using different spectroscopic techniques and better elucidate the conformational heterogeneity of HIV Env. Extension to datasets in the presence of bound neutralizing antibodies will help elucidate the key conformations recognized by broadly neutralizing anti-HIV antibodies and, we hope, further assist in vaccine design.

Reducing the question burden of patient reported outcome measures using Bayesian networks

Patient Reported Outcome Measures (PROMs) are questionnaires completed by patients about aspects of their health status. They are a vital part of learning health systems as they are the primary source of information about important outcomes that are best assessed by patients such as pain, disability, anxiety and depression. The volume of questions can easily become burdensome. Previous techniques reduced this burden by dynamically selecting questions from question item banks which are specifically built for different latent constructs being measured. These techniques analyzed the information function between each question in the item bank and the measured construct based on item response theory then used this information function to dynamically select questions by computerized adaptive testing. Here we extend those ideas by using Bayesian Networks (BNs) to enable Computerized Adaptive Testing (CAT) for efficient and accurate question selection on widely-used existing PROMs. BNs offer more comprehensive probabilistic models of the connections between different PROM questions, allowing the use of information theoretic techniques to select the most informative questions. We tested our methods using five clinical PROM datasets, demonstrating that answering a small subset of questions selected with CAT has similar predictions and error to answering all questions in the PROM BN. Our results show that answering 30% − 75% questions selected with CAT had an average area under the receiver operating characteristic curve (AUC) of 0.92 (min: 0.8 – max: 0.98) for predicting the measured constructs. BNs outperformed alternative CAT approaches with a 5% (min: 0.01% – max: 9%) average increase in the accuracy of predicting the responses to unanswered question items.

Quantum Distributed Complexity of Set Disjointness on a Line

Given \( x,y\in \lbrace 0,1\rbrace ^n \) , Set Disjointness consists in deciding whether \( x_i=y_i=1 \) for some index \( i \in [n] \) . We study the problem of computing this function in a distributed computing scenario in which the inputs \( x \) and \( y \) are given to the processors at the two extremities of a path of length \( d \) . Each vertex of the path has a quantum processor that can communicate with each of its neighbours by exchanging \( \operatorname{O}(\log n) \) qubits per round. We are interested in the number of rounds required for computing Set Disjointness with constant probability bounded away from \( 1/2 \) . We call this problem “Set Disjointness on a Line”. Set Disjointness on a Line was introduced by Le Gall and Magniez [ 14 ] for proving lower bounds on the quantum distributed complexity of computing the diameter of an arbitrary network in the CONGEST model. However, they were only able to provide a lower bound when the local memory used by the processors on the intermediate vertices of the path is severely limited. More precisely, their bound applies only when the local memory of each intermediate processor consists of \( \operatorname{O}(\log n) \) qubits. In this work, we prove an unconditional lower bound of \( \widetilde{\Omega }\big (\sqrt [3]{n d^2}+\sqrt {n} \, \big) \) rounds for Set Disjointness on a Line with \( d + 1 \) processors. This is the first non-trivial lower bound when there is no restriction on the memory used by the processors. The result gives us a new lower bound of \( \widetilde{\Omega } \big (\sqrt [3]{n\delta ^2}+\sqrt {n} \, \big) \) on the number of rounds required for computing the diameter \( \delta \) of any \( n \) -node network with quantum messages of size \( \operatorname{O}(\log n) \) in the CONGEST model. We draw a connection between the distributed computing scenario above and a new model of query complexity. In this model, an algorithm computing a bi-variate function \( f \) (such as Set Disjointness) has access to the inputs \( x \) and \( y \) through two separate oracles \( {\mathcal {O}}_x \) and \( {\mathcal {O}}_y \) , respectively. The restriction is that the algorithm is required to alternately make \( d \) queries to \( {\mathcal {O}}_x \) and \( d \) queries to \( {\mathcal {O}}_y \) , with input-independent computation in between queries. The model reflects a “switching delay” of \( d \) queries between a “round” of queries to \( x \) and the following “round” of queries to \( y \) . The information-theoretic technique we use for deriving the round lower bound for Set Disjointness on a Line also applies to the number of rounds in this query model. We provide an algorithm for Set Disjointness in this query model with round complexity that matches the round lower bound stated above, up to a polylogarithmic factor. This presents a barrier for obtaining a better round lower bound for Set Disjointness on the Line. At the same time, it hints at the possibility of better communication protocols for the problem.

Secure UAV-Aided Mobile Edge Computing for IoT: A Review

As the Internet of Things (IoT) ecosystem evolves, innovative applications with stringent demands with respect to latency will emerge. To handle computation-intensive tasks in a timely manner, data offloading to Mobile Edge Computing (MEC) servers has been suggested. On the other hand, prospective IoT networks are expected to include Unmanned Aerial Vehicles (UAVs) to enhance coverage and connectivity, while retaining reliable communication links with ground nodes in urban, suburban, and rural terrain. Nevertheless, the evolution of UAV-aided MEC-enabled IoT presupposes the mitigation of security threats through the implementation of efficient and robust countermeasures. As UAVs inherently have certain limitations in terms of energy, computational, and memory resources, designing lightweight security solutions is required. This paper provides an overview of the UAV-aided MEC-enabled IoT and a detailed presentation of use cases and application scenarios, where security is of utmost importance. Subsequently, up-to-date research works on security solutions for the UAV-aided MEC-enabled IoT are comprehensively presented. To this end, the adoption of information-theoretic techniques that ensure adequate Physical-Layer Security (PLS) is discussed along with sophisticated security approaches based on emerging technologies, such as Blockchain and Machine Learning (ML). In addition, research studies on software- and hardware-based methods for the identification and authentication of network nodes are presented. Finally, this paper provides future perspectives in this research domain, stimulating further work.

Generative trapdoors for public key cryptography based on automatic entropy optimization

Trapdoor is a key component of public key cryptography design which is the essential security foundation of modern cryptography. Normally, the traditional way in designing a trapdoor is to identify a computationally hard problem, such as the NPC problems. So the trapdoor in a public key encryption mechanism turns out to be a type of limited resource. In this paper, we generalize the methodology of adversarial learning model in artificial intelligence and introduce a novel way to conveniently obtain sub-optimal and computationally hard trapdoors based on the automatic information theoretic search technique. The basic routine is constructing a generative architecture to search and discover a probabilistic reversible generator which can correctly encoding and decoding any input messages. The architecture includes a trapdoor generator built on a variational autoencoder (VAE) responsible for searching the appropriate trapdoors satisfying a maximum of entropy, a random message generator yielding random noise, and a dynamic classifier taking the results of the two generator. The evaluation of our construction shows the architecture satisfying basic indistinguishability of outputs under chosen-plaintext attack model (CPA) and high efficiency in generating cheap trapdoors.

Novel Acoustic Snapshot of a Sarawak Forest

Auditory signals are often used by forest species to attract mates, define and defend territories, and locate prey, and thus these signals may be monitored and used to estimate species presence, richness and acoustic complexity of a patch of habitat. We analyzed recordings from a biodiversity hotspot in the rainforests of Batang Ai National Park in Sarawak, Malaysian Borneo. Three recording sites were established in the forest understory and continuous recordings were made for an acoustic snapshot of approximately 40 h. From these recordings, the bioacoustic index (BI) and acoustic complexity index (ACI) were computed. These acoustic indices exhibited clear periodicity with periods on the order of >6 h. The ACI and BI time series also showed oscillations, with peaks separated by 12 h corresponding to the alternation between bird and frog activity during the day and night, respectively. We quantified the relationships between the acoustic index values and anuran and avian richness, and environmental variables (rainfall intensity and time of day) using correlative and information theoretic techniques. ACI and BI were moderately positively and negatively correlated with rainfall intensity, respectively. ACI and BI were also weakly-to-moderately correlated to species richness, but with mixed directions between recording sites. However, the correlations and mutual information values, a model-free estimator of the relationship strength of random variables, were highest for the relationships between ACI and BI with respect to the rate of individual frog calls, rather than species richness alone. We conclude that acoustic indices are most useful for monitoring ecological dynamics on timescales longer than 6 h. We suggest that acoustic indices are best applied to studying changes in acoustic activity at the level of ecological populations rather than for estimating species richness, as they have been commonly applied in the past.

Probing modified gravity theories with multiple measurements of high-redshift quasars

ABSTRACT In this paper, we quantify the ability of multiple measurements of high-redshift quasars to constrain several theories of modified gravity, including the Dvali–Gabadadze–Porrati braneworld scenario, generalized Chaplygin gas, f(T) modified gravity, and modified polytropic Cardassian model. Recently released sample of 1598 quasars with X-ray and ultraviolet flux measurements in the redshift range of 0.036 ≤ z ≤ 5.1003, as well as a compilation of 120 intermediate-luminosity radio quasars covering the redshift of 0.46 < z < 2.76 are respectively used as standard probes at higher redshifts. For all considered modified gravity theories, our results show that there is still some possibility that the standard Λ cold dark matter (ΛCDM) scenario might not be the best cosmological model preferred by the current quasar observations. In order to improve cosmological constraints, the quasar data are also combined with the latest observations of baryon acoustic oscillations, which strongly complement the constraints. Finally, we discuss the support given by the data to modified gravity theories, applying different information theoretic techniques like the Akaike information criterion, Bayesian information criterion, and Jensen–Shannon divergence.

A Finite Time Analysis of Temporal Difference Learning with Linear Function Approximation

Temporal difference learning (TD) is a simple iterative algorithm widely used for policy evaluation in Markov reward processes. Bhandari et al. prove finite time convergence rates for TD learning with linear function approximation. The analysis follows using a key insight that establishes rigorous connections between TD updates and those of online gradient descent. In a model where observations are corrupted by i.i.d. noise, convergence results for TD follow by essentially mirroring the analysis for online gradient descent. Using an information-theoretic technique, the authors also provide results for the case when TD is applied to a single Markovian data stream where the algorithm’s updates can be severely biased. Their analysis seamlessly extends to the study of TD learning with eligibility traces and Q-learning for high-dimensional optimal stopping problems.

Detection of local mixing in time-series data using permutation entropy.

Mixing of neighboring data points in a sequence is a common, but understudied, effect in physical experiments. This can occur in the measurement apparatus (if material from multiple time points is pulled into a measurement chamber simultaneously, for instance) or the system itself, e.g., via diffusion of isotopes in an ice sheet. We propose a model-free technique to detect this kind of local mixing in time-series data using an information-theoretic technique called permutation entropy. By varying the temporal resolution of the calculation and analyzing the patterns in the results, we can determine whether the data are mixed locally, and on what scale. This can be used by practitioners to choose appropriate lower bounds on scales at which to measure or report data. After validating this technique on several synthetic examples, we demonstrate its effectiveness on data from a chemistry experiment, methane records from Mauna Loa, and an Antarctic ice core.

Entrack: Probabilistic Spherical Regression with Entropy Regularization for Fiber Tractography

White matter tractography, based on diffusion-weighted magnetic resonance images, is currently the only available in vivo method to gather information on the structural brain connectivity. The low resolution of diffusion MRI data suggests to employ probabilistic methods for streamline reconstruction, i.e., for fiber crossings. We propose a general probabilistic model for spherical regression based on the Fisher-von-Mises distribution, which efficiently estimates maximum entropy posteriors of local streamline directions with machine learning methods. The optimal precision of posteriors for streamlines is determined by an information-theoretic technique, the expected log-posterior agreement concept. It relies on the requirement that the posterior distributions of streamlines, inferred on retest measurements of the same subject, should yield stable results within the precision determined by the noise level of the data source.

Dynamical complexity in Swarm electron density time series using Block entropy

Our goal in this study is to investigate the dynamical complexity of the electron density profiles in the topside ionosphere as measured by the Swarm mission, employing the use of symbolic information-theoretic techniques. We perform a Block entropy analysis for a time interval associated with the most intense magnetic storm of solar cycle 24, which occurred on 17 March 2015. We produce entropy maps for varying degrees of magnetospheric disturbance, resolving the different effects that the various geomagnetic activity levels have in the dynamics of the complex magnetosphere-ionosphere coupling system. Understanding the impact of these effects on the ionospheric plasma constitutes a crucial factor for the functionality of the modern technological infrastructure operating around the Earth and, thus, human welfare.

Investigating the implications of shifting baseline syndrome on conservation

Abstract Shifting baseline syndrome (SBS) describes a persistent downgrading of perceived ‘normal’ environmental conditions with every sequential generation, leading to under‐estimation of the true magnitude of long‐term environmental change on a global scale. The presence of SBS should be considered when local ecological knowledge and participatory techniques are involved in conservation target‐setting. However, despite increasing recognition of the phenomenon, there is little empirical evidence for SBS. Here we provide evidence of SBS, and the first empirical investigation of the impacts of SBS on public perceptions of conservation need. Large‐scale online questionnaires were used to collect public perceptions of long‐term biological change regarding 10 UK bird species, as well as demographic information and measures of knowledge and experience of the local environment (n = 330). A paired data approach compared social perceptions to a large‐scale longitudinal biological dataset. Using information theoretic and model selection techniques, we estimate the relative importance of multiple demographic, social and psychological predictors of SBS. We provide a framework for investigating evidence of SBS and its impacts on perceptions of conservation need for species in decline. Evidence of generational amnesia was found as an age‐related difference in perceptions of past ecological conditions. The perceptions of older participants had significantly higher agreement with biological data than the perceptions of younger participants. Our results therefore support the expectation that younger, less experienced people are less aware of historical ecological conditions and show greater evidence of SBS. We also present evidence of a negative impact of SBS on future conservation, as older people were more likely than younger people to perceive a greater need for conservation action for three declining species. Our research supports the need to encourage greater intergenerational communication and increase experience of local nature. Discovering evidence of SBS in public perceptions of species experienced within everyday life demonstrates SBS as a pervasive social issue with the potential to impact public perceptions of local nature. A free Plain Language Summary can be found within the Supporting Information of this article.

Assessing optimal water quality monitoring network in road construction using integrated information-theoretic techniques

The environmental impacts of road construction on the aquatic environment necessitate the monitoring of receiving water quality. The main contribution of the paper is developing a feasible methodology for spatial optimization of the water quality monitoring network (WQMN) in surface water during road construction using the field data. First, using the Canadian Council of Ministers of the Environment (CCME) method, the water quality index (WQI) was computed in each potential monitoring station during construction. Then, the integrated form of the information-theoretic techniques consists of the transinformation entropy (TE), and the value of information (VOI) were calculated for the potential stations. To achieve the optimal WQMNs, the Non-dominated Sorting Genetic Algorithm II and III (NSGA-II, and III) based multi-objective optimization models were developed considering three objective functions, including i) minimizing the number of stations, ii) maximizing the VOI in the selected network, and iii) minimizing redundant information for the selected nodes. Finally, three multi-criteria decision-making models, including Technique for Order Preference by Similarity to Ideal Solution (TOPSIS), Preference Ranking Organisation Method for Enrichment Evaluations (PROMETHEE), and Analytical Hierarchy Process (AHP) were utilized for choosing the best alternative among Pareto optimal solutions considering various weighing scenarios assigned to criteria. The applicability of the presented methodology was assessed in a 22 km long road construction site in southern Norway. The results deliver significant knowledge for decision-makers on establishing a robust WQMN in surface water during road construction projects.

Direct and Indirect Effects-An Information Theoretic Perspective.

Information theoretic (IT) approaches to quantifying causal influences have experienced some popularity in the literature, in both theoretical and applied (e.g., neuroscience and climate science) domains. While these causal measures are desirable in that they are model agnostic and can capture non-linear interactions, they are fundamentally different from common statistical notions of causal influence in that they (1) compare distributions over the effect rather than values of the effect and (2) are defined with respect to random variables representing a cause rather than specific values of a cause. We here present IT measures of direct, indirect, and total causal effects. The proposed measures are unlike existing IT techniques in that they enable measuring causal effects that are defined with respect to specific values of a cause while still offering the flexibility and general applicability of IT techniques. We provide an identifiability result and demonstrate application of the proposed measures in estimating the causal effect of the El Niño–Southern Oscillation on temperature anomalies in the North American Pacific Northwest.

An Overview of Capacity Results for Synchronization Channels

Synchronization channels, such as the well-known deletion channel, are surprisingly harder to analyze than memoryless channels, and they are a source of many fundamental problems in information theory and theoretical computer science. One of the most basic open problems regarding synchronization channels is the derivation of an exact expression for their capacity. Unfortunately, most of the classic information-theoretic techniques at our disposal fail spectacularly when applied to synchronization channels. Therefore, new approaches must be considered to tackle this problem. This survey gives an account of the great effort made over the past few decades to better understand the (broadly defined) capacity of synchronization channels, including both the main results and the novel techniques underlying them. Besides the usual notion of channel capacity, we also discuss the zero-error capacity of adversarial synchronization channels.

Unifying thermodynamic uncertainty relations

We introduce a new technique to bound the fluctuations exhibited by a physical system, based on the Euclidean geometry of the space of observables. Through a simple unifying argument, we derive a sweeping generalization of so-called thermodynamic uncertainty relations (TURs). We not only strengthen the bounds but extend their realm of applicability and in many cases prove their optimality, without resorting to large deviation theory or information-theoretic techniques. In particular, we find the best TUR based on entropy production alone. We also derive a periodic uncertainty principle of which previous known bounds for periodic or stationary Markov chains known in the literature appear as limit cases. From it a novel bound for stationary Markov processes is derived, which surpasses previous known bounds. Our results exploit the non-invariance of the system under a symmetry which can be other than time reversal and thus open a wide new spectrum of applications.

A Near-Optimal Change-Detection Based Algorithm for Piecewise-Stationary Combinatorial Semi-Bandits

We investigate the piecewise-stationary combinatorial semi-bandit problem. Compared to the original combinatorial semi-bandit problem, our setting assumes the reward distributions of base arms may change in a piecewise-stationary manner at unknown time steps. We propose an algorithm, GLR-CUCB, which incorporates an efficient combinatorial semi-bandit algorithm, CUCB, with an almost parameter-free change-point detector, the Generalized Likelihood Ratio Test (GLRT). Our analysis shows that the regret of GLR-CUCB is upper bounded by O(√NKT log T), where N is the number of piecewise-stationary segments, K is the number of base arms, and T is the number of time steps. As a complement, we also derive a nearly matching regret lower bound on the order of Ω(√NKT), for both piecewise-stationary multi-armed bandits and combinatorial semi-bandits, using information-theoretic techniques and judiciously constructed piecewise-stationary bandit instances. Our lower bound is tighter than the best available regret lower bound, which is Ω(√T). Numerical experiments on both synthetic and real-world datasets demonstrate the superiority of GLR-CUCB compared to other state-of-the-art algorithms.