Information-theoretic Bound Research Articles

Estimator of entropy production for partially accessible Markov networks based on the observation of blurred transitions.

A central task in stochastic thermodynamics is the estimation of entropy production for partially accessible Markov networks. We establish an effective transition-based description for such networks with transitions that are not distinguishable and therefore blurred for an external observer. We demonstrate that, in contrast to a description based on fully resolved transitions, this effective description is typically non-Markovian at any point in time. Starting from an information-theoretic bound, we derive an operationally accessible entropy estimator for this observation scenario. We illustrate the operational relevance and the quality of this entropy estimator with a numerical analysis of various representative examples.

An Information-Theoretic Bound on p-Values for Detecting Communities Shared between Weighted Labeled Graphs

Extraction of subsets of highly connected nodes (“communities” or modules) is a standard step in the analysis of complex social and biological networks. We here consider the problem of finding a relatively small set of nodes in two labeled weighted graphs that is highly connected in both. While many scoring functions and algorithms tackle the problem, the typically high computational cost of permutation testing required to establish the p-value for the observed pattern presents a major practical obstacle. To address this problem, we here extend the recently proposed CTD (“Connect the Dots”) approach to establish information-theoretic upper bounds on the p-values and lower bounds on the size and connectedness of communities that are detectable. This is an innovation on the applicability of CTD, broadening its use to pairs of graphs.

Verification of Information Thermodynamics in a Trapped Ion System.

Information thermodynamics has developed rapidly over past years, and the trapped ions, as a controllable quantum system, have demonstrated feasibility to experimentally verify the theoretical predictions in the information thermodynamics. Here, we address some representative theories of information thermodynamics, such as the quantum Landauer principle, information equality based on the two-point measurement, information-theoretical bound of irreversibility, and speed limit restrained by the entropy production of system, and review their experimental demonstration in the trapped ion system. In these schemes, the typical physical processes, such as the entropy flow, energy transfer, and information flow, build the connection between thermodynamic processes and information variation. We then elucidate the concrete quantum control strategies to simulate these processes by using quantum operators and the decay paths in the trapped-ion system. Based on them, some significantly dynamical processes in the trapped ion system to realize the newly proposed information-thermodynamic models is reviewed. Although only some latest experimental results of information thermodynamics with a single trapped-ion quantum system are reviewed here, we expect to find more exploration in the future with more ions involved in the experimental systems.

Information-Theoretic Bounds on Quantum Advantage in Machine Learning.

We study the performance of classical and quantum machine learning (ML) models in predicting outcomes of physical experiments. The experiments depend on an input parameter x and involve execution of a (possibly unknown) quantum process E. Our figure of merit is the number of runs of E required to achieve a desired prediction performance. We consider classical ML models that perform a measurement and record the classical outcome after each run of E, and quantum ML models that can access E coherently to acquire quantum data; the classical or quantum data are then used to predict the outcomes of future experiments. We prove that for any input distribution D(x), a classical ML model can provide accurate predictions on average by accessing E a number of times comparable to the optimal quantum ML model. In contrast, for achieving an accurate prediction on all inputs, we prove that the exponential quantum advantage is possible. For example, to predict the expectations of all Pauli observables in an n-qubit system ρ, classical ML models require 2^{Ω(n)} copies of ρ, but we present a quantum ML model using only O(n) copies. Our results clarify where the quantum advantage is possible and highlight the potential for classical ML models to address challenging quantum problems in physics and chemistry.

Nested cover-free families for unbounded fault-tolerant aggregate signatures

Aggregate signatures are used to create one short proof of authenticity and integrity from a set of digital signatures. However, one invalid signature in the set invalidates the entire aggregate, giving no information on which signatures are valid. Hartung et al. (2016) [6] propose a fault-tolerant aggregate signature scheme based on combinatorial group testing. Given a bound d on the number of invalid signatures among n signatures to be aggregated, this scheme uses d-cover-free families to determine which signatures are invalid. These combinatorial structures guarantee a moderate increase on the size of the aggregate signature that can reach the best possible compression ratio of O(nlog⁡n), for fixed d, coming from an information theoretical bound. The case where the total number of signatures grows dynamically (unbounded scheme) was not satisfactorily solved in their original paper, since explicit constructions had constant compression ratios. In the present paper, we propose efficient solutions for the unbounded scheme, relying on sequences of d-cover-free families that we call nested families. Some of our constructions yield high compression ratio close to the best known upper bound. We also propose the use of (d,λ)-cover-free families to support the loss of up to λ−1 parts of the aggregate.

On the Information Capacity of Layered Undirected Unicast Networks

An important unsolved conjecture in network coding theory states that network coding has no rate benefit over routing in undirected unicast networks. Recently, a first non-trivial information-theoretic bound, called partition bound, was characterized for symmetric rate in undirected unicast networks, and it was shown that the bound is achievable for two classes of networks by a routing scheme. In this letter, we focus on the problem of characterization of networks for which the partition bound is tight. In particular, we consider layered undirected unicast networks. We show a routing scheme achieving the partition bound for a class of 3-layer networks and thus establish explicit information capacity expression and prove the conjecture for the class of networks. We also show that there exists a 4-layer network for which the partition bound cannot be achieved by an optimal routing scheme.

Single-atom verification of the information-theoretical bound of irreversibility at the quantum level

Quantitative measure of disorder or randomness based on the entropy production characterizes thermodynamical irreversibility, which is relevant to the conventional second law of thermodynamics. Here we report, in a quantum mechanical fashion, the first theoretical prediction and experimental exploration of an information-theoretical bound on the entropy production. Our theoretical model consists of a simplest two-level dissipative system driven by a purely classical field, and under the Markovian dissipation, we find that such an information-theoretical bound, not fully validating quantum relaxation processes, strongly depends on the drive-to-decay ratio and the initial state. Furthermore, we carry out experimental verification of this information-theoretical bound by means of a single spin embedded in an ultracold trapped $^{40}$Ca$^{+}$ ion. Our finding, based on a two-level model, is fundamental to any quantum thermodynamical process and indicates much difference and complexity in quantum thermodynamics with respect to the conventionally classical counterpart.

Learning Force Fields from Stochastic Trajectories

When monitoring the dynamics of stochastic systems, such as interacting particles agitated by thermal noise, disentangling deterministic forces from Brownian motion is challenging. Indeed, we show that there is an information-theoretic bound, the capacity of the system when viewed as a communication channel, that limits the rate at which information about the force field can be extracted from a Brownian trajectory. This capacity provides an upper bound to the system's entropy production rate, and quantifies the rate at which the trajectory becomes distinguishable from pure Brownian motion. We propose a practical and principled method, Stochastic Force Inference, that uses this information to approximate force fields and spatially variable diffusion coefficients. It is data efficient, including in high dimensions, robust to experimental noise, and provides a self-consistent estimate of the inference error. In addition to forces, this technique readily permits the evaluation of out-of-equilibrium currents and the corresponding entropy production with a limited amount of data.

Input perturbations for adaptive control and learning

This paper studies adaptive algorithms for simultaneous regulation (i.e., control) and estimation (i.e., learning) of Multiple Input Multiple Output (MIMO) linear dynamical systems. It proposes practical, easy to implement control policies based on perturbations of input signals. Such policies are shown to achieve a worst-case regret that scales as the square-root of the time horizon, and holds uniformly over time. Further, it discusses specific settings where such greedy policies attain the information theoretic lower bound of logarithmic regret. To establish the results, recent advances on self-normalized martingales together with a novel method of policy decomposition are leveraged.

Information-Theoretical Bound of the Irreversibility in Thermal Relaxation Processes.

We establish that entropy production, which is crucial to the characterization of thermodynamic irreversibility, is obtained through a variational principle involving the Kulback-Leibler divergence. A simple application of this representation leads to an information-theoretical bound on entropy production in thermal relaxation processes; this is a stronger inequality than the conventional second law of thermodynamics. This bound is also interpreted as a constraint on the possible path of a thermal relaxation process in terms of information geometry. Our results reveal a hidden universal law inherent to general thermal relaxation processes.

Rapid NMR Relaxation Measurements Using Optimal Nonuniform Sampling of Multidimensional Accordion Data Analyzed by a Sparse Reconstruction Method.

Nonuniform sampling (NUS) of multidimensional NMR data offers significant time savings while improving spectral resolution or increasing sensitivity per unit time. However, NUS has not been widely used for quantitative analysis because of the nonlinearity of most methods used to model NUS data, which leads to problems in estimating signal intensities, relaxation rate constants, and their error bounds. Here, we present an approach that avoids these limitations by combining accordion spectroscopy and NUS in the indirect dimensions of multidimensional spectra and then applying sparse exponential mode analysis, which is well suited for analyzing accordion-type relaxation data in a NUS context. By evaluating the Cramér-Rao lower bound of the variances of the estimated relaxation rate constants, we achieve a robust benchmark for the underlying reconstruction model. Furthermore, we design NUS schemes optimized with respect to the information theoretical lower bound of the error in the parameters of interest, given a specified number of sampling points. The accordion-NUS method compares favorably with conventional relaxation experiments in that it produces identical results, within error, while shortening the length of the experiment by an order of magnitude. Thus, our approach enables rapid acquisition of NMR relaxation data for optimized use of spectrometer time or accurate measurements on samples of limited lifetime.

Nearly Constant Tile Complexity for any Shape in Two-Handed Tile Assembly

Tile self-assembly is a well-studied theoretical model of geometric computation based on nanoscale DNA-based molecular systems. Here, we study the two-handed tile self-assembly model or 2HAM at general temperatures, in contrast with prior study limited to small constant temperatures, leading to surprising results. We obtain constructions at larger (i.e., hotter) temperatures that disprove prior conjectures and break well-known bounds for low-temperature systems via new methods of temperature-encoded information. In particular, for all $$n \in \mathbb {N}$$ , we assemble $$n \times n$$ squares using $$O(2^{\log ^*{n}})$$ tile types, thus breaking the well-known information theoretic lower bound of Rothemund and Winfree. Using this construction, we then show how to use the temperature to encode general shapes and construct them at scale with $$O(2^{\log ^*{K}})$$ tiles, where K denotes the Kolmogorov complexity of the target shape. Following, we refute a long-held conjecture by showing how to use temperature to construct $$n \times O(1)$$ rectangles using only $$O(\log {n}/\log \log {n})$$ tile types. We also give two small systems to generate nanorulers of varying length based solely on varying the system temperature. These results constitute the first real demonstration of the power of high temperature systems for tile assembly in the 2HAM. This leads to several directions for future explorations which we discuss in the conclusion.

Physical-Layer Multicasting Design for Downstream G.fast DSL Transmission

This paper studies the physical-layer multicasting design for downstream G.fast digital subscriber line (DSL) transmission, which corresponds to a multi-user multi-tone (i.e., multi-carrier) scenario. The design goal is to maximize the weighted-sum-group-rate (WSGR) under per-line power constraints. First, as an information-theoretic upper bound, full-rank precoding-based multicasting is considered with joint channel coding across tones. For a single multicast group, this problem corresponds to a non-linear convex semidefinite program (SDP), which is coupled across tones. To reduce the computational complexity, a Lagrange dual decomposition method is developed. This approach is then extended toward multiple multicast groups based on difference-of-convex (DC) programming. Furthermore, a practical multicasting scheme is considered based on rank-one single-stream precoding and independent per-tone channel coding. For this case, instead of relying on computationally complex semidefinite relaxation, a successive convex approximation-based trust-region algorithm is developed. Finally, the simulations of a G.fast cable binder show that the practical multicasting scheme operates close to the information-theoretic multicasting upper bound.

Decoding from Pooled Data: Sharp Information-Theoretic Bounds

Consider a population consisting of n individuals, each of whom has one of d types (e.g. their blood type, in which case d=4). We are allowed to query this database by specifying a subset of the population, and in response we observe a noiseless histogram (a d-dimensional vector of counts) of types of the pooled individuals. This measurement model arises in practical situations such as pooling of genetic data and may also be motivated by privacy considerations. We are interested in the number of queries one needs to unambiguously determine the type of each individual. In this paper, we study this information-theoretic question under the random, dense setting where in each query, a random subset of individuals of size proportional to n is chosen. This makes the problem a particular example of a random constraint satisfaction problem (CSP) with a "planted" solution. We establish almost matching upper and lower bounds on the minimum number of queries m such that there is no solution other than the planted one with probability tending to 1 as n tends to infinity. Our proof relies on the computation of the exact "annealed free energy" of this model in the thermodynamic limit, which corresponds to the exponential rate of decay of the expected number of solution to this planted CSP. As a by-product of the analysis, we show an identity of independent interest relating the Gaussian integral over the space of Eulerian flows of a graph to its spanning tree polynomial.

Information-Theoretic Bounds and Phase Transitions in Clustering, Sparse PCA, and Submatrix Localization

We study the problem of detecting a structured, low-rank signal matrix corrupted with additive Gaussian noise. This includes clustering in a Gaussian mixture model, sparse PCA, and submatrix localization. Each of these problems is conjectured to exhibit a sharp information-theoretic threshold, below which the signal is too weak for any algorithm to detect. We derive upper and lower bounds on these thresholds by applying the first and second moment methods to the likelihood ratio between these “planted models” and null models where the signal matrix is zero. For sparse PCA and submatrix localization, we determine this threshold exactly in the limit where the number of blocks is large or the signal matrix is very sparse; for the clustering problem, our bounds differ by a factor of $\sqrt {2}$ when the number of clusters is large. Moreover, our upper bounds show that for each of these problems there is a significant regime where reliable detection is information-theoretically possible but where known algorithms such as PCA fail completely, since the spectrum of the observed matrix is uninformative. This regime is analogous to the conjectured “hard but detectable” regime for community detection in sparse graphs.

Models and Information-Theoretic Bounds for Nanopore Sequencing

Nanopore sequencing is an emerging new technology for sequencing DNA, which can read long fragments of DNA (~50,000 bases) in contrast to most current short-read sequencing technologies which can only read hundreds of bases. While nanopore sequencers can acquire long reads, the high error rates (20%-30%) pose a technical challenge. In a nanopore sequencer, a DNA is migrated through a nanopore and current variations are measured. The DNA sequence is inferred from this observed current pattern using an algorithm called a base-caller. In this paper, we propose a mathematical model for the "channel" from the input DNA sequence to the observed current, and calculate bounds on the information extraction capacity of the nanopore sequencer. This model incorporates impairments like (non-linear) inter-symbol interference, deletions, as well as random response. These information bounds have two-fold application: (1) The decoding rate with a uniform input distribution can be used to calculate the average size of the plausible list of DNA sequences given an observed current trace. This bound can be used to benchmark existing base-calling algorithms, as well as serving a performance objective to design better nanopores. (2) When the nanopore sequencer is used as a reader in a DNA storage system, the storage capacity is quantified by our bounds.

Information-Theoretic Bounds and Approximations in Neural Population Coding.

While Shannon's mutual information has widespread applications in many disciplines, for practical applications it is often difficult to calculate its value accurately for high-dimensional variables because of the curse of dimensionality. This article focuses on effective approximation methods for evaluating mutual information in the context of neural population coding. For large but finite neural populations, we derive several information-theoretic asymptotic bounds and approximation formulas that remain valid in high-dimensional spaces. We prove that optimizing the population density distribution based on these approximation formulas is a convex optimization problem that allows efficient numerical solutions. Numerical simulation results confirmed that our asymptotic formulas were highly accurate for approximating mutual information for large neural populations. In special cases, the approximation formulas are exactly equal to the true mutual information. We also discuss techniques of variable transformation and dimensionality reduction to facilitate computation of the approximations.

On the DoF Region of the MIMO Gaussian Two-User Interference Channel With an Instantaneous Relay

This paper studies inner and outer bounds on the degrees of freedom (DoF) region of the multi-antenna two-user Gaussian interference channel with an instantaneous relay (IR) or relay without delay. It is assumed that the two transmitters and the two receivers have $\mathop {\mathrm {\mathsf {M}}}\nolimits$ antennas, while the IR receives through $\mathop {\mathrm {\mathsf {N_{r}}}}\nolimits$ antennas and transmits through $\mathop {\mathrm {\mathsf {N_{t}}}}\nolimits$ antennas. In the proposed achievable scheme, which generalizes a known one for the case $\mathop {\mathrm {\mathsf {M}}}\nolimits = \mathop {\mathrm {\mathsf {N_{r}}}}\nolimits = \mathop {\mathrm {\mathsf {N_{t}}}}\nolimits$ to any $(\mathop {\mathrm {\mathsf {M}}}\nolimits, \mathop {\mathrm {\mathsf {N_{r}}}}\nolimits, \mathop {\mathrm {\mathsf {N_{t}}}}\nolimits)$ , the IR performs memoryless linear operations on its received signal so as to neutralize interference at the receivers, and the beamforming matrices used by the IR and the transmitters are jointly designed. This joint design strictly outperforms known achievable schemes. Two outer bounds are derived. An information theoretic outer bound is obtained by giving the receivers or the IR genie side information, so that the DoF region of the resulting enhanced channel is known; this converse is valid for any type of processing at the IR and shows the optimality of the proposed achievable scheme for some $(\mathop {\mathrm {\mathsf {M}}}\nolimits, \mathop {\mathrm {\mathsf {N_{r}}}}\nolimits, \mathop {\mathrm {\mathsf {N_{t}}}}\nolimits)$ . A linear processing outer bound is obtained when the IR is restricted to performs linear operations, without any memoryless restriction, on its received signal and shows the optimality of the proposed achievable scheme among all linear processing schemes at the IR. As a result of independent interest, the DoF region of the classical multi-antenna two-user Gaussian interference channel without relay when the channel matrices can have any structure is also derived, which generalized available DoF region results that were derived under certain assumptions on the structure of the channel matrices.

Network Capacity Under Traffic Symmetry: Wireline and Wireless Networks

The problem of designing near optimal strategies for multiple unicast traffic in wireline networks is wide open; however, channel symmetry or traffic symmetry can be leveraged to show that routing can achieve with a polylogarithmic approximation factor of the edge-cut bound. For the same problem, the edge-cut bound is known to only upper bound rates of routing flows and unlike the information theoretic cut-set bound, it does not upper bound (capacity-achieving) information rates with general strategies. In this paper, we demonstrate that under channel or traffic symmetry, the edge-cut bound upper-bounds general information rates, thus providing a capacity approximation result. The key technique is a combinatorial result relating edge-cut bounds to generalized network sharing bounds. Finally, we generalize the results to wireless networks via an intermediary class of combinatorial graphs known as polymatroidal networks-our main result is that a natural architecture separating the physical and networking layers is near optimal when the traffic is symmetric among source-destination pairs, even when the channel is asymmetric (due to asymmetric power constraints, or prior frequency allocation like frequency division duplexing). This result is complementary to an earlier work of two of the authors proving a similar result under channel symmetry.

Two-Flow Capacity Region of the COPE Principle for Wireless Butterfly Networks With Broadcast Erasure Channels

This paper characterizes the full capacity region of the COPE principle for 2-flow wireless butterfly networks with broadcast packet erasure channels (PECs). The capacity results in this paper allow for random overhearing with arbitrary overhearing probabilities, arbitrary scheduling policies, network-wide channel state information (CSI) feedback after each transmission, and potential use of nonlinear network codes. An information-theoretic outer bound is derived that takes into account the delayed CSI feedback of the underlying broadcast packet erasure channels. For the achievability, this paper proposes a new class of linear network codes, named as the space-based linear network coding (SBLNC), that achieves the capacity outer bound. Further, the proposed outer and inner bounds are later generalized for the setting in which a transmission may be heard by its 2-hop neighbor(s), the so-called opportunistic routing scenario. When allowing the possibility of opportunistic routing, the proposed inner and outer bounds do not always meet. Numerical experiments, however, show that the relative gap of the two bounds is less than 0.08% in average. The proposed bounds thus tightly bracket the capacity region even when combining the COPE principle with opportunistic routing.