Information-theoretic Lower Bound Research Articles

Slowing down top trees for better worst-case compression

We consider the top tree compression scheme introduced by Bille et al. [ICALP 2013] and construct an infinite family of trees on n nodes labeled from an alphabet of size σ, for which the size of the top DAG is Θ(nlogσ⁡nlog⁡logσ⁡n). Our construction matches a previously known upper bound and exhibits a weakness of this scheme, as the information-theoretic lower bound is Ω(nlogσ⁡n). Lohrey et al. [IPL 2019] designed a more involved version of the original algorithm achieving the lower bound. We show that this can be also guaranteed by a very minor modification of the original scheme: informally, one only needs to ensure that different parts of the tree are not compressed too quickly. Arguably, our version is more uniform, and in particular, the compression procedure is oblivious to the value of σ.

Combinatorial-Probabilistic Trade-Off: P-Values of Community Property Test in the Stochastic Block Models

In this paper, we propose an inferential framework testing the general community combinatorial properties of the stochastic block model. Instead of estimating the community assignments, we aim to test the hypothesis on whether a certain community property is satisfied and provide p-values to assess the statistical uncertainty. For instance, we propose to test whether a given set of nodes belong to the same community or whether different network communities have the same size. We present a general inferential framework that can be applied to all symmetric community properties. To ease the challenges caused by the combinatorial nature of community properties, we develop a novel shadowing bootstrap testing method. By utilizing the symmetry, our method can find a shadowing representative of the true assignment and the number of assignments to be tested in the alternative can be largely reduced. In theory, we introduce a combinatorial distance between two community classes and show a combinatorial-probabilistic trade-off phenomenon in the community property test. Our test is honest as long as the product of the combinatorial distance between two community classes and the probabilistic distance between two assignment probabilities is sufficiently large. On the other hand, we show that such a trade-off also exists in the information-theoretic lower bound of the community property test. We also implement numerical experiments on both the synthetic data and the Protein-Protein Interaction networks to show the validity of our method.

Fast quantum circuit cutting with randomized measurements

We propose a new method to extend the size of a quantum computation beyond the number of physical qubits available on a single device. This is accomplished by randomly inserting measure-and-prepare channels to express the output state of a large circuit as a separable state across distinct devices. Our method employs randomized measurements, resulting in a sample overhead that is O~(4k/ε2), where ε is the accuracy of the computation and k the number of parallel wires that are "cut" to obtain smaller sub-circuits. We also show an information-theoretic lower bound of Ω(2k/ε2) for any comparable procedure. We use our techniques to show that circuits in the Quantum Approximate Optimization Algorithm (QAOA) with p entangling layers can be simulated by circuits on a fraction of the original number of qubits with an overhead that is roughly 2O(pκ), where κ is the size of a known balanced vertex separator of the graph which encodes the optimization problem. We obtain numerical evidence of practical speedups using our method applied to the QAOA, compared to prior work. Finally, we investigate the practical feasibility of applying the circuit cutting procedure to large-scale QAOA problems on clustered graphs by using a 30-qubit simulator to evaluate the variational energy of a 129-qubit problem as well as carry out a 62-qubit optimization.

Information-Theoretic Lower Bounds to Error Probability for the Models of Noisy Discrete Source Coding and Object Classification

Information-Theoretic Lower Bounds to Error Probability for the Models of Noisy Discrete Source Coding and Object Classification

Navigating planar topologies in near-optimal space and time

We show that any embedding of a planar graph can be encoded succinctly while efficiently answering a number of topological queries near-optimally. More precisely, we build on a succinct representation that encodes an embedding of m edges within 4m bits, which is close to the information-theoretic lower bound of about 3.58m. With 4m+o(m) bits of space, we show how to answer a number of topological queries relating nodes, edges, and faces, most of them in any time in ω(1). Indeed, 3.58m+o(m) bits suffice if the graph has no self-loops and no nodes of degree one. Further, we show that with O(m) bits of space we can solve all those operations in O(1) time.

Information-Theoretic Lower Bounds for Compressive Sensing With Generative Models

It has recently been shown that for compressive sensing, significantly fewer measurements may be required if the sparsity assumption is replaced by the assumption the unknown vector lies near the range of a suitably-chosen generative model. In particular, in (Bora et al., 2017) it was shown roughly O(k log L) random Gaussian measurements suffice for accurate recovery when the generative model is an L-Lipschitz function with bounded k-dimensional inputs, and O(kdlogw) measurements suffice when the generative model is a k-input ReLU network with depth d and width w. In this paper, we establish corresponding algorithm-independent lower bounds on the sample complexity using tools from minimax statistical analysis. In accordance with the above upper bounds, our results are summarized as follows: (i) We construct an L-Lipschitz generative model capable of generating group-sparse signals, and show that the resulting necessary number of measurements is Ω(k logL); (ii) Using similar ideas, we construct ReLU networks with high depth and/or high depth for which the necessary number of measurements scales as Ω(kd logw/logn) (with output dimension n), and in some cases Ω(kd logw). As a result, we establish that the scaling laws derived in (Bora et al., 2017) are optimal or near-optimal in the absence of further assumptions.

Improved average complexity for comparison-based sorting

This paper studies the average complexity on the number of comparisons for sorting algorithms. Its information-theoretic lower bound is nlg⁡n−1.4427n+O(log⁡n). For many efficient algorithms, the first nlg⁡n term is easy to achieve and our focus is on the (negative) constant factor of the linear term. The current best value is −1.3999 for the MergeInsertion sort. Our new value is −1.4106, narrowing the gap by some 25%. An important building block of our algorithm is “two-element insertion,” which inserts two elements A and B, A<B, into a sorted sequence T. This insertion algorithm is still sufficiently simple for rigorous mathematical analysis and works well for a certain range of the length of T for which the simple binary insertion does not, thus allowing us to take a complementary approach together with the binary insertion.

Nearly Optimal Sparse Group Testing

Group testing is the process of pooling arbitrary subsets from a set of $ {n}$ items so as to identify, with a minimal number of tests, a “small” subset of $ {d}$ defective items. In “classical” non-adaptive group testing, it is known that when $ {d}$ is substantially smaller than $ {n}$ , $\Theta ( {d}\log ( {n}))$ tests are both information-theoretically necessary and sufficient to guarantee recovery with high probability. Group testing schemes in the literature that meet this bound require most items to be tested $ {\Omega }(\log ( {n}))$ times, and most tests to incorporate $ {\Omega }({{n/d}})$ items. Motivated by physical considerations, we study group testing models in which the testing procedure is constrained to be “sparse.” Specifically, we consider (separately) scenarios in which 1) items are finitely divisible and hence may participate in at most $ {\gamma } \in {o}(\log ( {n}))$ tests; or 2) tests are size-constrained to pool no more than $\rho \in {o}({{n/d}})$ items per test. For both scenarios, we provide information-theoretic lower bounds on the number of tests required to guarantee high probability recovery. In particular, one of our main results shows that $ {\gamma }$ -finite divisibility of items forces any non-adaptive group testing algorithm with the probability of recovery error at most $ {\epsilon }$ to perform at least $ {\gamma } {d}({ {n/d}})^{({1}-{5} {\epsilon })/ {\gamma }}$ tests. Analogously, for $ {\rho }$ -sized constrained tests, we show an information-theoretic lower bound of $ {\Omega }( {n}/ {\rho })$ tests for high-probability recovery–hence in both settings the number of tests required grows dramatically (relative to the classical setting) as a function of $ {n}$ . In both scenarios, we provide both randomized constructions and explicit constructions of designs with computationally efficient reconstruction algorithms that require a number of tests that is optimal up to constant or small polynomial factors in some regimes of ${{n, d,}} {\gamma }$ , and $ {\rho }$ . The randomized design/reconstruction algorithm in the $ {\rho }$ -sized test scenario is universal –independent of the value of $ {d}$ , as long as $ {\rho } \in {o}({\textbf {n/d}})$ . We also investigate the effect of unreliability/noise in test outcomes, and show that whereas the impact of noise in test outcomes can be obviated with a small (constant factor) penalty in the number of tests in the $ {\rho }$ -sized tests scenario, there is no group-testing procedure, regardless of the number of tests, that can combat noise in the $ {\gamma }$ -divisible scenario.

Pachinko

Inspired by the Japanese game Pachinko, we study simple (perfectly “inelastic” collisions) dynamics of a unit ball falling amidst point obstacles (pins) in the plane. A classic example is that a checkerboard grid of pins produces the binomial distribution, but what probability distributions result from different pin placements? In the 50–50 model, where the pins form a subset of this grid, not all probability distributions are possible, but surprisingly the uniform distribution is possible for {1,2,4,8,16} possible drop locations. Furthermore, every probability distribution can be approximated arbitrarily closely, and every dyadic probability distribution can be divided by a suitable power of 2 and then constructed exactly (along with extra “junk” outputs). In a more general model, if a ball hits a pin off center, it falls left or right accordingly. Then we prove a universality result: any distribution of n dyadic probabilities, each specified by k bits, can be constructed using O(nk2) pins, which is close to the information-theoretic lower bound of Ω(nk).

Information-Theoretic Lower Bounds for Distributed Function Computation

We derive information-theoretic converses (i.e., lower bounds) for the minimum time required by any algorithm for distributed function computation over a network of point-to-point channels with finite capacity, where each node of the network initially has a random observation and aims to compute a common function of all observations to a given accuracy with a given confidence by exchanging messages with its neighbors. We obtain the lower bounds on computation time by examining the conditional mutual information between the actual function value and its estimate at an arbitrary node, given the observations in an arbitrary subset of nodes containing that node. The main contributions include the following. First, a lower bound on the conditional mutual information via so-called small ball probabilities, which captures the dependence of the computation time on the joint distribution of the observations at the nodes, the structure of the function, and the accuracy requirement. For linear functions, the small ball probability can be expressed by Levy concentration functions of sums of independent random variables, for which tight estimates are available that lead to strict improvements over existing lower bounds on computation time. Second, an upper bound on the conditional mutual information via strong data processing inequalities, which complements and strengthens existing cutset-capacity upper bounds. Finally, a multi-cutset analysis that quantifies the loss (dissipation) of the information needed for computation as it flows across a succession of cutsets in the network. This analysis is based on reducing a general network to a line network with bidirectional links and self-links, and the results highlight the dependence of the computation time on the diameter of the network, a fundamental parameter that is missing from most of the existing lower bounds on computation time.

Information-Theoretic Lower Bounds on Bayes Risk in Decentralized Estimation

We derive lower bounds on the Bayes risk in decentralized estimation, where the estimator does not have direct access to the random samples generated conditionally on the random parameter of interest, but only to the data received from local processors that observe the samples. The received data are subject to communication constraints, due to quantization and the noise in the communication channels from the processors to the estimator. We first derive general lower bounds on the Bayes risk using information-theoretic quantities, such as mutual information, information density, small ball probability, and differential entropy. We then apply these lower bounds to the decentralized case, using strong data processing inequalities to quantify the contraction of information due to communication constraints. We treat the cases of a single processor and of multiple processors, where the samples observed by different processors may be conditionally dependent given the parameter, for noninteractive and interactive communication protocols. Our results recover and improve recent lower bounds on the Bayes risk and the minimax risk for certain decentralized estimation problems, where previously only conditionally independent sample sets and noiseless channels have been considered. Moreover, our results provide a general way to quantify the degradation of estimation performance caused by distributing resources to multiple processors, which is only discussed for specific examples in existing works.

Analysis on the TOA Tracking With DVB-T Signals for Positioning

The digital television signal is a promising signal of opportunity for ubiquitous wireless positioning. This letter analyzes the time of arrival (TOA) tracking errors of the European standard digital video broadcasting-terrestrial (DVB-T) signals in the context of positioning. The Cramer–Rao lower bound on the unsmoothed TOA estimation is derived and an information-theoretic lower bound (ITLB) is further computed on the smoothed TOA tracking errors in delay locked loop (DLL). Theoretical analysis shows that, the increase of the bandwidth of the equivalent loop filter will lead to larger DLL tracking errors. Despite different bandwidth of the loop filters, the tracking errors derived in theory are asymptotic to ITLB with the increase of signal-noise ratio. Comparatively, the theoretical results of the DLL tracking errors are slightly larger than the ITLB, which is due to the inequality and nonlinear approximation used in the analysis. Theoretical analysis has also verified the effectiveness of the DVB-T signals to be able to achieve high accuracy in positioning.

Threshold group testing with consecutive positives

Threshold group testing introduced by Damaschke (2006) is a generalization of classical group testing where a group test yields a positive (negative) outcome if it contains at least u (at most l) positive items, and an arbitrary outcome for otherwise. Motivated by applications to DNA sequencing, group testing with consecutive positives has been proposed by Balding and Torney (1997) and Colbourn (1999) where n items are linearly ordered and up to d positive items are consecutive in the order. In this paper, we introduce threshold-constrained group tests to group testing with consecutive positives. We prove that all positive items can be identified in ⌈log2(⌈n/u⌉−1)⌉+2⌈log2(u+2)⌉+⌈log2(d−u+1)⌉−2 tests for the gap-free case (u=l+1) while the information-theoretic lower bound is ⌈log2n(d−u+1)⌉−1 when n≥d+u−2 and for u=1 the best adaptive algorithm provided by Juan and Chang (2008) takes at most ⌈log2n⌉+⌈log2d⌉ tests. We further show that the case with a gap (u>l+1) can be dealt with by the subroutines used to conquer the gap-free case.

A Space Lower Bound for Dynamic Approximate Membership Data Structures

An approximate membership data structure is a randomized data structure representing a set which supports membership queries. It allows for a small false positive error rate but has no false negative errors. Such data structures were first introduced by Bloom in the 1970s and have since had numerous applications, mainly in distributed systems, database systems, and networks. The algorithm of Bloom (known as a Bloom filter) is quite effective: it can store an approximation of a set $S$ of size $n$ by using only $\approx 1.44 n \log_2(1/\varepsilon)$ bits while having false positive error $\varepsilon$. This is within a constant factor of the information-theoretic lower bound of $n \log_2(1/\varepsilon)$ for storing such sets. Closing this gap is an important open problem, as Bloom filters are widely used in situations where storage is at a premium. Bloom filters have another property: they are dynamic. That is, they support the iterative insertions of up to $n$ elements. In fact, if one removes this requirement, there exist static data structures that receive the entire set at once and can almost achieve the information-theoretic lower bound; they require only $(1+o(1)) n \log_2(1/\varepsilon)$ bits. Our main result is a new lower bound for the space requirements of any dynamic approximate membership data structure. We show that for any constant $\varepsilon>0$, any such data structure that achieves false positive error rate of $\varepsilon$ must use at least $C(\varepsilon) \cdot n \log_2(1/\varepsilon)$ memory bits, where $C(\varepsilon)>1$ depends only on $\varepsilon$. This shows that the information-theoretic lower bound cannot be achieved by dynamic data structures for any constant error rate.

Learning the graph of epidemic cascades

We consider the problem of finding the graph on which an epidemic spreads, given only the times when each node gets infected. While this is a problem of central importance in several contexts -- offline and online social networks, e-commerce, epidemiology -- there has been very little work, analytical or empirical, on finding the graph. Clearly, it is impossible to do so from just one epidemic; our interest is in learning the graph from a small number of independent epidemics. For the classic and popular "independent cascade" epidemics, we analytically establish sufficient conditions on the number of epidemics for both the global maximum-likelihood (ML) estimator, and a natural greedy algorithm to succeed with high probability. Both results are based on a key observation: the global graph learning problem decouples into n local problems -- one for each node. For a node of degree d, we show that its neighborhood can be reliably found once it has been infected O(d 2 log n) times (for ML on general graphs) or O(d log n) times (for greedy on trees). We also provide a corresponding information-theoretic lower bound of Ω(d log n); thus our bounds are essentially tight. Furthermore, if we are given side-information in the form of a super-graph of the actual graph (as is often the case), then the number of epidemic samples required -- in all cases -- becomes independent of the network size n.

Boolean Compressed Sensing and Noisy Group Testing

The fundamental task of group testing is to recover a small distinguished subset of items from a large population while efficiently reducing the total number of tests (measurements). The key contribution of this paper is in adopting a new information-theoretic perspective on group testing problems. We formulate the group testing problem as a channel coding/decoding problem and derive a single-letter characterization for the total number of tests used to identify the defective set. Although the focus of this paper is primarily on group testing, our main result is generally applicable to other compressive sensing models. The single letter characterization is shown to be order-wise tight for many interesting noisy group testing scenarios. Specifically, we consider an additive Bernoulli($q$) noise model where we show that, for $N$ items and $K$ defectives, the number of tests $T$ is $O(\frac{K\log N}{1-q})$ for arbitrarily small average error probability and $O(\frac{K^2\log N}{1-q})$ for a worst case error criterion. We also consider dilution effects whereby a defective item in a positive pool might get diluted with probability $u$ and potentially missed. In this case, it is shown that $T$ is $O(\frac{K\log N}{(1-u)^2})$ and $O(\frac{K^2\log N}{(1-u)^2})$ for the average and the worst case error criteria, respectively. Furthermore, our bounds allow us to verify existing known bounds for noiseless group testing including the deterministic noise-free case and approximate reconstruction with bounded distortion. Our proof of achievability is based on random coding and the analysis of a Maximum Likelihood Detector, and our information theoretic lower bound is based on Fano's inequality.

Information-Theoretic Lower Bounds on the Oracle Complexity of Stochastic Convex Optimization

Relative to the large literature on upper bounds on complexity of convex optimization, lesser attention has been paid to the fundamental hardn4516420ess of these problems. Given the extensive use of convex optimization in machine learning and statistics, gaining an understanding of these complexity-theoretic issues is important. In this paper, we study the complexity of stochastic convex optimization in an oracle model of computation. We introduce a new notion of discrepancy between functions, and use it to reduce problems of stochastic convex optimization to statistical parameter estimation, which can be lower bounded using information-theoretic methods. Using this approach, we improve upon known results and obtain tight minimax complexity estimates for various function classes.

On Compact Encoding of Pagenumber $k$ Graphs

In this paper we show an information-theoretic lower bound of kn - o(kn) on the minimum number of bits to represent an unlabeled simple connected n-node graph of pagenumber k. This has to be compared with the efficient encoding scheme of Munro and Raman of 2kn + 2m + o(kn+m) bits (m the number of edges), that is 4kn + 2n + o(kn) bits in the worst-case. For m-edge graphs of pagenumber k (with multi-edges and loops), we propose a 2mlog2k + O(m) bits encoding improving the best previous upper bound of Munro and Raman whenever m ≤ 1 / 2kn/log2 k. Actually our scheme applies to k-page embedding containing multi-edge and loops. Moreover, with an auxiliary table of o(m log k) bits, our coding supports (1) the computation of the degree of a node in constant time, (2) adjacency queries with O(logk) queries of type rank, select and match, that is in O(logk *minlogk / loglogm, loglogk) time and (3) the access to δ neighbors in O(δ) runs of select, rank or match;.

Scheduling parallel jobs to minimize the makespan

We consider the NP-hard problem of scheduling parallel jobs with release dates on identical parallel machines to minimize the makespan. A parallel job requires simultaneously a prespecified, job-dependent number of machines when being processed. We prove that the makespan of any nonpreemptive list-schedule is within a factor of 2 of the optimal preemptive makespan. This gives the best-known approximation algorithms for both the preemptive and the nonpreemptive variant of the problem. We also show that no list-scheduling algorithm can achieve a better performance guarantee than 2 for the nonpreemptive problem, no matter which priority list is chosen. List-scheduling also works in the online setting where jobs arrive over time and the length of a job becomes known only when it completes; it therefore yields a deterministic online algorithm with competitive ratio 2 as well. In addition, we consider a different online model in which jobs arrive one by one and need to be scheduled before the next job becomes known. We show that no list-scheduling algorithm has a constant competitive ratio. Still, we present the first online algorithm for scheduling parallel jobs with a constant competitive ratio in this context. We also prove a new information-theoretic lower bound of 2.25 for the competitive ratio of any deterministic online algorithm for this model. Moreover, we show that 6/5 is a lower bound for the competitive ratio of any deterministic online algorithm of the preemptive version of the model jobs arriving over time.

Perfect minimally adaptive [formula omitted]-ary search with unreliable tests

We consider the problem of determining the minimum number of queries to find an unknown number in a finite set when up to a finite number e of the answers may be erroneous. In the vast literature regarding this problem, mostly the classical case of binary search is considered, i.e., when only yes–no questions are allowed. In this paper we consider a generalization of the problem which arises when questions with q many possible answers are allowed, q fixed and known beforehand. We prove that at most one question more than the information theoretic lower bound is sufficient to successfully find the unknown number. Moreover, we prove that there are infinitely many cases when the information theoretic lower bound is exactly attained and so-called perfect strategies exist. Our results are constructive and the search strategies are provided. An important issue in the area of combinatorial search is reducing adaptiveness in search strategies. We prove that the above bounds are attainable by strategies which use adaptiveness only once, a fundamental property in many practical situations. In terms of minimization of adaptiveness, this is the best possible result, since complete elimination of adaptiveness is impossible in general without significantly increasing the strategy length.