Binary Entropy Research Articles

Dimension in Complexity Classes

A theory of resource-bounded dimension is developed using gales, which are natural generalizations of martingales. When the resource bound $\Delta$ (a parameter of the theory) is unrestricted, the resulting dimension is precisely the classical Hausdorff dimension (sometimes called "fractal dimension"). Other choices of the parameter $\Delta$ yield internal dimension theories in E, E2, ESPACE, and other complexity classes, and in the class of all decidable problems. In general, if $\mathcal{C}$ is such a class, then every set X of languages has a dimension in $\mathcal{C}$, which is a real number $\dim (X \mid \mathcal{C}) \in [0, 1]$. Along with the elements of this theory, two preliminary applications are presented: For every real number $0 \le \alpha \le \frac 1 2$, the set ${\rm FREQ}(\le \alpha)$, consisting of all languages that asymptotically contain at most $\alpha$ of all strings, has dimension $\mathcal{H}(\alpha)$---the binary entropy of $\alpha$---in E and in E2. For every real number $0 \le \alpha \le 1$, the set ${\rm SIZE}(\alpha \frac {2^n} n)$, consisting of all languages decidable by Boolean circuits of at most $\alpha \frac {2^n} n$ gates, has dimension $\alpha$ in ESPACE.

Protocols for Asymmetric Communication Channels

In this paper we examine the problem of sending an n-bit data item from a client to a server across an asymmetric communication channel. We demonstrate that there are scenarios in which a high-speed link from the server to the client can be used to greatly reduce the number of bits sent from the client to the server across a slower link. In particular, we assume that the data item is drawn from a probability distribution D that is known to the server but not to the client. We present several protocols in which the expected number of bits transmitted by the server and client are O(n) and O(H(D)+1), respectively, where H(D) is the binary entropy of D (and can range from 0 to n). These protocols are within a small constant factor of optimal in terms of the number of bits sent by the client. The expected number of rounds of communication between the server and client in the simplest of our protocols is O(H(D)). We also give a protocol for which the expected number of rounds is only O(1), but which requires more computational effort on the part of the server. A third technique provides a tradeoff between the computational effort and the number of rounds. These protocols are complemented by several lower bounds and impossibility results. We prove that all of our protocols are existentially optimal in terms of the number of bits sent by the server, i.e., there are distributions for which the total number of bits exchanged has to be at least n. In addition, we show that there is no protocol that is optimal for every distribution (as opposed to just existentially optimal) in terms of bits sent by the server. We demonstrate this by proving that it is undecidable to compute (even approximately), for an arbitrary distribution D, the expected number of bits that must be exchanged by the server and client on the distribution D.

Total Path Length for Random Recursive Trees

Total path length, or search cost, for a rooted tree is defined as the sum of all root-to-node distances. Let Tn be the total path length for a random recursive tree of order n. Mahmoud [10] showed that Wn := (Tn − E[Tn])/n converges almost surely and in L2 to a nondegenerate limiting random variable W. Here we give recurrence relations for the moments of Wn and of W and show that Wn converges to W in Lp for each 0 < p < ∞. We confirm the conjecture that the distribution of W is not normal. We also show that the distribution of W is characterized among all distributions having zero mean and finite variance by the distributional identityformula herewhere [Escr ](x) := − x ln x − (1 minus; x) ln(1 − x) is the binary entropy function, U is a uniform (0, 1) random variable, W* and W have the same distribution, and U, W and W* are mutually independent. Finally, we derive an approximation for the distribution of W using a Pearson curve density estimator.

Identification without randomization

In the theory of identification via noisy channels randomization in the encoding has a dramatic effect on the optimal code size, namely, it grows double-exponentially in the blocklength, whereas in the theory of transmission it has the familiar exponential growth. We consider now instead of the discrete memoryless channel (DMC) more robust channels such as the familiar compound (CC) and arbitrarily varying channels (AVC). They can be viewed as models for jamming situations. We make the pessimistic assumption that the jammer knows the input sequence before he acts. This forces communicators to use the maximal error concept and also makes randomization in the encoding superfluous. Now, for a DMC W by a simple observation, made by Ahlswede and Dueck (1989), in the absence of randomization the identification capacity, say C/sub NRI/(W), equals the logarithm of the number of different row-vectors in W. We generalize this to compound channels. A formidable problem arises if the DMC W is replaced by the AVC W. In fact, for 0-1-matrices only in W we are-exactly as for transmission-led to the equivalent zero-error-capacity of Shannon. But for general W the identification capacity C/sub NRI/(W) is quite different from the transmission capacity C(W). An observation is that the separation codes of Ahlswede (1989) are also relevant here. We present a lower bound on C/sub NRI/(W). It implies for instance for W={(/sub 0 1//sup 1 0/), (/sub /spl delta/ (1-/spl delta/)(1 0)/)}, /spl delta//spl isin/(0, 1/2 ) that C/sub NRI/(W)=1, which is obviously tight. It exceeds C(W), which is known to exceed 1-h(/spl delta/), where h is the binary entropy function. We observe that a separation code, with worst case average list size L~ (which we call an NRA code) can be partitioned into L~2/sup ne/ transmission codes. This gives a nonsingle-letter characterization of the capacity of AVC with maximal probability of error in terms of the capacity of codes with list decoding. We also prove that randomization in the decoding does not increase C/sub I/(W) and C/sub NRI/(W). Finally, we draw attention to related work on source coding.

Good quantum error-correcting codes exist.

A quantum error-correcting code is defined to be a unitary mapping (encoding) of k qubits (2-state quantum systems) into a subspace of the quantum state space of n qubits such that if any t of the qubits undergo arbitrary decoherence, not necessarily independently, the resulting n qubits can be used to faithfully reconstruct the original quantum state of the k encoded qubits. Quantum error-correcting codes are shown to exist with asymptotic rate k/n = 1 - 2H(2t/n) where H(p) is the binary entropy function -p log p - (1-p) log (1-p). Upper bounds on this asymptotic rate are given.

Finding the ?n-th largest element

We describe an algorithm for selecting the αn-th largest element (where 0<α<1), from a totally ordered set ofn elements, using at most (1+(1+o(1))H(α))·n comparisons whereH(α) is the binary entropy function and theo(1) stands for a function that tends to 0 as α tends to 0. For small values of α this is almost the best possible as there is a lower bound of about (1+H(α))·n comparisons. The algorithm obtained beats the global 3n upper bound of Schonhage, Paterson and Pippenger for α<1/3.

Corrective memory by a symmetric sparsely encoded network

A neural network that retrieves stored binary vectors, when probed by possibly corrupted versions of them, is presented. It employs sparse ternary internal coding and autocorrelation (Hebbian) storage. It is symmetrically structured and, consequently, can be folded into a feedback configuration. Bounds on the network parameters are derived from probabilistic considerations. It is shown that when the input dimension is n, the proportional activation radius is /spl rho/ and the network size is 2/sup /spl nu/n/ with /spl nu/>1-h/sub 2/(/spl rho/), the equilibrium capacity is at least 2/sup /spl alpha/n//8n/spl rho/(1-/spl rho/) for any /spl alpha/<1-h/sub 2/(/spl rho/), where h/sub 2/(/spl middot/) is the binary entropy. A similar capacity bound is derived for the correction of errors of proportional size /spl rho/ or less, when /spl rho//spl les/0.3. The performance of a finite-size symmetric network is examined by simulation and found to exceed, at the cost of higher connectivity, that of the Kanerva (1988) model, operating as a content addressable memory.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

A random coding bound for fixed convolutional codes of rate 1/n

We show that the ensemble average of the block error probability for the ensemble of terminated rate 1/n fixed convolutional codes, used on the binary symmetric channel with a maximum likelihood decoder, is bounded by exp/sub 2/-NE/sub r/(1-K/N), where N=(L+m)n is the block length, L being the message length, K the constraint length, and E/sub r/() is the random coding exponent for block codes. Hence, E/sub r/(1-K/N)>0 for H(p)<K/N/spl les/1, where H() is the binary entropy function and p is the cross-over probability of the binary symmetric channel.<<ETX>>

On the average delay for routing subject to independent deflections

Consider a packet walking along a directed graph with each node having two edges directed out. The packet is headed towards one of N destinations, chosen according to a probability distribution p. At each step, the packet is forced to use a nonpreferred edge with some probability q, independently of past events. Using information theory and sequential analysis, it is shown that the mean number of steps required by the packet to reach the destination is roughly, at least H(p)/(1-h(q), where h is the binary entropy function and H(p) is the entropy (base two) of p. This lower bound is shown to be asymptotically achievable in the case where the packet always begins at a fixed node. Also considered is the maximum, over all pairs of nodes in a graph, of the mean transit time from one node to the other. The work is motivated by the search for graphs that work well in conjunction with deflection routing in communication networks.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

On the capacity region of the discrete additive multiple-access arbitrarily varying channel

The discrete additive multiple-access arbitrarily varying channel (AVC) with two senders and one receiver is considered. Necessary and sufficient conditions are given for its deterministic-code average-probability-of-error capacity region under a state constraint to have a nonempty interior. In the case that no state constraint is present, the capacity region is characterized exactly. In the case of the noiseless mod-2 adder AVC using state constraint function l(s)=s and subject to a state constraint L less than or equal to 0.13616917, the capacity region is shown to be a 45-degree triangle whose legs have length 1-h(L), where h denotes the binary entropy function. >

A binary analog to the entropy-power inequality

Let (X/sub n/), (Y/sub n/) be independent stationary binary random sequences with entropy H(X), H(Y), respectively. Let h( zeta )=- zeta log zeta -(1- zeta )log(1- zeta ), 0<or= zeta <or=1/2, be the binary entropy function and let sigma (X)=h/sup -1/ (H(X)), sigma (Y)=h/sup -1/ (H(Y)). Let z/sub n/=X/sub n/(+)Y/sub n/, where (+) denotes modulo-2 addition. The following analog of the entropy-power inequality provides a lower bound on H(Z), the entropy of (Z/sub n/): sigma (Z)>or= sigma (X)* sigma (Y), where sigma (Z)=h/sup -1/ (H(Z)), and alpha * beta = alpha (1- beta )+ beta (1- alpha ). When (Y/sub n/) are independent identically distributed, this reduces to Mrs. Gerber's Lemma from A.D. Wyner and J. Ziv (1973).<<ETX>>

The capacity of the Kanerva associative memory

Asymptotic expressions for the capacity of an associative memory proposed by P. Kanerva (1984) are derived. Capacity is defined as the maximum number of random binary words that can be stored at random addresses so that the probability that a word is in error is arbitrarily small when it is retrieved by an n-bit address containing fewer than delta n errors, delta <or=1/2. Sphere-packing arguments show that the capacity of any associative memory can grow exponentially in n at a rate of at most 1-h/sub 2/( delta ), where h/sub 2/( delta ) is the binary entropy function in bits. It turns out that the Kanerva associative memory achieves this upper bound when its parameters are optimally set. Thus, the capacity of the Kanerva associative memory has an exponential growth rate equal to the rate of the best information-theoretic codes, that is 1-h/sub 2/( delta ). However, the Kanerva memory achieves its exponential growth in capacity at the expense of an exponential growth in hardware.<<ETX>>

An achievable bound for optimal noiseless coding of a random variable (Corresp.)

For a discrete <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">N</tex> -valued random variable ( <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">N</tex> possibly denumerably infinite) Leung-Yan-Cheong and Cover have given bounds for the minimal expected length of a one-to-one (not necessarily uniquely decodable) code <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">L_{1:1}=\sum_{i=1}^{N} p_{i} \log \left( \frac{1}{2} + 1 \right).</tex> It is shown that the best possible case occurs for certain denumerably infinite sets of nonzero probabilities. This absolute bound is related to the Shannon entropy <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">H</tex> of the distribution by <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">(h (\cdot)</tex> is the binary entropy function).

A nonconstructive upper bound on covering radius

Let <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">t(n,k)</tex> denote the minimum covering radius of a binary linear <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">(n,k)</tex> code. We give a nonconstructive upper bound on <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">t(n,k)</tex> , which coincides asymptotically with the known lower bound, namely <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">n^{-1}t(n,nR)=H^{-1}(1-R)+O(n^{-l}\log n)</tex> , where <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">R</tex> is fixed, <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">0&lt;R&lt;1</tex> , and <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">H^{-1}</tex> is the inverse of the binary entropy function.

Syntactic compression codes at the zero entropy point

Syntactic compression codes compress the tree, which is the syntax of a binary source message. The ones considered here originate from image processing. The syntactic trees usedhave a constant valency and their binary labels distinguish whether the source substring derived from a node is completely zero or not. We compress them simply by deleting some redundant subtrees. These codes fall into a theoretically new class of codes which is wider than the classical ones. They are here studied in the neighborhood of a zero of the binary entropy function. There, their behavior is close to that of an infinite run length encoding and the optimum valency is three. Finally, we open a problem, related with automata theory, which perhaps could provide a further link between Information Theory and Algorithmic Information Theory.

On the fractional weight of distinct binary&amp;lt;tex&amp;gt;n&amp;lt;/tex&amp;gt;-tuples (Corresp.)

It is shown that the fraction <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">p</tex> of ones in the <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Mn</tex> positions of <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">M</tex> distinct binary <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">n</tex> -tuples satisfies the inequality \begin{equation} h(p) \geq (l/n) \log_2 M \end{equation} where <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">h(p) = - p log_2 p - (1 - p) log_2 (1 - p)</tex> is the binary entropy function. This inequality, which simplifies the derivation of the distance property of the Justesen codes, is proved using an elegant information-theoretic argument due to Kriz.

Generalization of Mrs. Gerber’s lemma

Mrs. Gerber's Lemma (MGL) hinges on the convexity of $H(p*H^{-1}(u))$, where $H(u)$ is the binary entropy function. In this work, we prove that $H(p*f(u))$ is convex in $u$ for every $p\in [0,1]$ provided $H(f(u))$ is convex in $u$, where $f(u) : (a, b) \to [0, \frac12]$. Moreover, our result subsumes MGL and simplifies the original proof. We show that the generalized MGL can be applied in binary broadcast channel to simplify some discussion.