Optimal Error Exponent Research Articles

Optimal 1-bit Error Exponent for 2-Hop Relaying With Binary-Input Channels

Optimal 1-bit Error Exponent for 2-Hop Relaying With Binary-Input Channels

Universal Neyman–Pearson classification with a partially known hypothesis

Abstract We propose a universal classifier for binary Neyman–Pearson classification where the null distribution is known, while only a training sequence is available for the alternative distribution. The proposed classifier interpolates between Hoeffding’s classifier and the likelihood ratio test and attains the same error probability prefactor as the likelihood ratio test, i.e. the same prefactor as if both distributions were known. In addition, such as Hoeffding’s universal hypothesis test, the proposed classifier is shown to attain the optimal error exponent tradeoff attained by the likelihood ratio test whenever the ratio of training to observation samples exceeds a certain value. We propose a lower bound and an upper bound to the optimal training to observation ratio. In addition, we propose a sequential classifier that attains the optimal error exponent tradeoff.

On the Optimal Error Exponent of Type-Based Distributed Hypothesis Testing.

Distributed hypothesis testing (DHT) has emerged as a significant research area, but the information-theoretic optimality of coding strategies is often typically hard to address. This paper studies the DHT problems under the type-based setting, which is requested from the popular federated learning methods. Specifically, two communication models are considered: (i) DHT problem over noiseless channels, where each node observes i.i.d. samples and sends a one-dimensional statistic of observed samples to the decision center for decision making; and (ii) DHT problem over AWGN channels, where the distributed nodes are restricted to transmit functions of the empirical distributions of the observed data sequences due to practical computational constraints. For both of these problems, we present the optimal error exponent by providing both the achievability and converse results. In addition, we offer corresponding coding strategies and decision rules. Our results not only offer coding guidance for distributed systems, but also have the potential to be applied to more complex problems, enhancing the understanding and application of DHT in various domains.

Multi-Hop Network With Multiple Decision Centers Under Expected-Rate Constraints

We consider a multi-hop distributed hypothesis testing problem with multiple decision centers (DCs) for testing against independence and where the observations obey some Markov chain. For this system, we characterize the fundamental type-II error exponents region, i.e., the type-II error exponents that the various DCs can achieve simultaneously, under expected rate-constraints. Our results show that this fundamental exponents region is boosted compared to the region under maximum-rate constraints, and that it depends on the permissible type-I error probabilities. When all DCs have equal permissible type-I error probabilities, the exponents region is rectangular and all DCs can simultaneously achieve their optimal type-II error exponents. When the DCs have different permissible type-I error probabilities, a tradeoff between the type-II error exponents at the different DCs arises. New achievability and converse proofs are presented. For the achievability, a new multiplexing and rate-sharing strategy is proposed. The converse proof is based on applying different change of measure arguments in parallel and on proving asymptotic Markov chains. For the special cases <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">K</i> ∈ {2, 3}, and for arbitrary <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">K</i> ≥ 2 when all permissible type-I error probabilities at the various DCs are equal, we provide simplified expressions for the exponents region; a similar simplification is conjectured for the general case.

Sequential Transmission Over Binary Asymmetric Channels With Feedback

In this paper, we consider variable-length coding over the memoryless binary asymmetric channel (BAC) with full noiseless feedback, including the binary symmetric channel (BSC) as a special case. In 2012, Naghshvar et al. introduced a coding scheme, which we refer to as the small-enough-difference (SED) coding scheme. For symmetric binary-input channels, the deterministic variable-length feedback (VLF) code constructed with the SED coding scheme asymptotically achieves both capacity and Burnashev’s optimal error exponent. Building on the work of Naghshvar et al., this paper extends the SED coding scheme to the BAC and develops a non-asymptotic VLF achievability bound that is shown to achieve both capacity and the optimal error exponent. For the specific case of the BSC, we develop an additional non-asymptotic VLF achievability bound using a two-phase analysis that leverages both a submartingale synthesis and a Markov chain time of first passage analysis. Numerical evaluations show that both new VLF achievability bounds outperform Polyanskiy’s achievability bound for variable-length stop-feedback codes.

Duality Between Source Coding With Quantum Side Information and Classical-Quantum Channel Coding

In this paper, we establish an interesting duality between two different quantum information-processing tasks, namely, classical source coding with quantum side information, and channel coding over classical-quantum channels. The duality relates the optimal error exponents of these two tasks, generalizing the classical results of Ahlswede and Dueck [ <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">IEEE Trans. Inf. Theory</i> , 28(3):430–443, 1982]. We establish duality both at the operational level and at the level of the entropic quantities characterizing these exponents. For the latter, the duality is given by an exact relation, whereas for the former, duality manifests itself in the following sense: an optimal coding strategy for one task can be used to construct an optimal coding strategy for the other task. Along the way, we derive a bound on the error exponent for classical-quantum channel coding with constant composition codes which might be of independent interest. Finally, we consider the task of variable-length classical compression with quantum side information, and a duality relation between this task and classical-quantum channel coding can also be established correspondingly. Furthermore, we study the strong converse of this task, and show that the strong converse property does not hold even in the i.i.d. scenario.

Analysis on Optimal Error Exponents of Binary Classification for Source with Multiple Subclasses

We consider a binary classification problem for a test sequence to determine from which source the sequence is generated. The system classifies the test sequence based on empirically observed (training) sequences obtained from unknown sources and . We analyze the asymptotic fundamental limits of statistical classification for sources with multiple subclasses. We investigate the first- and second-order maximum error exponents under the constraint that the type-I error probability for all pairs of distributions decays exponentially fast and the type-II error probability is upper bounded by a small constant. In this paper, we first give a classifier which achieves the asymptotically maximum error exponent in the class of deterministic classifiers for sources with multiple subclasses, and then provide a characterization of the first-order error exponent. We next provide a characterization of the second-order error exponent in the case where only has multiple subclasses but does not. We generalize our results to classification in the case that and are a stationary and memoryless source and a mixed memoryless source with general mixture, respectively.

On Distributed Learning With Constant Communication Bits

In this paper, we study a distributed learning problem constrained by constant communication bits. Specifically, we consider the distributed hypothesis testing (DHT) problem where two distributed nodes are constrained to transmit a constant number of bits to a central decoder. In such settings, we show that in order to achieve the optimal error exponents, it suffices to consider the empirical distributions of observed data sequences and encode them to the transmission bits. With such a coding strategy, we develop a geometric approach in the distribution spaces and establish an inner bound of error exponent regions. In particular, we show the optimal achievable error exponents and coding schemes for the following cases: (i) both nodes can transmit <inline-formula> <tex-math notation="LaTeX">$\log _{2}3$ </tex-math></inline-formula> bits; (ii) one of the nodes can transmit 1 bit, and the other node is not constrained; (iii) the joint distribution of the nodes are conditionally independent given one hypothesis. Furthermore, we provide several numerical examples for illustrating the theoretical results. Our results provide theoretical guidance for designing practical distributed learning rules, and the developed approach also reveals new potentials for establishing error exponents for DHT with more general communication constraints.

Asymptotically Optimal One- and Two-Sample Testing With Kernels

We characterize the asymptotic performance of nonparametric one- and two-sample testing. The exponential decay rate or error exponent of the type-II error probability is used as the asymptotic performance metric, and an optimal test achieves the maximum rate subject to a constant level constraint on the type-I error probability. With Sanov's theorem, we derive a sufficient condition for one-sample tests to achieve the optimal error exponent in the universal setting, i.e., for any distribution defining the alternative hypothesis. We then show that two classes of Maximum Mean Discrepancy (MMD) based tests attain the optimal type-II error exponent on \mathbb R <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">d</sup> , while the quadratic-time Kernel Stein Discrepancy (KSD) based tests achieve this optimality with an asymptotic level constraint. For general two-sample testing, however, Sanov's theorem is insufficient to obtain a similar sufficient condition. We proceed to establish an extended version of Sanov's theorem and derive an exact error exponent for the quadratic-time MMD based two-sample tests. The obtained error exponent is further shown to be optimal among all two-sample tests satisfying a given level constraint. Our work hence provides an achievability result for optimal nonparametric one- and two-sample testing in the universal setting. Application to off-line change detection and related issues are also discussed.

On Covert Quantum Sensing and the Benefits of Entanglement

Motivated by applications to covert quantum radar, we analyze a covert quantum sensing problem, in which a legitimate user aims at estimating an unknown parameter taking finitely many values by probing a quantum channel while remaining undetectable from an adversary receiving the probing signals through another quantum channel. When channels are classical-quantum, we characterize the optimal error exponent under a covertness constraint for sensing strategies in which probing signals do not depend on past observations. When the legitimate user’s channel is a unitary depending on the unknown parameter, we provide achievability and converse results that show how one can significantly improve covertness using an entangled input state.

Distributed Detection With Empirically Observed Statistics

Consider a distributed detection problem in which the underlying distributions of the observations are unknown; instead of these distributions, noisy versions of empirically observed statistics are available to the fusion center. These empirically observed statistics, together with source (test) sequences, are transmitted through different channels to the fusion center. The fusion center decides which distribution the source sequence is sampled from based on these data. For the binary case, we derive the optimal type-II error exponent given that the type-I error decays exponentially fast. The type-II error exponent is maximized over the proportions of channels for both source and training sequences. We conclude that as the ratio of the lengths of training to test sequences $\alpha$ tends to infinity, using only one channel is optimal. By calculating the derived exponents numerically, we conjecture that the same is true when $\alpha$ is finite. We relate our results to the classical distributed detection problem studied by Tsitsiklis, in which the underlying distributions are known. Finally, our results are extended to the case of $m$-ary distributed detection with a rejection option.

Two Measures of Dependence.

Two families of dependence measures between random variables are introduced. They are based on the Rényi divergence of order and the relative -entropy, respectively, and both dependence measures reduce to Shannon’s mutual information when their order is one. The first measure shares many properties with the mutual information, including the data-processing inequality, and can be related to the optimal error exponents in composite hypothesis testing. The second measure does not satisfy the data-processing inequality, but appears naturally in the context of distributed task encoding.

Hypothesis Testing Over the Two-Hop Relay Network

Coding and testing schemes and the corresponding achievable type-II error exponents are presented for binary hypothesis testing over two-hop relay networks. The schemes are based on cascade source coding techniques and {unanimous decision-forwarding}, the latter meaning that a terminal decides on the null hypothesis only if all previous terminals have decided on the null hypothesis. If the observations at the transmitter, the relay, and the receiver form a Markov chain in this order, then, without loss in performance, the proposed cascade source code can be replaced by two independent point-to-point source codes, one for each hop. The decoupled scheme (combined with decision-forwarding) is shown to attain the optimal type-II error exponents for various instances of "testing against conditional independence." The same decoupling is shown to be optimal also for some instances of "testing against independence," when the observations at the transmitter, the receiver, and the relay form a Markov chain in this order, and when the relay-to-receiver link is of sufficiently high rate. For completeness, the paper also presents an analysis of the Shimokawa-Han-Amari binning scheme for the point-to-point hypothesis testing setup.

Operational Interpretation of Rényi Information Measures via Composite Hypothesis Testing Against Product and Markov Distributions

We revisit the problem of asymmetric binary hypothesis testing against a composite alternative hypothesis. We introduce a general framework to treat such problems when the alternative hypothesis adheres to certain axioms. In this case we find the threshold rate, the optimal error and strong converse exponents (at large deviations from the threshold) and the second order asymptotics (at small deviations from the threshold). We apply our results to find operational interpretations of various Renyi information measures. In case the alternative hypothesis is comprised of bipartite product distributions, we find that the optimal error and strong converse exponents are determined by variations of Renyi mutual information. In case the alternative hypothesis consists of tripartite distributions satisfying the Markov property, we find that the optimal exponents are determined by variations of Renyi conditional mutual information. In either case the relevant notion of Renyi mutual information depends on the precise choice of the alternative hypothesis. As such, our work also strengthens the view that different definitions of Renyi mutual information, conditional entropy and conditional mutual information are adequate depending on the context in which the measures are used.

Tight Asymptotic Bounds on Local Hypothesis Testing Between a Pure Bipartite State and the White Noise State

We consider asymptotic hypothesis testing (or state discrimination with asymmetric treatment of errors) between an arbitrary fixed bipartite pure state $ | \Psi \rangle $ and the white noise state (the completely mixed state) under one-way local operations and classical communications (LOCC), two-way LOCC, and separable Positive Operator Valued Measures (POVMs). As a result, we derive the Hoeffding bounds under two-way LOCC POVMs and separable POVMs. Further, we derive Stein’s lemma type of optimal error exponents under one-way LOCC, two-way LOCC, and separable POVMs up to the third order, which clarifies the difference between one-way and two-way LOCC POVM. Our results clarify the relationship between the entanglement of Renyi entropy and the hypothesis testing under LOCC, since the entanglement of Renyi entropy appears in the formula of both the Hoeffding bounds and Stein’s lemma type of error exponents. This paper gives a very rare example in which the optimal performance under the infinite-round two-way LOCC is also equal to that under separable operations and can be attained with two-round communication, but not with the one-way LOCC.

Discriminating quantum states: The multiple Chernoff distance

We consider the problem of testing multiple quantum hypotheses $\{\rho_1^{\otimes n},\ldots,\rho_r^{\otimes n}\}$, where an arbitrary prior distribution is given and each of the $r$ hypotheses is $n$ copies of a quantum state. It is known that the average error probability $P_e$ decays exponentially to zero, that is, $P_e=\exp\{-\xi n+o(n)\}$. However, this error exponent $\xi$ is generally unknown, except for the case that $r=2$. In this paper, we solve the long-standing open problem of identifying the above error exponent, by proving Nussbaum and Szko\l a's conjecture that $\xi=\min_{i\neq j}C(\rho_i,\rho_j)$. The right-hand side of this equality is called the multiple quantum Chernoff distance, and $C(\rho_i,\rho_j):=\max_{0\leq s\leq 1}\{-\log\operatorname{Tr}\rho_i^s\rho_j^{1-s}\}$ has been previously identified as the optimal error exponent for testing two hypotheses, $\rho_i^{\otimes n}$ versus $\rho_j^{\otimes n}$. The main ingredient of our proof is a new upper bound for the average error probability, for testing an ensemble of finite-dimensional, but otherwise general, quantum states. This upper bound, up to a states-dependent factor, matches the multiple-state generalization of Nussbaum and Szko\l a's lower bound. Specialized to the case $r=2$, we give an alternative proof to the achievability of the binary-hypothesis Chernoff distance, which was originally proved by Audenaert et al.

Extrinsic Jensen–Shannon Divergence: Applications to Variable-Length Coding

This paper considers the problem of variable-length coding over a discrete memoryless channel (DMC) with noiseless feedback. The paper provides a stochastic control view of the problem whose solution is analyzed via a newly proposed symmetrized divergence, termed extrinsic Jensen-Shannon (EJS) divergence. It is shown that strictly positive lower bounds on EJS divergence provide non-asymptotic upper bounds on the expected code length. The paper presents strictly positive lower bounds on EJS divergence, and hence non-asymptotic upper bounds on the expected code length, for the following two coding schemes: variable-length posterior matching and MaxEJS coding scheme which is based on a greedy maximization of the EJS divergence. As an asymptotic corollary of the main results, this paper also provides a rate-reliability test. Variable-length coding schemes that satisfy the condition(s) of the test for parameters $R$ and $E$, are guaranteed to achieve rate $R$ and error exponent $E$. The results are specialized for posterior matching and MaxEJS to obtain deterministic one-phase coding schemes achieving capacity and optimal error exponent. For the special case of symmetric binary-input channels, simpler deterministic schemes of optimal performance are proposed and analyzed.

Optimum Tradeoffs Between the Error Exponent and the Excess-Rate Exponent of Variable-Rate Slepian–Wolf Coding

We analyze the optimal tradeoff between the error exponent and the excess-rate exponent for variable-rate Slepian-Wolf codes. In particular, we first derive upper (converse) bounds on the optimal error and excess-rate exponents, and then lower (achievable) bounds, via a simple class of variable-rate codes which assign the same rate to all source blocks of the same type class. Then, using the exponent bounds, we derive bounds on the optimal rate functions, namely, the minimal rate assigned to each type class, needed in order to achieve a given target error exponent. The resulting excess-rate exponent is then evaluated. Iterative algorithms are provided for the computation of both bounds on the optimal rate functions and their excess-rate exponents. The resulting Slepian-Wolf codes bridge between the two extremes of fixed-rate coding, which has minimal error exponent and maximal excess-rate exponent, and average-rate coding, which has maximal error exponent and minimal excess-rate exponent.

The Error Exponent of Zero-Rate Multiterminal Hypothesis Testing for Sources with Common Information

SUMMARY The multiterminal hypothesis testing problem with zerorate constraint is considered. For this problem, an upper bound on the optimal error exponent is given by Shalaby and Papamarcou, provided that the positivity condition holds. Our contribution is to prove that Shalaby and Papamarcou’s upper bound is valid under a weaker condition: (i) two remote observations have a common random variable in the sense of G´ acks and K¨ orner, and (ii) when the value of the common random variable is fixed, the conditional distribution of remaining random variables satisfies the positivity condition. Moreover, a generalization of the main result is

A Controlled Sensing Approach to Graph Classification

The problem of classifying graphs with respect to connectivity via partial observations of nodes is posed as a composite hypothesis testing problem with controlled sensing. An observation at a node is a subset of edges incident to the node on the complete graph drawn according to a probability model, which is a function of a fixed underlying graph. Connectivity is measured through average node degree and is classified with respect to a threshold. A simple approximation of the controlled sensing test is derived and simulated on Erdös-Rènyi graphs to characterize the error probabilities as a function of the expected stopping times. The test is also experimentally validated on a real-world example of the social structure of Long-Tailed Manakins. It is shown that the proposed test achieves favorable tradeoffs between the classification error and the number of measurements. Furthermore, the test outperforms existing approaches, especially at low target error rates. In addition, the proposed test achieves the optimal error exponent.