Non-Asymptotic Bounds Research Articles

Nonasymptotic Bounds for Adversarial Excess Risk under Misspecified Models

Nonasymptotic Bounds for Adversarial Excess Risk under Misspecified Models

Improved bounds for codes correcting insertions and deletions

AbstractThis paper studies the cardinality of codes correcting insertions and deletions. We give improved upper and lower bounds on code size. Our upper bound is obtained by utilizing the asymmetric property of list decoding for insertions and deletions and can be seen as analogous to the Elias bound in the Hamming metric. Our non-asymptotic bound is better than the existing bounds when the minimum Levenshtein distance is relatively large. The asymptotic bound exceeds the Elias and the MRRW bounds adapted from the Hamming-metric bounds for the binary and the quaternary cases. Our lower bound improves on the bound by Levenshtein, but its effect is limited and vanishes asymptotically.

Dimension Free Nonasymptotic Bounds on the Accuracy of High-Dimensional Laplace Approximation

Dimension Free Nonasymptotic Bounds on the Accuracy of High-Dimensional Laplace Approximation

Explicit Non-Asymptotic Bounds for the Distance to the First-Order Edgeworth Expansion

Explicit Non-Asymptotic Bounds for the Distance to the First-Order Edgeworth Expansion

Non-Asymptotic Bounds of Cumulant Generating Function of Codeword Lengths in Variable-Length Lossy Compression

This paper investigates the problem of variable-length source coding with the criteria of the normalized cumulant generating function of codeword lengths and the excess distortion probability. We analyze the non-asymptotic fundamental limit of the normalized cumulant generating function of codeword lengths under the constraint that the excess distortion probability is allowed up to ϵ ∈ [0, 1). Our non-asymptotic achievability and converse bounds are characterized by the quantity related to the Rényi entropy.

Nonasymptotic Bounds for Stochastic Optimization With Biased Noisy Gradient Oracles

We introduce biased gradient oracles to capture a setting where the function measurements have an estimation error that can be controlled through a batch size parameter. Our proposed oracles are appealing in several practical contexts, for instance, risk measure estimation from a batch of independent and identically distributed samples, or simulation optimization, where the function measurements are “biased” due to computational constraints. In either case, increasing the batch size reduces the estimation error. We highlight the applicability of our biased gradient oracles in a risk-sensitive reinforcement learning setting. In the stochastic nonconvex optimization context, we analyze a variant of the randomized stochastic gradient algorithm with a biased gradient oracle. We quantify the convergence rate of this algorithm by deriving nonasymptotic bounds on its performance. Next, in the stochastic convex optimization setting, we derive nonasymptotic bounds for the last iterate of a stochastic gradient descent algorithm with a biased gradient oracle.

Non-Asymptotic Bounds of AIPW Estimators for Means with Missingness at Random

The augmented inverse probability weighting is well known for its double robustness in missing data and causal inference. If either the propensity score model or the outcome regression model is correctly specified, the estimator is guaranteed to be consistent. Another important property of the augmented inverse probability weighting is that it can achieve first-order equivalence to the oracle estimator in which all nuisance parameters are known, even if the fitted models do not converge at the parametric root-n rate. We explore the non-asymptotic properties of the augmented inverse probability weighting estimator to infer the population mean with missingness at random. We also consider inferences of the mean outcomes on the observed group and on the unobserved group.

Sparse Laplacian shrinkage for nonparametric transformation survival model

The rank estimation is an effective inference method for the nonparametric transformation model. This approach avoids any nonparametric estimation about the transformation function and can be applied to the high-dimensional censored data. However, most existing methods do not utilize the potential correlation structures among predictors. In order to incorporate such priori information, we propose a penalized smoothed partial rank with sparse Laplacian shrinkage (PSPRL) method and develop a forward and backward stagewise with sparse Laplacian shrinkage (LFabs) algorithm to compute the estimator. The non-asymptotic bound and algorithm properties are established. Simulation results show that the proposed method outperforms the competing alternatives with better variable selection and prediction. We apply our method to a glioblastoma gene expression study to further demonstrate the advantages.

The CEO Problem With Inter-Block Memory

An $n$ -dimensional source with memory is observed by $K$ isolated encoders via parallel channels, who compress their observations to transmit to the decoder via noiseless rate-constrained links while leveraging their memory of the past. At each time instant, the decoder receives $K$ new codewords from the observers, combines them with the past received codewords, and produces a minimum-distortion estimate of the latest block of $n$ source symbols. This scenario extends the classical one-shot CEO problem to multiple rounds of communication with communicators maintaining the memory of the past. We extend the Berger-Tung inner and outer bounds to the scenario with inter-block memory, showing that the minimum asymptotically (as $n \to \infty $ ) achievable sum rate required to achieve a target distortion is bounded by minimal directed mutual information problems. For the Gauss-Markov source observed via $K$ parallel AWGN channels, we show that the inner bound is tight and solve the corresponding minimal directed mutual information problem, thereby establishing the minimum asymptotically achievable sum rate. Finally, we explicitly bound the rate loss due to a lack of communication among the observers; that bound is attained with equality in the case of identical observation channels. The general coding theorem is proved via a new nonasymptotic bound that uses stochastic likelihood coders and whose asymptotic analysis yields an extension of the Berger-Tung inner bound to the causal setting. The analysis of the Gaussian case is facilitated by reversing the channels of the observers.

Fast Proximal Gradient Methods for Nonsmooth Convex Optimization for Tomographic Image Reconstruction

The Fast Proximal Gradient Method (FPGM) and the Monotone FPGM (MFPGM) for minimization of nonsmooth convex functions are introduced and applied to tomographic image reconstruction. Convergence properties of the sequence of objective function values are derived, including a $$O\left( 1/k^{2}\right)$$ non-asymptotic bound. The presented theory broadens current knowledge and explains the convergence behavior of certain methods that are known to present good practical performance. Numerical experimentation involving computerized tomography image reconstruction shows the methods to be competitive in practical scenarios. Experimental comparison with Algebraic Reconstruction Techniques are performed uncovering certain behaviors of accelerated Proximal Gradient algorithms that apparently have not yet been noticed when these are applied to tomographic image reconstruction.

Rates of Convergence to Stationarity for Reflected Brownian Motion

We provide the first rate of convergence to stationarity analysis for reflected Brownian motion (RBM) as the dimension grows under some uniformity conditions. In particular, if the underlying routing matrix is uniformly contractive, uniform stability of the drift vector holds, and the variances of the underlying Brownian motion (BM) are bounded, then we show that the RBM converges exponentially fast to stationarity with a relaxation time of order [Formula: see text] as the dimension d → ∞. Our bound for the relaxation time follows as a corollary of the nonasymptotic bound we obtain for the initial transient effect, which is explicit in terms of the RBM parameters.

Non-Asymptotic Converse Bounds and Refined Asymptotics for Two Source Coding Problems

In this paper, we revisit two multi-terminal lossy source coding problems: the lossy source coding problem with side information available at the encoder and one of the two decoders, which we term as the Kaspi problem (Kaspi, 1994), and the multiple description coding problem with one semi-deterministic distortion measure, which we refer to as the Fu-Yeung problem (Fu and Yeung, 2002). For the Kaspi problem, we first present the properties of optimal test channels. Subsequently, we generalize the notion of the distortion-tilted information density for the lossy source coding problem to the Kaspi problem and prove a non-asymptotic converse bound using the properties of optimal test channels and the well-defined distortion-tilted information density. Finally, for discrete memoryless sources, we derive refined asymptotics which includes the second-order, large, and moderate deviations asymptotics. In the converse proof of second-order asymptotics, we apply the Berry-Esseen theorem to the derived non-asymptotic converse bound. The achievability proof follows by first proving a type-covering lemma tailored to the Kaspi problem, then properly Taylor expanding the well-defined distortion-tilted information densities and finally applying the Berry-Esseen theorem. We then generalize the methods used in the Kaspi problem to the Fu-Yeung problem. As a result, we obtain the properties of optimal test channels for the minimum sum-rate function, a non-asymptotic converse bound and refined asymptotics for discrete memoryless sources. Since the successive refinement problem is a special case of the Fu-Yeung problem, as a by-product, we obtain a non-asymptotic converse bound for the successive refinement problem, which is a strict generalization of the non-asymptotic converse bound for successively refinable sources (Zhou, Tan, and Motani, 2017).

Detection and Analysis of Spikes in a Random Sequence

Motivated by the more frequent natural and anthropogenic hazards, we revisit the problem of assessing whether an apparent temporal clustering in a sequence of randomly occurring events is a genuine surprise and should call for an examination. We study the problem in both discrete and continuous time formulation. In the discrete formulation, the problem reduces to deriving the probability that p independent people all have birthdays within d days of each other. We provide an analytical expression for a warning limit such that if a subset of p people among n are observed to have birthdays within d days of each other and d is smaller than our warning limit, then it should be treated as a surprising cluster. In the continuous time framework, three different sets of results are given. First, we provide an asymptotic analysis of the problem by embedding it into an extreme value problem for high order spacings of iid samples from the U[0, 1] density. Second, a novel analytical nonasymptotic bound is derived by using certain tools of empirical process theory. Finally, the required probability is approximated by using various bounds and asymptotic results on the supremum of the scanning process of a one dimensional stationary Poisson process. We apply the theories to climate change related datasets, datasets on temperatures, and mass shooting records in the United States. These real data applications of our theoretical methods lead to supporting evidence for climate change and recent spikes in gun violence.

Non-Asymptotic Bounds and a General Formula for the Rate-Distortion Region of the Successive Refinement Problem

In the successive refinement problem, a fixed-length sequence emitted from an information source is encoded into two codewords by two encoders in order to give two reconstructions of the sequence. One of two reconstructions is obtained by one of two codewords, and the other reconstruction is obtained by all two codewords. For this coding problem, we give non-asymptotic inner and outer bounds on pairs of numbers of codewords of two encoders such that each probability that a distortion exceeds a given distortion level is less than a given probability level. We also give a general formula for the rate-distortion region for general sources, where the rate-distortion region is the set of rate pairs of two encoders such that each maximum value of possible distortions is less than a given distortion level.

New Non-Asymptotic Bounds on Numbers of Codewords for the Fixed-Length Lossy Compression

New Non-Asymptotic Bounds on Numbers of Codewords for the Fixed-Length Lossy Compression

Expurgated Random-Coding Ensembles: Exponents, Refinements, and Connections

This paper studies expurgated random-coding bounds and exponents for channel coding with a given (possibly suboptimal) decoding rule. Variations of Gallager's analysis are presented, yielding several asymptotic and non-asymptotic bounds on the error probability for an arbitrary codeword distribution. A simple non-asymptotic bound is shown to attain an exponent of Csisz\'ar and K\"orner under constant-composition coding. Using Lagrange duality, this exponent is expressed in several forms, one of which is shown to permit a direct derivation via cost-constrained coding which extends to infinite and continuous alphabets. The method of type class enumeration is studied, and it is shown that this approach can yield improved exponents and better tightness guarantees for some codeword distributions. A generalization of this approach is shown to provide a multi-letter exponent which extends immediately to channels with memory. Finally, a refined analysis expurgated i.i.d. random coding is shown to yield a O\big(\frac{1}{\sqrt{n}}\big) prefactor, thus improving on the standard O(1) prefactor. Moreover, the implied constant is explicitly characterized.

A sharp nonasymptotic bound and phase diagram of L 1/2 regularization

We derive a sharp nonasymptotic bound of parameter estimation of the L 1/2 regularization. The bound shows that the solutions of the L 1/2 regularization can achieve a loss within logarithmic factor of an ideal mean squared error and therefore underlies the feasibility and effectiveness of the L 1/2 regularization. Interestingly, when applied to compressive sensing, the L 1/2 regularization scheme has exhibited a very promising capability of completed recovery from a much less sampling information. As compared with the L p (0 < p < 1) penalty, it is appeared that the L 1/2 penalty can always yield the most sparse solution among all the L p penalty when 1/2 ≤ p< 1, and when 0 < p< 1/2, the L p penalty exhibits the similar properties as the L 1/2 penalty. This suggests that the L 1/2 regularization scheme can be accepted as the best and therefore the representative of all the L p (0 < p< 1) regularization schemes.

Non-Asymptotic Bounds on the Performance of Dual Methods for Resource Allocation Problems

In this paper, dual methods based on Lagrangian relaxation for multiuser multicarrier resource allocation problems are analyzed. Their application to non-convex resource allocation problems is based on results guaranteeing asymptotic optimality as the number of subcarriers tends to infinity. This work analyzes the workings and performance of dual methods for resource allocation problems with concave rate functions and a finite number of subcarriers. The core results are the convexity of resource allocation problems with subcarrier sharing and an upper bound on the number of subcarriers being shared. Based on these results, absolute and relative performance bounds are presented for dual methods when applied to the resource allocation problem without subcarrier sharing. The exemplary problems considered in this work are sum rate maximization with global and individual power budgets and sum power minimization with global and individual rate demands.

On properties of the algorithm for pursuing a drifting quantile

The recurrent algorithm for pursuing a time-varying ("drifting") quantile is suggested. The common nonasymptotic upper bound of the algorithm quality is established, which is then used in a few examples of the conditions for the quantile drift function. Estimates of the degree (rate) of convergence of the algorithm for the considered examples are obtained.

Adaptive sequential estimation for ergodic diffusion processes in quadratic metric

An adaptive nonparametric procedure is constructed for estimating the unknown drift coefficient in ergodic diffusion processes. A sharp non-asymptotic upper bound (an oracle inequality) is obtained for a quadratic risk. Furthermore, an asymptotic lower bound for the minimax quadratic risk is found that equals to the Pinsker constant. Asymptotic efficiency is proved, that is, the asymptotic quadratic risk of the constructed estimator coincides with this constant.