Generic Chaining Research Articles

Sample average approximation with heavier tails I: non-asymptotic bounds with weak assumptions and stochastic constraints

We derive new and improved non-asymptotic deviation inequalities for the sample average approximation (SAA) of an optimization problem. Our results give strong error probability bounds that are “sub-Gaussian” even when the randomness of the problem is fairly heavy tailed. Additionally, we obtain good (often optimal) dependence on the sample size and geometrical parameters of the problem. Finally, we allow for random constraints on the SAA and unbounded feasible sets, which also do not seem to have been considered before in the non-asymptotic literature. Our proofs combine different ideas of potential independent interest: an adaptation of Talagrand’s “generic chaining” bound for sub-Gaussian processes; “localization” ideas from the Statistical Learning literature; and the use of standard conditions in Optimization (metric regularity, Slater-type conditions) to control fluctuations of the feasible set.

A Sharp Analysis of Covariate Adjusted Precision Matrix Estimation via Alternating Projected Gradient Descent

In this letter, we present a sharp algorithmic analysis for alternating projected gradient descent which is used to solve the covariate adjusted precision matrix estimation problem in high-dimensional settings. By introducing a new analytical tool (the generic chaining), we remove the impractical resampling assumption used in the literature. The new analysis also demonstrates that this algorithm not only enjoys a linear convergence rate in the absence of convexity, but also attains the minimax rate with optimal order of sample complexity. Our results, meanwhile, reveal a time-data tradeoff in this problem. Numerical experiments are provided to verify our theoretical results.

High dimensional decision making, upper and lower bounds

A decision maker’s utility depends on her action a∈A⊂Rd and the payoff relevant state of the world θ∈Θ. One can define the value of acquiring new information as the difference between the maximum expected utility pre- and post information acquisition. In this paper, I find asymptotic results on the expected value of information as d→∞, by using tools from the theory of (sub)-Gaussian processes and generic chaining.

Stability of Traveling Waves on Exponentially Long Timescales in Stochastic Reaction-Diffusion Equations

In this paper we establish the meta-stability of traveling waves for a class of reaction-diffusion equations forced by a multiplicative noise term. In particular, we show that the phase-tracking technique developed in [C. H. S. Hamster and H. J. Hupkes, SIAM J. Appl. Dyn. Syst., 18, pp. 205--278; Phys. D, 401, 132233] can be maintained over timescales that are exponentially long with respect to the noise intensity. This is achieved by combining the generic chaining principle with a mild version of the Burkholder--Davis--Gundy inequality to establish logarithmic supremum bounds for stochastic convolutions in the critical regularity regime.

Squared-norm empirical processes

This note extends a result of Mendelson on the supremum of a quadratic process to squared norms of functions taking values in a Banach space. Our method of proof is a reduction by a symmetrization argument and simple observation about the additivity of the generic chaining functional. We demonstrate an application to positive linear functionals of the sample covariance matrix and the apparent variance explained by principal components analysis (PCA).

Quasi-random multilinear polynomials

We consider multilinear Littlewood polynomials, polynomials in n variables in which a specified set of monomials U have ±1 coefficients, and all other coefficients are 0. We provide upper and lower bounds (which are close for U of degree below log n) on the minimum, over polynomials h consistent with U, of the maximum of |h| over ±1 assignments to the variables. (This is a variant of a question posed by Erdős regarding the maximum on the unit disk of univariate polynomials of given degree with unit coefficients.) We outline connections to the theory of quasi-random graphs and hypergraphs, and to statistical mechanics models. Our methods rely on the analysis of the Gale–Berlekamp game; on the constructive side of the generic chaining method; on a Khintchine-type inequality for polynomials of degree greater than 1; and on Bernstein’s approximation theory inequality.

Structured Signal Recovery From Non-Linear and Heavy-Tailed Measurements

We study high-dimensional signal recovery from non-linear measurements with design vectors having elliptically symmetric distribution. Special attention is devoted to the situation when the unknown signal belongs to a set of low statistical complexity, while both the measurements and the design vectors are heavy-tailed. We propose and analyze a new estimator that adapts to the structure of the problem, while being robust both to the possible model misspecification characterized by arbitrary non-linearity of the measurements as well as to data corruption modeled by the heavy-tailed distributions. Moreover, this estimator has low computational complexity. Our results are expressed in the form of exponential concentration inequalities for the error of the proposed estimator. On the technical side, our proofs rely on the generic chaining methods, and illustrate the power of this approach for statistical applications. Theory is supported by numerical experiments demonstrating that our estimator outperforms existing alternatives when data is heavy-tailed.

Chaining, interpolation, and convexity

We show that classical chaining bounds on the suprema of random processes in terms of entropy numbers can be systematically improved when the underlying set is convex: the entropy numbers need not be computed for the entire set, but only for certain “thin” subsets. This phenomenon arises from the observation that real interpolation can be used as a natural chaining mechanism. Unlike the general form of Talagrand’s generic chaining method, which is sharp but often difficult to use, the resulting bounds involve only entropy numbers but are nonetheless sharp in many situations in which classical entropy bounds are suboptimal. Such bounds are readily amenable to explicit computations in specific examples, and we discover some old and new geometric principles for the control of chaining functionals as special cases.

On Bernstein type inequalities for stochastic integrals of multivariate point processes

In this paper, we first obtain a Bernstein type of concentration inequality for stochastic integrals of multivariate point processes under some conditions through the Doléans-Dade exponential formula, and then derive a uniform exponential inequality using a generic chaining argument. As a direct consequence, we obtain an upper bound for a sequence of discrete time martingales indexed by a class of functionals. Finally, we apply the uniform exponential bound to nonparametric maximum likelihood estimators and provide a rate of convergence in terms of Hellinger distance, which is an improvement of earlier work of van de Geer (1995).

Chaining, interpolation and convexity II: The contraction principle

The generic chaining method provides a sharp description of the suprema of many random processes in terms of the geometry of their index sets. The chaining functionals that arise in this theory are however notoriously difficult to control in any given situation. In the first paper in this series, we introduced a particularly simple method for producing the requisite multi scale geometry by means of real interpolation. This method is easy to use, but does not always yield sharp bounds on chaining functionals. In the present paper, we show that a refinement of the interpolation method provides a canonical mechanism for controlling chaining functionals. The key innovation is a simple but powerful contraction principle that makes it possible to efficiently exploit interpolation. We illustrate the utility of this approach by developing new dimension-free bounds on the norms of random matrices and on chaining functionals in Banach lattices. As another application, we give a remarkably short interpolation proof of the majorizing measure theorem that entirely avoids the greedy construction that lies at the heart of earlier proofs.

Tail bounds via generic chaining

We modify Talagrand's generic chaining method to obtain upper bounds for all p-th moments of the supremum of a stochastic process. These bounds lead to an estimate for the upper tail of the supremum with optimal deviation parameters. We apply our procedure to improve and extend some known deviation inequalities for suprema of unbounded empirical processes and chaos processes. As an application we give a significantly simplified proof of the restricted isometry property of the subsampled discrete Fourier transform.

Cover Times and Generic Chaining

A recent result of Ding, Lee and Peres (2012) expressed the cover time of the random walk on a graph in terms of generic chaining for the commute distance. Their argument is based on Dynkin's isomorphism theorem. The purpose of this article is to present an alternative approach to this problem, based only on elementary hitting time estimates and chaining arguments.

Cover Times and Generic Chaining

A recent result of Ding, Lee and Peres (2012) expressed the cover time of the random walk on a graph in terms of generic chaining for the commute distance. Their argument is based on Dynkin's isomorphism theorem. The purpose of this article is to present an alternative approach to this problem, based only on elementary hitting time estimates and chaining arguments.

Generic chaining and the ℓ1-penalty

We address the choice of the tuning parameter λ in ℓ1-penalized M-estimation. Our main concern is models which are highly non-linear, such as the Gaussian mixture model. The number of parameters p is moreover large, possibly larger than the number of observations n. The generic chaining technique of Talagrand (2005) is tailored for this problem. It leads to the choice λ≈logp/n, as in the standard Lasso procedure (which concerns the linear model and least squares loss).

Comment to “Generic chaining and the ℓ1-penalty” by Sara van de Geer

Comment to “Generic chaining and the ℓ1-penalty” by Sara van de Geer

Discussion: “Generic chaining and the [formula omitted]-penalty” by Sara van de Geer

Discussion: “Generic chaining and the [formula omitted]-penalty” by Sara van de Geer

The Bernstein–Orlicz norm and deviation inequalities

We introduce two new concepts designed for the study of empirical processes. First, we introduce a new Orlicz norm which we call the Bernstein–Orlicz norm. This new norm interpolates sub-Gaussian and sub-exponential tail behavior. In particular, we show how this norm can be used to simplify the derivation of deviation inequalities for suprema of collections of random variables. Secondly, we introduce chaining and generic chaining along a tree. These simplify the well-known concepts of chaining and generic chaining. The supremum of the empirical process is then studied as a special case. We show that chaining along a tree can be done using entropy with bracketing. Finally, we establish a deviation inequality for the empirical process for the unbounded case.

On generic chaining and the smallest singular value of random matrices with heavy tails

We present a very general chaining method which allows one to control the supremum of the empirical process suph∈H|N−1∑i=1Nh2(Xi)−Eh2| in rather general situations. We use this method to establish two main results. First, a quantitative (non-asymptotic) version of the celebrated Bai–Yin Theorem on the singular values of a random matrix with i.i.d. entries that have heavy tails, and second, a sharp estimate on the quadratic empirical process when H={〈t,⋅〉:t∈T}, T⊂Rn and μ is an isotropic, unconditional, log-concave measure.

A Bernstein-type inequality for suprema of random processes with applications to model selection in non-Gaussian regression

Let $(X_t)_{t∈T}$ be a family of real-valued centered random variables indexed by a countable set $T$. In the first part of this paper, we establish exponential bounds for the deviation probabilities of the supremum $Z = \sup_{t∈T} X_t$ by using the generic chaining device introduced in Talagrand (Inst. Hautes Études Sci. Publ. Math. 81 (1995) 73–205). Compared to concentration-type inequalities, these bounds offer the advantage of holding under weaker conditions on the family $(X_t)_{t∈T}$. The second part of the paper is oriented toward statistics. We consider the regression setting $Y = f + ξ$, where $f$ is an unknown vector in $ℝ^n$ and $ξ$ is a random vector, the components of which are independent, centered and admit finite Laplace transforms in a neighborhood of $0$. Our aim is to estimate $f$ from the observation of $Y$ by means of a model selection approach among a collection of linear subspaces of $ℝ^n$. The selection procedure we propose is based on the minimization of a penalized criterion, the penalty of which is calibrated by using the deviation bounds established in the first part of this paper. More precisely, we study suprema of random variables of the form $X_t = ∑_{i=1}^n t_iξ_i$, where $t$ varies in the unit ball of a linear subspace of $ℝ^n$. Finally, we show that our estimator satisfies an oracle-type inequality under suitable assumptions on the metric structures of the linear spaces of the collection.

Uniform Uncertainty Principle for Bernoulli and Subgaussian Ensembles

The paper considers random matrices with independent subgaussian columns and provides a new elementary proof of the Uniform Uncertainty Principle for such matrices. The Principle was introduced by Candes, Romberg and Tao in 2004; for subgaussian random matrices it was carlier proved by the present authors, as a consequence of a general result based on a generic chaining method of Talagrand. The present proof combines a simple measure concentration and a covering argument, which are standard tools of high-dimensional convexity.