Gaussian Sequence Model Research Articles

Minimax rates for sparse signal detection under correlation

Abstract We fully characterize the nonasymptotic minimax separation rate for sparse signal detection in the Gaussian sequence model with $p$ equicorrelated observations, generalizing a result of Collier, Comminges and Tsybakov. As a consequence of the rate characterization, we find that strong correlation is a blessing, moderate correlation is a curse and weak correlation is irrelevant. Moreover, the threshold correlation level yielding a blessing exhibits phase transitions at the $\sqrt{p}$ and $p-\sqrt{p}$ sparsity levels. We also establish the emergence of new phase transitions in the minimax separation rate with a subtle dependence on the correlation level. Additionally, we study group structured correlations and derive the minimax separation rate in a model including multiple random effects. The group structure turns out to fundamentally change the detection problem from the equicorrelated case and different phenomena appear in the separation rate.

On the Minimax Rate of the Gaussian Sequence Model Under Bounded Convex Constraints

We determine the exact minimax rate of a Gaussian sequence model under bounded convex constraints, purely in terms of the local geometry of the given constraint set <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$K$ </tex-math></inline-formula> . Our main result shows that the minimax risk <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\vphantom {_{\int _{\int }}}$ </tex-math></inline-formula> (up to constant factors) under the squared <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\ell _{2}$ </tex-math></inline-formula> loss is given by <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\varepsilon ^{*2} \wedge \mathrm {diam} (K)^{2}$ </tex-math></inline-formula> with <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\varepsilon ^{*} = \sup \bigg \{\varepsilon: ({\varepsilon ^{2}}/{\sigma ^{2}}) \leq \log M^{\mathrm {loc}}(\varepsilon)\bigg \}$ </tex-math></inline-formula> , where <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\log M^{\mathrm {loc}}(\varepsilon)$ </tex-math></inline-formula> denotes the local entropy of the set <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$K$ </tex-math></inline-formula> , and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\sigma ^{2}$ </tex-math></inline-formula> is the variance of the noise. We utilize our abstract result to re-derive known minimax rates for some special sets <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$K$ </tex-math></inline-formula> such as hyperrectangles, ellipses, and more generally quadratically convex orthosymmetric sets. Finally, we extend our results to the unbounded case with known <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\sigma ^{2}$ </tex-math></inline-formula> to show that the minimax rate in that case is <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\varepsilon ^{*2}$ </tex-math></inline-formula> .

Adaptive and efficient estimation in the Gaussian sequence model

This paper deals with the problem of estimating the mean of an infinite-dimensional Gaussian vector by the principle of minimizing unbiased risk estimation (URE). The aim is to obtain an adaptive estimator to mimic the oracle with the smallest squared L2 loss/risk in a given class of linear estimators Λ. It is noticed that there is an essential balance between enlarging Λ for more efficient oracle risk and contracting Λ for the adaptivity to the oracle. This paper proves that this balance can be well achieved by minimizing URE in a list of intermediate classes between the projection and monotone classes. The resulting estimators are adaptive on the nonparametric space and have more efficient risks than the projection estimators under certain conditions.

SUPERCONSISTENCY OF TESTS IN HIGH DIMENSIONS

To assess whether there is some signal in a big database, aggregate tests for the global null hypothesis of no effect are routinely applied in practice before more specialized analysis is carried out. Although a plethora of aggregate tests is available, each test has its strengths but also its blind spots. In a Gaussian sequence model, we study whether it is possible to obtain a test with substantially better consistency properties than the likelihood ratio (LR; i.e., Euclidean norm-based) test. We establish an impossibility result, showing that in the high-dimensional framework we consider, the set of alternatives for which a test may improve upon the LR test (i.e., its superconsistency points) is always asymptotically negligible in a relative volume sense.

Calibrating the Scan Statistic: Finite Sample Performance Versus Asymptotics

Abstract We consider the problem of detecting an elevated mean on an interval with unknown location and length in the univariate Gaussian sequence model. Recent results have shown that using scale-dependent critical values for the scan statistic allows to attain asymptotically optimal detection simultaneously for all signal lengths, thereby improving on the traditional scan, but this procedure has been criticised for losing too much power for short signals. We explain this discrepancy by showing that these asymptotic optimality results will necessarily be too imprecise to discern the performance of scan statistics in a practically relevant way, even in a large sample context. Instead, we propose to assess the performance with a new finite sample criterion. We then present three calibrations for scan statistics that perform well across a range of relevant signal lengths: The first calibration uses a particular adjustment to the critical values and is therefore tailored to the Gaussian case. The second calibration uses a scale-dependent adjustment to the significance levels rather than to the critical values and this adjustment is therefore applicable to arbitrary known null distributions. The third calibration restricts the scan to a particular sparse subset of the scan windows and then applies a weighted Bonferroni adjustment to the corresponding test statistics. This Bonferroni scan is also applicable to arbitrary null distributions and in addition is very simple to implement. We show how to apply these calibrations for scanning in a number of distributional settings: for normal observations with an unknown baseline and a known or unknown constant variance, for observations from a natural exponential family, for potentially heteroscadastic observations from a symmetric density by employing self-normalisation in a novel way, and for exchangeable observations using tests based on permutations, ranks or signs.

On a Phase Transition in General Order Spline Regression

In the Gaussian sequence model Y = θ0+ε in Rn, we study the fundamental limit of statistical estimation when the signal θ0 belongs to a class Θn(d, d0, k) of (generalized) splines with free knots located at equally spaced design points. Here d is the degree of the spline, d0 is the order of differentiability at each inner knot, and k is the maximal number of pieces. We show that, given any integer d≥0 and d0 ∈ {-1, 0, ..., d-1}, the minimax rate of estimation over Θn(d, d0, k) exhibits the following phase transition: inf~θ supθ∈n(d,d0,k)Eθ‖~θ - θ‖2 ≍ d {k log log(16n/k), 2 ≤ k ≤ k0, k log(en/k), k ≥ k0 + 1. The transition boundary k0, which takes the form ⌊(d + 1)/(d - d0)⌋ + 1, demonstrates the critical role of the regularity parameter d0 in the separation between a faster log log(16n) and a slower log(en) rate. We further show that, once encouraging an additional ‘d-monotonicity’ shape constraint (including monotonicity for d = 0 and convexity for d = 1), the above phase transition is removed and the faster k log log(16n/k) rate can be achieved for all k. These results provide theoretical support for developing ℓ0-penalized (shape-constrained) spline regression procedures as useful alternatives to ℓ1- and ℓ2-penalized ones.

Distributed nonparametric function estimation: Optimal rate of convergence and cost of adaptation

Distributed minimax estimation and distributed adaptive estimation under communication constraints for Gaussian sequence model and white noise model are studied. The minimax rate of convergence for distributed estimation over a given Besov class, which serves as a benchmark for the cost of adaptation, is established. We then quantify the exact communication cost for adaptation and construct an optimally adaptive procedure for distributed estimation over a range of Besov classes. The results demonstrate significant differences between nonparametric function estimation in the distributed setting and the conventional centralized setting. For global estimation, adaptation in general cannot be achieved for free in the distributed setting. The new technical tools to obtain the exact characterization for the cost of adaptation can be of independent interest.

High-dimensional asymptotics of likelihood ratio tests in the Gaussian sequence model under convex constraints

In the Gaussian sequence model Y=μ+ξ, we study the likelihood ratio test (LRT) for testing H0:μ=μ0 versus H1:μ∈K, where μ0∈K, and K is a closed convex set in Rn. In particular, we show that under the null hypothesis, normal approximation holds for the log-likelihood ratio statistic for a general pair (μ0,K), in the high-dimensional regime where the estimation error of the associated least squares estimator diverges in an appropriate sense. The normal approximation further leads to a precise characterization of the power behavior of the LRT in the high-dimensional regime. These characterizations show that the power behavior of the LRT is in general nonuniform with respect to the Euclidean metric, and illustrate the conservative nature of existing minimax optimality and suboptimality results for the LRT. A variety of examples, including testing in the orthant/circular cone, isotonic regression, Lasso and testing parametric assumptions versus shape-constrained alternatives, are worked out to demonstrate the versatility of the developed theory.

Second-order Stein: SURE for SURE and other applications in high-dimensional inference

Stein’s formula states that a random variable of the form z⊤f(z)−divf(z) is mean-zero for all functions f with integrable gradient. Here, divf is the divergence of the function f and z is a standard normal vector. This paper aims to propose a second-order Stein formula to characterize the variance of such random variables for all functions f(z) with square integrable gradient, and to demonstrate the usefulness of this second-order Stein formula in various applications. In the Gaussian sequence model, a remarkable consequence of Stein’s formula is Stein’s Unbiased Risk Estimate (SURE), an unbiased estimate of the mean squared risk for almost any given estimator μˆ of the unknown mean vector. A first application of the second-order Stein formula is an Unbiased Risk Estimate for SURE itself (SURE for SURE): an unbiased estimate providing information about the squared distance between SURE and the squared estimation error of μˆ. SURE for SURE has a simple form as a function of the data and is applicable to all μˆ with square integrable gradient, for example, the Lasso and the Elastic Net. In addition to SURE for SURE, the following statistical applications are developed: (1) upper bounds on the risk of SURE when the estimation target is the mean squared error; (2) confidence regions based on SURE and using the second-order Stein formula; (3) oracle inequalities satisfied by SURE-tuned estimates under a mild Lipschtiz assumption; (4) an upper bound on the variance of the size of the model selected by the Lasso, and more generally an upper bound on the variance of the empirical degrees-of-freedom of convex penalized estimators; (5) explicit expressions of SURE for SURE for the Lasso and the Elastic Net; (6) in the linear model, a general semiparametric scheme to de-bias a differentiable initial estimator for the statistical inference of a low-dimensional projection of the unknown regression coefficient vector, with a characterization of the variance after debiasing; and (7) an accuracy analysis of a Gaussian Monte Carlo scheme to approximate the divergence of functions f:Rn→Rn.

Coverage of credible intervals in nonparametric monotone regression

For nonparametric univariate regression under a monotonicity constraint on the regression function f, we study the coverage of a Bayesian credible interval for f(x0), where x0 is an interior point. Analysis of the posterior becomes a lot more tractable by considering a “projection-posterior” distribution based on a finite random series of step functions with normal basis coefficients as a prior for f. A sample f from the resulting conjugate posterior distribution is projected on the space of monotone increasing functions to obtain a monotone function f∗ closest to f, inducing the “projection-posterior.” We use projection-posterior samples to obtain credible intervals for f(x0). We obtain the asymptotic coverage of the credible interval thus constructed and observe that it is free of nuisance parameters involving the true function. We observe a very interesting phenomenon that the coverage is typically higher than the nominal credibility level, the opposite of a phenomenon observed by Cox (Ann. Statist. 21 (1993) 903–923) in the Gaussian sequence model. We further show that a recalibration gives the right asymptotic coverage by starting from a lower credibility level that can be explicitly calculated.

Approximate separability of symmetrically penalized least squares in high dimensions: characterization and consequences

Abstract We show that the high-dimensional behavior of symmetrically penalized least squares with a possibly non-separable, symmetric, convex penalty in both (i) the Gaussian sequence model and (ii) the linear model with uncorrelated Gaussian designs nearly agrees with the behavior of least squares with an appropriately chosen separable penalty in these same models. This agreement is established by finite-sample concentration inequalities which precisely characterize the behavior of symmetrically penalized least squares in both models via a comparison to a simple scalar statistical model. The concentration inequalities are novel in their precision and generality. Our results help clarify that the role non-separability can play in high-dimensional M-estimation. In particular, if the empirical distribution of the coordinates of the parameter is known—exactly or approximately—there are at most limited advantages to use non-separable, symmetric penalties over separable ones. In contrast, if the empirical distribution of the coordinates of the parameter is unknown, we argue that non-separable, symmetric penalties automatically implement an adaptive procedure, which we characterize. We also provide a partial converse which characterizes the adaptive procedures which can be implemented in this way.

Online control of the familywise error rate.

Biological research often involves testing a growing number of null hypotheses as new data are accumulated over time. We study the problem of online control of the familywise error rate, that is testing an a priori unbounded sequence of hypotheses (p-values) one by one over time without knowing the future, such that with high probability there are no false discoveries in the entire sequence. This paper unifies algorithmic concepts developed for offline (single batch) familywise error rate control and online false discovery rate control to develop novel online familywise error rate control methods. Though many offline familywise error rate methods (e.g., Bonferroni, fallback procedures and Sidak's method) can trivially be extended to the online setting, our main contribution is the design of new, powerful, adaptive online algorithms that control the familywise error rate when the p-values are independent or locally dependent in time. Our numerical experiments demonstrate substantial gains in power, that are also formally proved in an idealized Gaussian sequence model. A promising application to the International Mouse Phenotyping Consortium is described.

Bayesian variance estimation in the Gaussian sequence model with partial information on the means

Consider the Gaussian sequence model under the additional assumption that a fixed fraction of the means is known. We study the problem of variance estimation from a frequentist Bayesian perspective. The maximum likelihood estimator (MLE) for $\sigma^{2}$ is biased and inconsistent. This raises the question whether the posterior is able to correct the MLE in this case. By developing a new proving strategy that uses refined properties of the posterior distribution, we find that the marginal posterior is inconsistent for any i.i.d. prior on the mean parameters. In particular, no assumption on the decay of the prior needs to be imposed. Surprisingly, we also find that consistency can be retained for a hierarchical prior based on Gaussian mixtures. In this case we also establish a limiting shape result and determine the limit distribution. In contrast to the classical Bernstein-von Mises theorem, the limit is non-Gaussian. We show that the Bayesian analysis leads to new statistical estimators outperforming the correctly calibrated MLE in a numerical simulation study.

Optimal Rates and Tradeoffs in Multiple Testing

Multiple hypothesis testing is a central topic in statistics, but despite abundant work on the false discovery rate (FDR) and the corresponding Type-II error concept known as the false non-discovery rate (FNR), a fine-grained understanding of the fundamental limits of multiple testing has not been developed. Our main contribution is to derive a precise non-asymptotic tradeoff between FNR and FDR for a variant of the generalized Gaussian sequence model. Our analysis is flexible enough to permit analyses of settings where the problem parameters vary with the number of hypotheses $n$, including various sparse and dense regimes (with $o(n)$ and $\mathcal{O}(n)$ signals). Moreover, we prove that the Benjamini-Hochberg algorithm as well as the Barber-Candes algorithm are both rate-optimal up to constants across these regimes.

The geometry of hypothesis testing over convex cones: Generalized likelihood ratio tests and minimax radii

We consider a compound testing problem within the Gaussian sequence model in which the null and alternative are specified by a pair of closed, convex cones. Such cone testing problem arises in various applications, including detection of treatment effects, trend detection in econometrics, signal detection in radar processing and shape-constrained inference in nonparametric statistics. We provide a sharp characterization of the GLRT testing radius up to a universal multiplicative constant in terms of the geometric structure of the underlying convex cones. When applied to concrete examples, this result reveals some interesting phenomena that do not arise in the analogous problems of estimation under convex constraints. In particular, in contrast to estimation error, the testing error no longer depends purely on the problem complexity via a volume-based measure (such as metric entropy or Gaussian complexity); other geometric properties of the cones also play an important role. In order to address the issue of optimality, we prove information-theoretic lower bounds for the minimax testing radius again in terms of geometric quantities. Our general theorems are illustrated by examples including the cases of monotone and orthant cones, and involve some results of independent interest.

Rate optimal estimation of quadratic functionals in inverse problems with partially unknown operator and application to testing problems

We consider the estimation of quadratic functionals in a Gaussian sequence model where the eigenvalues are supposed to be unknown and accessible through noisy observations only. Imposing smoothness assumptions both on the signal and the sequence of eigenvalues, we develop a minimax theory for this problem. We propose a truncated series estimator and show that it attains the optimal rate of convergence if the truncation parameter is chosen appropriately. Consequences for testing problems in inverse problems are equally discussed: in particular, the minimax rates of testing for signal detection and goodness-of-fit testing are derived.

Bayesian inverse problems with partial observations

We study a nonparametric Bayesian approach to linear inverse problems under discrete observations. We use the discrete Fourier transform to convert our model into a truncated Gaussian sequence model, that is closely related to the classical Gaussian sequence model. Upon placing the truncated series prior on the unknown parameter, we show that as the number of observations n→∞, the corresponding posterior distribution contracts around the true parameter at a rate depending on the smoothness of the true parameter and the prior, and the ill-posedness degree of the problem. Correct combinations of these values lead to optimal posterior contraction rates (up to logarithmic factors). Similarly, the frequentist coverage of Bayesian credible sets is shown to be dependent on a combination of smoothness of the true parameter and the prior, and the ill-posedness of the problem. Oversmoothing priors lead to zero coverage, while undersmoothing priors produce highly conservative results. Finally, we illustrate our theoretical results by numerical examples.

On spike and slab empirical Bayes multiple testing

This paper explores a connection between empirical Bayes posterior distributions and false discovery rate (FDR) control. In the Gaussian sequence model this work shows that empirical Bayes-calibrated spike and slab posterior distributions allow a correct FDR control under sparsity. Doing so, it offers a frequentist theoretical validation of empirical Bayes methods in the context of multiple testing. Our theoretical results are illustrated with numerical experiments.

Minimax Euclidean separation rates for testing convex hypotheses in $\mathbb{R}^{d}$

We consider composite-composite testing problems for the expectation in the Gaussian sequence model where the null hypothesis corresponds to a closed convex subset $\mathcal{C}$ of $\mathbb{R}^{d}$. We adopt a minimax point of view and our primary objective is to describe the smallest Euclidean distance between the null and alternative hypotheses such that there is a test with small total error probability. In particular, we focus on the dependence of this distance on the dimension $d$ and variance $\frac{1}{n}$ giving rise to the minimax separation rate. In this paper we discuss lower and upper bounds on this rate for different smooth and non-smooth choices for $\mathcal{C}$.

Minimax goodness-of-fit testing in ill-posed inverse problems with partially unknown operators

Nous considerons un modele sequentiel gaussien incluant des modeles de type « probleme inverse » comme cas particulier. Nous supposons que l’operateur associe est partiellement connu au sens ou les fonctions propres associees a la decomposition en valeurs singulieres sont connues, mais pas les valeurs propres. Ces dernieres sont malgre tout observables en presence d’un bruit gaussien. Dans ce cadre, nous nous concentrons sur l’etude d’un probleme de test d’adequation. En utilisant a la fois une condition d’energie sur le signal considere ainsi que des proprietes de regularite, nous etablissons des bornes superieures et inferieures pour la vitesse minimax de separation dans un cadre asymptotique, i.e. pour des valeurs fixees des niveaux de bruit impliques dans les observations. Pour finir, des exemples particuliers de problemes inverses et de regularites sont consideres et les vitesses minimax de separation correspondantes sont discutees en detail afin d’illustrer les resultats obtenus.