Minimax Rate Research Articles

Rate-optimal graphon estimation

Network analysis is becoming one of the most active research areas in statistics. Significant advances have been made recently on developing theories, methodologies and algorithms for analyzing networks. However, there has been little fundamental study on optimal estimation. In this paper, we establish optimal rate of convergence for graphon estimation. For the stochastic block model with $k$ clusters, we show that the optimal rate under the mean squared error is $n^{-1}\log k+k^{2}/n^{2}$. The minimax upper bound improves the existing results in literature through a technique of solving a quadratic equation. When $k\leq\sqrt{n\log n}$, as the number of the cluster $k$ grows, the minimax rate grows slowly with only a logarithmic order $n^{-1}\log k$. A key step to establish the lower bound is to construct a novel subset of the parameter space and then apply Fano’s lemma, from which we see a clear distinction of the nonparametric graphon estimation problem from classical nonparametric regression, due to the lack of identifiability of the order of nodes in exchangeable random graph models. As an immediate application, we consider nonparametric graphon estimation in a Hölder class with smoothness $\alpha$. When the smoothness $\alpha\geq1$, the optimal rate of convergence is $n^{-1}\log n$, independent of $\alpha$, while for $\alpha\in(0,1)$, the rate is $n^{-2\alpha/(\alpha+1)}$, which is, to our surprise, identical to the classical nonparametric rate.

Volume Ratio, Sparsity, and Minimaxity Under Unitarily Invariant Norms

This paper studies non-asymptotic minimax estimation of high-dimensional matrices and provides tight minimax rates for a large collection of loss functions in a variety of problems via information-theoretic methods. Based on the convex geometry of finite-dimensional Banach spaces, we first develop a volume ratio approach for determining minimax estimation rates of unconstrained mean matrices under all unitarily invariant norm losses, which turn out to only depend on the norm of identity matrix. In addition, we establish the minimax rates for estimating normal mean matrices with submatrix sparsity, where the sparsity constraint introduces an additional term in the rate which, in contrast to the unconstrained case, is determined by the smoothness (Lipschitz constant) of the norm. This method is also applicable to the low-rank matrix completion problem and extends well beyond the additive noise model. In particular, it yields tight rates in covariance matrix estimation and Poisson rate matrix estimation problems for all unitarily invariant norms.

ESTIMATION OF FUNCTIONALS OF SPARSE COVARIANCE MATRICES.

High-dimensional statistical tests often ignore correlations to gain simplicity and stability leading to null distributions that depend on functionals of correlation matrices such as their Frobenius norm and other ℓ r norms. Motivated by the computation of critical values of such tests, we investigate the difficulty of estimation the functionals of sparse correlation matrices. Specifically, we show that simple plug-in procedures based on thresholded estimators of correlation matrices are sparsity-adaptive and minimax optimal over a large class of correlation matrices. Akin to previous results on functional estimation, the minimax rates exhibit an elbow phenomenon. Our results are further illustrated in simulated data as well as an empirical study of data arising in financial econometrics.

Information bounds for inverse problems with application to deconvolution and Lévy models

Si une fonctionnelle dans un problème inverse non-paramétrique peut être estimée à vitesse paramétrique, alors la vitesse minimax ne donne aucune information sur le caractère mal posé du problème. Pour avoir une borne inférieure plus précise, nous étudions l’efficacité semi-paramétrique dans le sens de Hájek–Le Cam pour l’estimation fonctionnelle dans des modèles indirects réguliers. Ces derniers sont caractérisés comme modèles que l’on peut approcher localement par un modèle linéaire de bruit blanc décrit par l’opérateur de score généralisé. Un théorème de convolution pour des modèles indirects réguliers est prouvé. Ceci s’applique à une large classe de problèmes statistiques inverses, comme montré pour les modèles prototypes du bruit blanc et de la déconvolution. Il est spécialement utile pour des modèles non-linéaires. Nous discutons en détails un modèle non-linéaire de déconvolution où un processus de Lévy est observé à basse fréquence, en obtenant une borne d’information pour l’estimation de fonctionnelles linéaires de la mesure de sauts.

Minimax estimation in sparse canonical correlation analysis

Canonical correlation analysis is a widely used multivariate statistical technique for exploring the relation between two sets of variables. This paper considers the problem of estimating the leading canonical correlation directions in high-dimensional settings. Recently, under the assumption that the leading canonical correlation directions are sparse, various procedures have been proposed for many high-dimensional applications involving massive data sets. However, there has been few theoretical justification available in the literature. In this paper, we establish rate-optimal nonasymptotic minimax estimation with respect to an appropriate loss function for a wide range of model spaces. Two interesting phenomena are observed. First, the minimax rates are not affected by the presence of nuisance parameters, namely the covariance matrices of the two sets of random variables, though they need to be estimated in the canonical correlation analysis problem. Second, we allow the presence of the residual canonical correlation directions. However, they do not influence the minimax rates under a mild condition on eigengap. A generalized sin-theta theorem and an empirical process bound for Gaussian quadratic forms under rank constraint are used to establish the minimax upper bounds, which may be of independent interest.

Adaptive testing on a regression function at a point

We consider the problem of inference on a regression function at a point when the entire function satisfies a sign or shape restriction under the null. We propose a test that achieves the optimal minimax rate adaptively over a range of H\"{o}lder classes, up to a $\log\log n$ term, which we show to be necessary for adaptation. We apply the results to adaptive one-sided tests for the regression discontinuity parameter under a monotonicity restriction, the value of a monotone regression function at the boundary and the proportion of true null hypotheses in a multiple testing problem.

Inference for High-dimensional Differential Correlation Matrices.

Motivated by differential co-expression analysis in genomics, we consider in this paper estimation and testing of high-dimensional differential correlation matrices. An adaptive thresholding procedure is introduced and theoretical guarantees are given. Minimax rate of convergence is established and the proposed estimator is shown to be adaptively rate-optimal over collections of paired correlation matrices with approximately sparse differences. Simulation results show that the procedure significantly outperforms two other natural methods that are based on separate estimation of the individual correlation matrices. The procedure is also illustrated through an analysis of a breast cancer dataset, which provides evidence at the gene co-expression level that several genes, of which a subset has been previously verified, are associated with the breast cancer. Hypothesis testing on the differential correlation matrices is also considered. A test, which is particularly well suited for testing against sparse alternatives, is introduced. In addition, other related problems, including estimation of a single sparse correlation matrix, estimation of the differential covariance matrices, and estimation of the differential cross-correlation matrices, are also discussed.

Nonparametric estimation of the mixing density using polynomials

We consider the problem of estimating the mixing density $f$ from $n$ i.i.d. observations distributed according to a mixture density with unknown mixing distribution. In contrast with finite mixtures models, here the distribution of the hidden variable is not bounded to a finite set but is spread out over a given interval. We propose an approach to construct an orthogonal series estimator of the mixing density $f$ involving Legendre polynomials. The construction of the orthonormal sequence varies from one mixture model to another. Minimax upper and lower bounds of the mean integrated squared error are provided which apply in various contexts. In the specific case of exponential mixtures, it is shown that the estimator is adaptive over a collection of specific smoothness classes, more precisely, there exists a constant $A>0$ such that, when the order $m$ of the projection estimator verifies $m\sim A \log(n)$, the estimator achieves the minimax rate over this collection. Other cases are investigated such as Gamma shape mixtures and scale mixtures of compactly supported densities including Beta mixtures. Finally, a consistent estimator of the support of the mixing density $f$ is provided.

Functional sufficient dimension reduction: Convergence rates and multiple functional case

Although sufficient dimension reduction for functional data has received some attention in the literature, its theoretical properties are less understood. Besides, the current literature only focused on sliced inverse regression (SIR). In this paper we consider functional version of SIR and SAVE (sliced average variance estimation) via a Tikhonov regularization approach. Besides consistency, we show that their convergence rates are the same as the minimax rates for functional linear regression, which we think is an interesting theoretical result given that sufficient dimension reduction is much more flexible than functional linear regression. In sufficient dimension reduction, it is well known that estimation of multiple directions requires extraction of multiple eigenfunctions. We also consider multiple functional dimension reduction, in which one eigenfunction surprisingly recovers multiple index functions at once, despite its similarity with single functional case. The numerical properties are illustrated using several simulation examples as well as a Japanese weather dataset.

Minimax rate of convergence for an estimator of the functional component in a semiparametric multivariate partially linear model

A multivariate semiparametric partial linear model for both fixed and random design cases is considered. Earlier, in Brown et al. (2014), the model has been analyzed using a difference sequence approach. In particular, the functional component has been estimated using a multivariate Nadaraya–Watson kernel smoother of the residuals of the linear fit. Moreover, this functional component estimator has been shown to be rate optimal if the Lipschitz smoothness index exceeds half the dimensionality of the functional component domain. In the current manuscript, we take this research further and show that, for both fixed and random designs, the rate achieved is the minimax rate under both risk at a point and the L 2 risk. The result is achieved by proving lower bounds on both pointwise risk and the L 2 risk of possible estimators of the functional component.

On the sample covariance matrix estimator of reduced effective rank population matrices, with applications to fPCA

This work provides a unified analysis of the properties of the sample covariance matrix $\Sigma_{n}$ over the class of $p\times p$ population covariance matrices $\Sigma$ of reduced effective rank $r_{e}(\Sigma)$. This class includes scaled factor models and covariance matrices with decaying spectrum. We consider $r_{e}(\Sigma)$ as a measure of matrix complexity, and obtain sharp minimax rates on the operator and Frobenius norm of $\Sigma_{n}-\Sigma$, as a function of $r_{e}(\Sigma)$ and $\|\Sigma\|_{2}$, the operator norm of $\Sigma$. With guidelines offered by the optimal rates, we define classes of matrices of reduced effective rank over which $\Sigma_{n}$ is an accurate estimator. Within the framework of these classes, we perform a detailed finite sample theoretical analysis of the merits and limitations of the empirical scree plot procedure routinely used in PCA. We show that identifying jumps in the empirical spectrum that consistently estimate jumps in the spectrum of $\Sigma$ is not necessarily informative for other goals, for instance for the selection of those sample eigenvalues and eigenvectors that are consistent estimates of their population counterparts. The scree plot method can still be used for selecting consistent eigenvalues, for appropriate threshold levels. We provide a threshold construction and also give a rule for checking the consistency of the corresponding sample eigenvectors. We specialize these results and analysis to population covariance matrices with polynomially decaying spectra, and extend it to covariance operators with polynomially decaying spectra. An application to fPCA illustrates how our results can be used in functional data analysis.

Minimax-optimal nonparametric regression in high dimensions

Minimax $L_2$ risks for high-dimensional nonparametric regression are derived under two sparsity assumptions: (1) the true regression surface is a sparse function that depends only on $d=O(\log n)$ important predictors among a list of $p$ predictors, with $\log p=o(n)$; (2) the true regression surface depends on $O(n)$ predictors but is an additive function where each additive component is sparse but may contain two or more interacting predictors and may have a smoothness level different from other components. For either modeling assumption, a practicable extension of the widely used Bayesian Gaussian process regression method is shown to adaptively attain the optimal minimax rate (up to $\log n$ terms) asymptotically as both $n,p\to\infty$ with $\log p=o(n)$.

Rate-optimal posterior contraction for sparse PCA

Principal component analysis (PCA) is possibly one of the most widely used statistical tools to recover a low-rank structure of the data. In the high-dimensional settings, the leading eigenvector of the sample covariance can be nearly orthogonal to the true eigenvector. A sparse structure is then commonly assumed along with a low rank structure. Recently, minimax estimation rates of sparse PCA were established under various interesting settings. On the other side, Bayesian methods are becoming more and more popular in high-dimensional estimation, but there is little work to connect frequentist properties and Bayesian methodologies for high-dimensional data analysis. In this paper, we propose a prior for the sparse PCA problem and analyze its theoretical properties. The prior adapts to both sparsity and rank. The posterior distribution is shown to contract to the truth at optimal minimax rates. In addition, a computationally efficient strategy for the rank-one case is discussed.

Penalized least squares estimation in the additive model with different smoothness for the components

We consider the standard additive regression model consisting of two components f0 and g0. The first component f0 is assumed to be in some sense “smoother” than the second g0. It is known that in that case one can construct estimators that estimate the smoother component f0 with fast minimax rate as if the non-smooth component g0 were known. Our contribution shows that this phenomenon also occurs when one uses the penalized least squares estimator (fˆ,gˆ) of (f0,g0). We describe smoothness in terms of a semi-norm on the class of regression functions. This covers the case of Sobolev smoothness where the penalized estimator is a standard spline estimator. The theory is illustrated by a simulation study. Our proofs rely on recent results from empirical process theory.

Penalized contrast estimation in functional linear models with circular data

Our aim is to estimate the unknown slope function in the functional linear model when the response Y is real and the random function X is a second-order stationary and periodic process. We obtain our estimator by minimizing a standard (and very simple) mean-square contrast on linear finite dimensional spaces spanned by trigonometric bases. Our approach provides a penalization procedure which allows to automatically select the adequate dimension, in a non-asymptotic point of view. In fact, we can show that our penalized estimator reaches the optimal (minimax) rate of convergence in the sense of the prediction error. We complete the theoretical results by a simulation study and a real example that illustrates how the procedure works in practice.

Adaptive pointwise estimation for pure jump Lévy processes

This paper is concerned with adaptive kernel estimation of the Levy density $$N(x)$$ for bounded-variation pure-jump Levy processes. The sample path is observed at $$n$$ discrete instants in the “high frequency” context ( $$\Delta = \Delta (n) $$ tends to zero while $$n \Delta $$ tends to infinity). We construct a collection of kernel estimators of the function $$g(x)=xN(x)$$ and propose a method of local adaptive selection of the bandwidth. We provide an oracle inequality and a rate of convergence for the quadratic pointwise risk. This rate is proved to be the optimal minimax rate. We give examples and simulation results for processes fitting in our framework. We also consider the case of irregular sampling.

Tight minimax rates for manifold estimation under Hausdorff loss

This paper deals with minimax rates of convergence for manifold estimation. A new lower bound is obtained by a novel construction of two sets of manifolds and an application of convex hull testing method of Le Cam (1973). The minimax lower bound matches the upper bound up to a constant factor considered by Genovese et al. (2012b).

Optimal Prediction in an Additive Functional Model

The functional generalized additive model (FGAM) provides a more flexible nonlinear functional regression model than the well-studied functional linear regression model. This paper restricts attention to the FGAM with identity link and additive errors, which we will call the additive functional model, a generalization of the functional linear model. This paper studies the minimax rate of convergence of predictions from the additive functional model in the framework of reproducing kernel Hilbert space. It is shown that the optimal rate is determined by the decay rate of the eigenvalues of a specific kernel function, which in turn is determined by the reproducing kernel and the joint distribution of any two points in the random predictor function. For the special case of the functional linear model, this kernel function is jointly determined by the covariance function of the predictor function and the reproducing kernel. The easily implementable roughness-regularized predictor is shown to achieve the optimal rate of convergence. Numerical studies are carried out to illustrate the merits of the predictor. Our simulations and real data examples demonstrate a competitive performance against the existing approach.

Asymptotic performance of projection estimators in standard and hyperbolic wavelet bases

We provide a novel treatment of the ability of the standard (wavelet-tensor) and of the hyperbolic (tensor product) wavelet bases to build nonparametric estimators of multivariate functions. First, we give new results about the limitations of wavelet estimators based on the standard wavelet basis regarding their inability to optimally reconstruct functions with anisotropic smoothness. Next, we provide optimal or near optimal rates at which both linear and non-linear hyperbolic wavelet estimators are well-suited to reconstruct functions from anisotropic Besov spaces and subsequently we characterize the set of all the functions that are well reconstructed by these methods with respect to these rates. As a first main result, we furnish novel arguments to understand the primordial role of sparsity and thresholding in multivariate contexts, in particular by showing a stronger exposure of linear methods to the curse of dimensionality. Second, we propose an adaptation of the well known block thresholding method to a hyperbolic wavelet basis and show its ability to estimate functions with anisotropic smoothness at the optimal minimax rate. Therefore, we prove the pertinence of horizontal information pooling even in high dimensional settings. Numerical experiments illustrate the finite samples properties of the studied estimators.

Minimax rate of convergence and the performance of empirical risk minimization in phase recovery

We study the performance of Empirical Risk Minimization in both noisy and noiseless phase retrieval problems, indexed by subsets of $\mathbb{R}^n$ and relative to subgaussian sampling; that is, when the given data is $y_i=\left a_i,x_0\right^2+w_i$ for a subgaussian random vector $a$, independent noise $w$ and a fixed but unknown $x_0$ that belongs to a given subset of $\mathbb{R}^n$. We show that ERM produces $\hat{x}$ whose Euclidean distance to either $x_0$ or $-x_0$ depends on the gaussian mean-width of the indexing set and on the signal-to-noise ratio of the problem. The bound coincides with the one for linear regression when $\|x_0\|_2$ is of the order of a constant. In addition, we obtain a minimax lower bound for the problem and identify sets for which ERM is a minimax procedure. As examples, we study the class of $d$-sparse vectors in $\mathbb{R}^n$ and the unit ball in $\ell_1^n$.