Minimax Rate Research Articles

Block thresholding for wavelet-based estimation of function derivatives from a heteroscedastic multichannel convolution model

We observe $n$ heteroscedastic stochastic processes $\{Y_v(t)\}_{v}$, where for any $v\in\{1,\ldots,n\}$ and $t \in [0,1]$, $Y_v(t)$ is the convolution product of an unknown function $f$ and a known blurring function $g_v$ corrupted by Gaussian noise. Under an ordinary smoothness assumption on $g_1,\ldots,g_n$, our goal is to estimate the $d$-th derivatives (in weak sense) of $f$ from the observations. We propose an adaptive estimator based on wavelet block thresholding, namely the "BlockJS estimator". Taking the mean integrated squared error (MISE), our main theoretical result investigates the minimax rates over Besov smoothness spaces, and shows that our block estimator can achieve the optimal minimax rate, or is at least nearly-minimax in the least favorable situation. We also report a comprehensive suite of numerical simulations to support our theoretical findings. The practical performance of our block estimator compares very favorably to existing methods of the literature on a large set of test functions.

Adaptive estimation of convex polytopes and convex sets from noisy data

We estimate convex polytopes and general convex sets in $\mathbb{R}^{d}$, $d\geq 2$ in the regression framework. We measure the risk of our estimators using a $L^{1}$-type loss function and prove upper bounds on these risks. We show, in the case of convex polytopes, that these estimators achieve the minimax rate. For convex polytopes, this minimax rate is $\frac{\ln n}{n}$, which differs from the parametric rate for non-regular families by a logarithmic factor, and we show that this extra factor is essential. Using polytopal approximations we extend our results to general convex sets, and we achieve the minimax rate up to a logarithmic factor. In addition we provide an estimator that is adaptive with respect to the number of vertices of the unknown polytope, and we prove that this estimator is optimal in all classes of convex polytopes with a given number of vertices.

Nonparametric estimation of a renewal reward process from discrete data

We study the nonparametric estimation of the jump density of a renewal reward process from one discretely observed sample path over [0,T]. We consider regimes where the sampling rate goes to 0 as T tends to infinity. We propose an adaptive wavelet threshold density estimator and study its performance for the Lp loss, over Besov spaces. We achieve minimax rates of convergence for sampling rates that vanish with T at arbitrary polynomial rate. In the same spirit as Buchmann and Grubel (2003) the estimation procedure is based on the inversion of the compounding operator. The inverse has no closed form expression and is approached with a fixed point technique.

Estimation of the activity of jumps in time-changed Lévy models

In this paper we consider a class of time-changed Lévy processes that can be represented in the form $Y_{s}=X_{\mathcal{T}(s)}$, where $X$ is a Lévy process and $\mathcal{T}$ is a non-negative and non-decreasing stochastic process independent of $X$. The aim of this work is to infer on the Blumenthal-Getoor index of the process $X$ from low-frequency observations of the time-changed Lévy process $Y$. We propose a consistent estimator for this index, derive the minimax rates of convergence and show that these rates can not be improved in general. The performance of the estimator is illustrated by numerical examples.

Model selection in regression under structural constraints

The paper considers model selection in regression under the additional structural constraints on admissible models where the number of potential predictors miht be even larger than the available sample size. We develop a Bayesian formalism which is used as a natural tool for generating a wide class of model selection criteria based on penalized least squares estimation with various complexity penalties associated with a prior on a model size. The resulting criteria are adaptive to structural constraints. We establish the upper bound for the quadratic risk of the resulting MAP estimator and the corresponding lower bound for the minimax risk over a set of admissible models of a given size. We then specify the class of priors (and, therefore, the class of complexity penalties) where for the “nearly-orthogonal” design the MAP estimator is asymptotically at least nearly-minimax (up to a log-factor) simultaneously over an entire range of sparse and dense setups. Moreover, when the numbers of admissible models are “small” (e.g., ordered variable selection) or, on the opposite, for the case of complete variable selection, the proposed estimator achieves the exact minimax rates.

Bayesian inverse problems with non-conjugate priors

We investigate the frequentist posterior contraction rate of nonparametric Bayesian procedures in linear inverse problems in both the mildly and severely ill-posed cases. A theorem is proved in a general Hilbert space setting under approximation-theoretic assumptions on the prior. The result is applied to non-conjugate priors, notably sieve and wavelet series priors, as well as in the conjugate setting. In the mildly ill-posed setting minimax optimal rates are obtained, with sieve priors being rate adaptive over Sobolev classes. In the severely ill-posed setting, oversmoothing the prior yields minimax rates. Previously established results in the conjugate setting are obtained using this method. Examples of applications include deconvolution, recovering the initial condition in the heat equation and the Radon transform.

Why minimax is not that pessimistic

In nonparametric statistics a classical optimality criterion for estimation procedures is provided by the minimax rate of convergence. However this point of view can be subject to controversy as it requires to look for the worst behavior of an estimation procedure in a given space. The purpose of this paper is to introduce a new criterion based on generic behavior of estimators. We are here interested in the rate of convergence obtained with some classical estimators on almost every, in the sense of prevalence, function in a Besov space. We also show that generic results coincide with minimax ones in these cases.

Minimax optimal estimation of general bandable covariance matrices

Cai et al. (2010) [4] have studied the minimax optimal estimation of a collection of large bandable covariance matrices whose off-diagonal entries decay to zero at a polynomial rate. They have shown that the minimax optimal procedures are fundamentally different under Frobenius and spectral norms, regardless of the rate of polynomial decay. To gain more insight into this interesting problem, we study minimax estimation of large bandable covariance matrices over a parameter space characterized by a general positive decay function. We obtain explicit results to show how the decay function determines the minimax rates of convergence and the optimal procedures. From the general minimax analysis we find that for certain decay functions there is a tapering estimator that simultaneously attains the minimax optimal rates of convergence under the two norms. Moreover, we show that under the ultra-high dimension scenario it is possible to achieve adaptive minimax optimal estimation under the spectral norm. These new findings complement previous work.

Adaptive nonparametric density estimation with missing observations

It is well known that if some observations in a sample from the probability density are not available, then in general the density cannot be estimated. A possible remedy is to use an auxiliary variable that explains the missing mechanism. For this setting a data-driven estimator is proposed that mimics performance of an oracle that knows all observations from the sample. It is also proved that the estimator adapts to unknown smoothness of the density and its mean integrated squared error converges with a minimax rate. A numerical study, together with the analysis of a real data, shows that the estimator is feasible for small samples.

Estimation in functional regression for general exponential families

This paper studies a class of exponential family models whose canonical parameters are specified as linear functionals of an unknown infinite-dimensional slope function. The optimal minimax rates of convergence for slope function estimation are established. The estimators that achieve the optimal rates are constructed by constrained maximum likelihood estimation with parameters whose dimension grows with sample size. A change-of-measure argument, inspired by Le Cam's theory of asymptotic equivalence, is used to eliminate the bias caused by the nonlinearity of exponential family models.

Optimal rates of convergence for sparse covariance matrix estimation

This paper considers estimation of sparse covariance matrices and establishes the optimal rate of convergence under a range of matrix operator norm and Bregman divergence losses. A major focus is on the derivation of a rate sharp minimax lower bound. The problem exhibits new features that are significantly different from those that occur in the conventional nonparametric function estimation problems. Standard techniques fail to yield good results, and new tools are thus needed. We first develop a lower bound technique that is particularly well suited for treating "two-directional" problems such as estimating sparse covariance matrices. The result can be viewed as a generalization of Le Cam's method in one direction and Assouad's Lemma in another. This lower bound technique is of independent interest and can be used for other matrix estimation problems. We then establish a rate sharp minimax lower bound for estimating sparse covariance matrices under the spectral norm by applying the general lower bound technique. A thresholding estimator is shown to attain the optimal rate of convergence under the spectral norm. The results are then extended to the general matrix $\ell_w$ operator norms for $1\le w\le \infty$. In addition, we give a unified result on the minimax rate of convergence for sparse covariance matrix estimation under a class of Bregman divergence losses.

SURE-tuned tapering estimation of large covariance matrices

Bandable covariance matrices are often used to model the dependence structure of variables that follow a nature order. It has been shown that the tapering covariance estimator attains the optimal minimax rates of convergence for estimating large bandable covariance matrices. The estimation risk critically depends on the choice of the tapering parameter. We develop a Stein’s Unbiased Risk Estimation (SURE) theory for estimating the Frobenius risk of the tapering estimator. SURE tuning selects the minimizer of SURE curve as the chosen tapering parameter. An extensive Monte Carlo study shows that SURE tuning is often comparable to the oracle tuning and outperforms cross-validation. We further illustrate SURE tuning using rock sonar spectrum data. The real data analysis results are consistent with simulation findings.

Minimax and Adaptive Prediction for Functional Linear Regression

This article considers minimax and adaptive prediction with functional predictors in the framework of functional linear model and reproducing kernel Hilbert space. Minimax rate of convergence for the excess prediction risk is established. It is shown that the optimal rate is determined jointly by the reproducing kernel and the covariance kernel. In particular, the alignment of these two kernels can significantly affect the difficulty of the prediction problem. In contrast, the existing literature has so far focused only on the setting where the two kernels are nearly perfectly aligned. This motivates us to propose an easily implementable data-driven roughness regularization predictor that is shown to attain the optimal rate of convergence adaptively without the need of knowing the covariance kernel. Simulation studies are carried out to illustrate the merits of the adaptive predictor and to demonstrate the theoretical results.

Variance reduction in purely random forests

Random forests (RFs), introduced by Leo Breiman in 2001, are a very effective statistical method. The complex mechanism of the method makes theoretical analysis difficult. Therefore, simplified versions of RF, called purely RFs (PRF), which can be theoretically handled more easily, have been considered. In this paper, we study the variance of such forests. First, we show a general upper bound which emphasises the fact that a forest reduces the variance. We then introduce a simple variant of PRFs, that we call purely uniformly RFs. For this variant and in the context of regression problems with a one-dimensional predictor space, we show that both random trees and RFs reach minimax rate of convergence. In addition, we prove that compared with random trees, RFs improve accuracy by reducing the estimator variance by a factor of three-fourths.

Adaptive covariance matrix estimation through block thresholding

Estimation of large covariance matrices has drawn considerable recent attention, and the theoretical focus so far has mainly been on developing a minimax theory over a fixed parameter space. In this paper, we consider adaptive covariance matrix estimation where the goal is to construct a single procedure which is minimax rate optimal simultaneously over each parameter space in a large collection. A fully data-driven block thresholding estimator is proposed. The estimator is constructed by carefully dividing the sample covariance matrix into blocks and then simultaneously estimating the entries in a block by thresholding. The estimator is shown to be optimally rate adaptive over a wide range of bandable covariance matrices. A simulation study is carried out and shows that the block thresholding estimator performs well numerically. Some of the technical tools developed in this paper can also be of independent interest.

Adaptive test on components of densities mixture

We are interested in testing whether two independent samples of n independent random variables are based on the same mixing-components or not. We provide a test procedure to detect if at least two mixing-components are distinct when their difference is a smooth function. Our test procedure is proved to be optimal according to the minimax adaptive setting that differs from the minimax setting by the fact that the smoothness is not known. Moreover, we show that the adaptive minimax rate suffers from a loss of order ($(\sqrt {\log (\log (n))} )^{ - 1}$ compared to the minimax rate.

Model selection and sharp asymptotic minimaxity

We obtain sharp minimax results for estimation of an n-dimensional normal mean under quadratic loss. The estimators are chosen by penalized least squares with a penalty that grows like ck log(n/k), for k equal to the number of nonzero elements in the estimating vector. For a wide range of sparse parameter spaces, we show that the penalized estimator achieves the exact minimax rate with the correct multiplication constant if and only if c equals 2. Our results unify the theory obtained by many other authors for penalized estimation of normal means. In particular we establish that a conjecture by Abramovich et al. (Ann Stat 34:584–653, 2006) is true.

Optimal rates of convergence for estimating Toeplitz covariance matrices

Toeplitz covariance matrices are used in the analysis of stationary stochastic processes and a wide range of applications including radar imaging, target detection, speech recognition, and communications systems. In this paper, we consider optimal estimation of large Toeplitz covariance matrices and establish the minimax rate of convergence for two commonly used parameter spaces under the spectral norm. The properties of the tapering and banding estimators are studied in detail and are used to obtain the minimax upper bound. The results also reveal a fundamental difference between the tapering and banding estimators over certain parameter spaces. The minimax lower bound is derived through a novel construction of a more informative experiment for which the minimax lower bound is obtained through an equivalent Gaussian scale model and through a careful selection of a finite collection of least favorable parameters. In addition, optimal rate of convergence for estimating the inverse of a Toeplitz covariance matrix is also established.

Suboptimality of nonlocal means for images with sharp edges

We conduct an asymptotic risk analysis of the nonlocal means image denoising algorithm for the Horizon class of images that are piecewise constant with a sharp edge discontinuity. We prove that the mean square risk of an optimally tuned nonlocal means algorithm decays according to n−1log1/2+ϵn, for an n×n-pixel image with ϵ>0. This decay rate is an improvement over some of the predecessors of this algorithm, including the linear convolution filter, median filter, and the SUSAN filter, each of which provides a rate of only n−2/3. It is also within a logarithmic factor from optimally tuned wavelet thresholding. However, it is still substantially lower than the optimal minimax rate of n−4/3.

Non asymptotic minimax rates of testing in signal detection with heterogeneous variances

The aim of this paper is to establish non-asymptotic minimax rates for goodness-of-fit hypotheses testing in an heteroscedastic setting. More precisely, we deal with sequences (Yj)j∈J of independent Gaussian random variables, having mean (θj)j∈J and variance (σj)j∈J. The set J will be either finite or countable. In particular, such a model covers the inverse problem setting where few results in test theory have been obtained. The rates of testing are obtained with respect to l2 norm, without assumption on (σj)j∈J and on several functions spaces. Our point of view is entirely non-asymptotic.