Minimax Rate Of Convergence Research Articles

Shapley Curves: A Smoothing Perspective

This article fills the limited statistical understanding of Shapley values as a variable importance measure from a nonparametric (or smoothing) perspective. We introduce population-level Shapley curves to measure the true variable importance, determined by the conditional expectation function and the distribution of covariates. Having defined the estimand, we derive minimax convergence rates and asymptotic normality under general conditions for the two leading estimation strategies. For finite sample inference, we propose a novel version of the wild bootstrap procedure tailored for capturing lower-order terms in the estimation of Shapley curves. Numerical studies confirm our theoretical findings, and an empirical application analyzes the determining factors of vehicle prices.

High-Dimensional Multivariate Linear Regression with Weighted Nuclear Norm Regularization

We consider a low-rank matrix estimation problem when the data is assumed to be generated from the multivariate linear regression model. To induce the low-rank coefficient matrix, we employ the weighted nuclear norm (WNN) penalty defined as the weighted sum of the singular values of the matrix. The weights are set in a nondecreasing order, which yields the non-convexity of the WNN objective function in the parameter space. Although the objective function has been widely applied, studies on the estimation properties of its resulting estimator are limited. We propose an efficient algorithm under the framework of the alternative directional method of multipliers (ADMM) to estimate the coefficient matrix. The estimator from the suggested algorithm converges to a stationary point of an augmented Lagrangian function. Under the orthogonal design setting, the effects of the weights for estimating the singular values of the ground-truth coefficient matrix are derived. Under the Gaussian design setting, a minimax convergence rate on the estimation error is derived. We also propose a generalized cross-validation (GCV) criterion for selecting the tuning parameter and an iterative algorithm for updating the weights. Simulations and a real data analysis demonstrate the competitive performance of our new method. Supplementary materials for this article are available online.

Optimal estimation of the rough Hurst parameter in additive noise

We estimate the Hurst parameter H∈(0,1) of a fractional Brownian motion from discrete noisy data, observed along a high-frequency sampling scheme. When the intensity τn of the noise is smaller in order than n−H we establish the LAN property with optimal rate n−1/2. Otherwise, we establish that the minimax rate of convergence is (n/τn2)−1/(4H+2) even when τn is of order 1. Our construction of an optimal procedure relies on a Whittle type construction possibly pre-averaged, together with techniques developed in Fukasawa et al. (2019). We establish in all cases a central limit theorem with explicit variance, extending the classical results of Gloter and Hoffmann (2007).

Fast Convergence on Perfect Classification for Functional Data

We investigate the availability of approaching perfect classification on functional data with finite samples. The seminal work (Delaigle and Hall (2012)) showed that perfect classification for functional data is easier to achieve than for finite-dimensional data. This result is based on their finding that a sufficient condition for the existence of a perfect classifier, named a Delaigle--Hall condition, is only available for functional data. However, there is a danger that a large sample size is required to achieve the perfect classification even though the Delaigle--Hall condition holds, because a minimax convergence rate of errors with functional data has a logarithm order in sample size. This study solves this complication by proving that the Delaigle--Hall condition also achieves fast convergence of the misclassification error in sample size, under the bounded entropy condition on functional data. We study a reproducing kernel Hilbert space-based classifier under the Delaigle--Hall condition, and show that a convergence rate of its misclassification error has an exponential order in sample size. Technically, our proof is based on (i) connecting the Delaigle--Hall condition and a margin of classifiers, and (ii) handling metric entropy of functional data. Our experiments support our result, and also illustrate that some other classifiers for functional data have a similar property.

Sparse multiple kernel learning: Minimax rates with random projection

In kernel-based learning, the random projection method, also called random sketching, has been successfully used in kernel ridge regression to reduce the computational burden in the big data setting, and at the same time retain the minimax convergence rate. In this work, we consider its use in sparse multiple kernel learning problems where a closed-form optimizer is not available, which poses significant technical challenges, for which the existing results do not carry over directly. Even when random projection is not used, our risk bound improves on the existing results in several aspects. We also illustrate the use of random projection via some numerical examples.

A spline-assisted semiparametric approach to nonparametric measurement error models

A spline-assisted approach is proposed to handle the measurement error problem in treating the pollution and asthma data. It is well known that the minimax rate of convergence in nonparametric regression function estimation of a random variable measured with error is much slower than the rate in the error free case. A different problem is considered. It is shown that if one is willing to impose a relatively mild assumption in requiring that the error-prone variable has a compact support, then standard nonparametric results are obtainable for measurement error models. New and constructive methods to take full advantage of the compact support assumption via spline-assisted semiparametric methods are proposed. It is proven that the new estimator achieves the usual nonparametric rate as if there were no measurement error. Furthermore it is shown that similar spline approach can be used to assess the probability density function on a compact set, using observations with error, and the resulting estimator differs from the true density function only by a constant scale. In addition, it retains the nonparametric density convergence properties of the error free case. The performance of the new methods is demonstrated through simulations and the methods are implemented to analyze the relation between asthma and pollution.

Minimax estimation of the mode of functional data

We give a definition of the mode of a probability distribution defined on an infinite-dimensional space and establish the minimax convergence rate. Therein, the existence of a probability density with respect to some abstract reference measure will not be required, while instead we use small ball probabilities to define the mode. The optimal rates of convergence are of logarithmic type under additional constraints on the small ball probabilities and entropy. We assume that the mode is contained in some totally bounded set and use covering methods to define a mode estimator and deduce strong consistency and rate optimality.

Minimax estimation of norms of a probability density: II. Rate-optimal estimation procedures

In this paper we develop rate–optimal estimation procedures in the problem of estimating the Lp–norm, p∈(1,∞) of a probability density from independent observations. The density is assumed to be defined on Rd, d≥1 and to belong to a ball in the anisotropic Nikolskii space. We adopt the minimax approach and construct rate–optimal estimators in the case of integer p≥2. We demonstrate that, depending on the parameters of the Nikolskii class and the norm index p, the minimax rates of convergence may vary from inconsistency to the parametric n–estimation. The results in this paper complement the minimax lower bounds derived in the companion paper (Goldenshluger and Lepski (2020)).

Consistency of Bayesian inference with Gaussian process priors for a parabolic inverse problem

We consider the statistical non-linear inverse problem of recovering the absorption term f > 0 in the heat equation where is a bounded domain, T < ∞ is a fixed time, and g, u 0 are given sufficiently smooth functions describing boundary and initial values respectively. The data consists of N discrete noisy point evaluations of the solution u f on . We study the statistical performance of Bayesian nonparametric procedures based on a large class of Gaussian process priors. We show that, as the number of measurements increases, the resulting posterior distributions concentrate around the true parameter generating the data, and derive a convergence rate for the reconstruction error of the associated posterior means. We also consider the optimality of the contraction rates and prove a lower bound for the minimax convergence rate for inferring f from the data, and show that optimal rates can be achieved with truncated Gaussian priors.

Function Approximation by Deep Neural Networks with Parameters {0,pm frac{1}{2}, pm 1, 2}

In this paper, it is shown that C_beta -smooth functions can be approximated by deep neural networks with ReLU activation function and with parameters {0,pm frac{1}{2}, pm 1, 2}. The l_0 and l_1 parameter norms of considered networks are thus equivalent. The depth, the width and the number of active parameters of the constructed networks have, up to a logarithmic factor, the same dependence on the approximation error as the networks with parameters in [-1,1]. In particular, this implies that the nonparametric regression estimation with constructed networks achieves, up to logarithmic factors, the same minimax convergence rates as with sparse networks with parameters in [-1,1].

Nonconvex Sparse Regularization for Deep Neural Networks and Its Optimality.

Recent theoretical studies proved that deep neural network (DNN) estimators obtained by minimizing empirical risk with a certain sparsity constraint can attain optimal convergence rates for regression and classification problems. However, the sparsity constraint requires knowing certain properties of the true model, which are not available in practice. Moreover, computation is difficult due to the discrete nature of the sparsity constraint. In this letter, we propose a novel penalized estimation method for sparse DNNs that resolves the problems existing in the sparsity constraint. We establish an oracle inequality for the excess risk of the proposed sparse-penalized DNN estimator and derive convergence rates for several learning tasks. In particular, we prove that the sparse-penalized estimator can adaptively attain minimax convergence rates for various nonparametric regression problems. For computation, we develop an efficient gradient-based optimization algorithm that guarantees the monotonic reduction of the objective function.

Total variation regularized Fréchet regression for metric-space valued data

Non-Euclidean data that are indexed with a scalar predictor such as time are increasingly encountered in data applications, while statistical methodology and theory for such random objects are not well developed yet. To address the need for new methodology in this area, we develop a total variation regularization technique for nonparametric Fréchet regression, which refers to a regression setting where a response residing in a metric space is paired with a scalar predictor and the target is a conditional Fréchet mean. Specifically, we seek to approximate an unknown metric-space valued function by an estimator that minimizes the Fréchet version of least squares and at the same time has small total variation, appropriately defined for metric-space valued objects. We show that the resulting estimator is representable by a piece-wise constant function and establish the minimax convergence rate of the proposed estimator for metric data objects that reside in Hadamard spaces. We illustrate the numerical performance of the proposed method for both simulated and real data, including metric spaces of symmetric positive-definite matrices with the affine-invariant distance, of probability distributions on the real line with the Wasserstein distance, and of phylogenetic trees with the Billera–Holmes–Vogtmann metric.

Minimax Rates of ℓp-Losses for High-Dimensional Linear Errors-in-Variables Models over ℓq-Balls.

In this paper, the high-dimensional linear regression model is considered, where the covariates are measured with additive noise. Different from most of the other methods, which are based on the assumption that the true covariates are fully obtained, results in this paper only require that the corrupted covariate matrix is observed. Then, by the application of information theory, the minimax rates of convergence for estimation are investigated in terms of the -losses under the general sparsity assumption on the underlying regression parameter and some regularity conditions on the observed covariate matrix. The established lower and upper bounds on minimax risks agree up to constant factors when , which together provide the information-theoretic limits of estimating a sparse vector in the high-dimensional linear errors-in-variables model. An estimator for the underlying parameter is also proposed and shown to be minimax optimal in the -loss.

Estimation of stopping times for stopped self-similar random processes

Let X=(X_t)_{tge 0} be a known process and T an unknown random time independent of X. Our goal is to derive the distribution of T based on an iid sample of X_T. Belomestny and Schoenmakers (Stoch Process Appl 126(7):2092–2122, 2015) propose a solution based the Mellin transform in case where X is a Brownian motion. Applying their technique we construct a non-parametric estimator for the density of T for a self-similar one-dimensional process X. We calculate the minimax convergence rate of our estimator in some examples with a particular focus on Bessel processes where we also show asymptotic normality.

ON THE CONVERGENCE RATE OF POTENTIALS OF BRENIER MAPS

The theory of optimal transportation has experienced a sharp increase in interest in many areas of economic research such as optimal matching theory and econometric identification. A particularly valuable tool, due to its convenient representation as the gradient of a convex function, has been the Brenier map: the matching obtained as the optimizer of the Monge–Kantorovich optimal transportation problem with the euclidean distance as the cost function. Despite its popularity, the statistical properties of the Brenier map have yet to be fully established, which impedes its practical use for estimation and inference. This article takes a first step in this direction by deriving a convergence rate for the simple plug-in estimator of the potential of the Brenier map via the semi-dual Monge–Kantorovich problem. Relying on classical results for the convergence of smoothed empirical processes, it is shown that this plug-in estimator converges in standard deviation to its population counterpart under the minimax rate of convergence of kernel density estimators if one of the probability measures satisfies the Poincaré inequality. Under a normalization of the potential, the result extends to convergence in the $L^2$ norm, while the Poincaré inequality is automatically satisfied. The main mathematical contribution of this article is an analysis of the second variation of the semi-dual Monge–Kantorovich problem, which is of independent interest.

Minimax bounds for Besov classes in density estimation

We study the problem of density estimation on [0,1] under Lp norm. We carry out a new piecewise polynomial estimator and prove that it is simultaneously (near)-minimax over a very wide range of Besov classes Bπ,∞α(R). In particular, we may deal with unbounded densities and shed light on the minimax rates of convergence when π<p and α∈(1∕π−1∕p,1∕π].

Nonparametric density estimation for intentionally corrupted functional data

We consider statistical models where functional data are artificially contaminated by independent Wiener processes in order to satisfy privacy constraints. We show that the corrupted observations have a Wiener density which determines the distribution of the original functional random variables, masked near the origin, uniquely, and we construct a nonparametric estimator of that density. We derive an upper bound for its mean integrated squared error which has a polynomial convergence rate, and we establish an asymptotic lower bound on the minimax convergence rates which is close to the rate attained by our estimator. Our estimator requires the choice of a basis and of two smoothing parameters. We propose data-driven ways of choosing them and prove that the asymptotic quality of our estimator is not significantly affected by the empirical parameter selection. We examine the numerical performance of our method via simulated examples.

Bayesian inference for high-dimensional decomposable graphs

In this paper, we consider high-dimensional Gaussian graphical models where the true underlying graph is decomposable. A hierarchical $G$-Wishart prior is proposed to conduct a Bayesian inference for the precision matrix and its graph structure. Although the posterior asymptotics using the $G$-Wishart prior has received increasing attention in recent years, most of results assume moderate high-dimensional settings, where the number of variables $p$ is smaller than the sample size $n$. However, this assumption might not hold in many real applications such as genomics, speech recognition and climatology. Motivated by this gap, we investigate asymptotic properties of posteriors under the high-dimensional setting where $p$ can be much larger than $n$. The pairwise Bayes factor consistency, posterior ratio consistency and graph selection consistency are obtained in this high-dimensional setting. Furthermore, the posterior convergence rate for precision matrices under the matrix $\ell_1$-norm is derived, which turns out to coincide with the minimax convergence rate for sparse precision matrices. A simulation study confirms that the proposed Bayesian procedure outperforms competitors.

Jump or kink: on super-efficiency in segmented linear regression breakpoint estimation

Summary We consider the problem of segmented linear regression with a single breakpoint, with the focus on estimating the location of the breakpoint. If $n$ is the sample size, we show that the global minimax convergence rate for this problem in terms of the mean absolute error is $O(n^{-1/3})$. On the other hand, we demonstrate the construction of a super-efficient estimator that achieves the pointwise convergence rate of either $O(n^{-1})$ or $O(n^{-1/2})$ for every fixed parameter value, depending on whether the structural change is a jump or a kink. The implications of this example and a potential remedy are discussed.

Approximate nonparametric quantile regression in reproducing kernel Hilbert spaces via random projection

Nonparametric quantile regression is a commonly used nonlinear quantile model. One general and popular approach is based on the use of kernels within a reproducing kernel Hilbert space (RKHS) framework, with the smoothing splines estimation as a special case. However, when the sample size n is large, the computational burden is heavy. Motivated by the recent advances in random projection for kernel nonparametric (mean) ridge regression (KRR), we consider an m-dimensional random projection approach for kernel quantile regression (KQR) with m≪n. We establish a theoretical result showing that the sketched KQR still achieves the minimax convergence rate when m is at least as large as the effective statistical dimension of the problem. Some Monte Carlo studies are carried out for illustration purposes.