Minimax Rate Of Convergence Research Articles

A reproducing kernel Hilbert space approach to high dimensional partially varying coefficient model

Partially varying coefficient model (PVCM) provides a useful class of tools for modeling complex data by incorporating a combination of constant and time-varying covariate effects. One natural question is that how to decide which covariates correspond to constant coefficients and which correspond to time-dependent coefficient functions. To handle this two-type structure selection problem on PVCM, those existing methods are either based on a finite truncation way of coefficient functions, or based on a two-phase procedure to estimate the constant and function parts separately. This paper attempts to provide a complete theoretical characterization for estimation and structure selection issues of PVCM, via proposing two new penalized methods for PVCM within a reproducing kernel Hilbert space (RKHS). The proposed strategy is partially motivated by the so-called “Non-Constant Theorem” of radial kernels, which ensures a unique and unified representation of each candidate component in the hypothesis space. Within a high-dimensional framework, minimax convergence rates for the prediction risk of the first method is established when each unknown time-dependent coefficient can be well approximated within a specified RKHS. On the other hand, under certain regularity conditions, it is shown that the second proposed estimator is able to identify the underlying structure correctly with high probability. Several simulated experiments are implemented to examine the finite sample performance of the proposed methods.

Kernel regression, minimax rates and effective dimensionality: Beyond the regular case

We investigate if kernel regularization methods can achieve minimax convergence rates over a source condition regularity assumption for the target function. These questions have been considered in past literature, but only under specific assumptions about the decay, typically polynomial, of the spectrum of the the kernel mapping covariance operator. In the perspective of distribution-free results, we investigate this issue under much weaker assumption on the eigenvalue decay, allowing for more complex behavior that can reflect different structure of the data at different scales.

Optimal Estimation of Wasserstein Distance on a Tree With an Application to Microbiome Studies

The weighted UniFrac distance, a plug-in estimator of the Wasserstein distance of read counts on a tree, has been widely used to measure the microbial community difference in microbiome studies. Our investigation however shows that such a plug-in estimator, although intuitive and commonly used in practice, suffers from potential bias. Motivated by this finding, we study the problem of optimal estimation of the Wasserstein distance between two distributions on a tree from the sampled data in the high-dimensional setting. The minimax rate of convergence is established. To overcome the bias problem, we introduce a new estimator, referred to as the moment-screening estimator on a tree (MET), by using implicit best polynomial approximation that incorporates the tree structure. The new estimator is computationally efficient and is shown to be minimax rate-optimal. Numerical studies using both simulated and real biological datasets demonstrate the practical merits of MET, including reduced biases and statistically more significant differences in microbiome between the inactive Crohn’s disease patients and the normal controls. Supplementary materials for this article are available online.

On change-point estimation under Sobolev sparsity

In this paper, we consider the estimation of a change-point for possibly high-dimensional data in a Gaussian model, using a maximum likelihood method. We are interested in how dimension reduction can affect the performance of the method. We provide an estimator of the change-point that has a minimax rate of convergence, up to a logarithmic factor. The minimax rate is in fact composed of a fast rate —dimension-invariant— and a slow rate —increasing with the dimension. Moreover, it is proved that considering the case of sparse data, with a Sobolev regularity, there is a bound on the separation of the regimes above which there exists an optimal choice of dimension reduction, leading to the fast rate of estimation. We propose an adaptive dimension reduction procedure based on Lepski’s method and show that the resulting estimator attains the fast rate of convergence. Our results are then illustrated by a simulation study. In particular, practical strategies are suggested to perform dimension reduction.

Estimating piecewise monotone signals

We study the problem of estimating piecewise monotone vectors. This problem can be seen as a generalization of the isotonic regression that allows a small number of order-violating changepoints. We focus mainly on the performance of the nearly-isotonic regression proposed by Tibshirani et al. (2011). We derive risk bounds for the nearly-isotonic regression estimators that are adaptive to piecewise monotone signals. The estimator achieves a near minimax convergence rate over certain classes of piecewise monotone signals under a weak assumption. Furthermore, we present an algorithm that can be applied to the nearly-isotonic type estimators on general weighted graphs. The simulation results suggest that the nearly-isotonic regression performs as well as the ideal estimator that knows the true positions of changepoints.

Functional estimation and hypothesis testing in nonparametric boundary models

Consider a Poisson point process with unknown support boundary curve $g$, which forms a prototype of an irregular statistical model. We address the problem of estimating non-linear functionals of the form $\int\Phi(g(x))\,dx$. Following a nonparametric maximum-likelihood approach, we construct an estimator which is UMVU over Hölder balls and achieves the (local) minimax rate of convergence. These results hold under weak assumptions on $\Phi$ which are satisfied for $\Phi(u)=|u|^{p}$, $p\ge1$. As an application, we consider the problem of estimating the $L^{p}$-norm and derive the minimax separation rates in the corresponding nonparametric hypothesis testing problem. Structural differences to results for regular nonparametric models are discussed.

Doubly penalized estimation in additive regression with high-dimensional data

Additive regression provides an extension of linear regression by modeling the signal of a response as a sum of functions of covariates of relatively low complexity. We study penalized estimation in high-dimensional nonparametric additive regression where functional semi-norms are used to induce smoothness of component functions and the empirical $L_{2}$ norm is used to induce sparsity. The functional semi-norms can be of Sobolev or bounded variation types and are allowed to be different amongst individual component functions. We establish oracle inequalities for the predictive performance of such methods under three simple technical conditions: a sub-Gaussian condition on the noise, a compatibility condition on the design and the functional classes under consideration and an entropy condition on the functional classes. For random designs, the sample compatibility condition can be replaced by its population version under an additional condition to ensure suitable convergence of empirical norms. In homogeneous settings where the complexities of the component functions are of the same order, our results provide a spectrum of minimax convergence rates, from the so-called slow rate without requiring the compatibility condition to the fast rate under the hard sparsity or certain $L_{q}$ sparsity to allow many small components in the true regression function. These results significantly broaden and sharpen existing ones in the literature.

Sparse covariance matrix estimation in high-dimensional deconvolution

We study the estimation of the covariance matrix $\Sigma$ of a $p$-dimensional normal random vector based on $n$ independent observations corrupted by additive noise. Only a general nonparametric assumption is imposed on the distribution of the noise without any sparsity constraint on its covariance matrix. In this high-dimensional semiparametric deconvolution problem, we propose spectral thresholding estimators that are adaptive to the sparsity of $\Sigma$. We establish an oracle inequality for these estimators under model miss-specification and derive non-asymptotic minimax convergence rates that are shown to be logarithmic in $n/\log p$. We also discuss the estimation of low-rank matrices based on indirect observations as well as the generalization to elliptical distributions. The finite sample performance of the threshold estimators is illustrated in a numerical example.

Minimax-rate adaptive nonparametric regression with unknown correlations of errors

Minimax-rate adaptive nonparametric regression has been intensively studied under the assumption of independent or uncorrelated errors in the literature. In many applications, however, the errors are dependent, including both short- and long-range dependent situations. In such a case, adaptation with respect to the unknown dependence is important. We present a general result in this direction under Gaussian errors. It is assumed that the covariance matrix of the errors is known to be in a list of specifications possibly including independence, short-range dependence and long-range dependence as well. The regression function is known to be in a countable (or uncountable but well-structured) collection of function classes. Adaptive estimators are constructed to attain the minimax rate of convergence automatically for each function class under each correlation specification in the corresponding lists.

Bayesian sparse linear regression with unknown symmetric error

Abstract We study Bayesian procedures for sparse linear regression when the unknown error distribution is endowed with a non-parametric prior. Specifically, we put a symmetrized Dirichlet process mixture of Gaussian prior on the error density, where the mixing distributions are compactly supported. For the prior on regression coefficients, a mixture of point masses at zero and continuous distributions is considered. Under the assumption that the model is well specified, we study behavior of the posterior with diverging number of predictors. The compatibility and restricted eigenvalue conditions yield the minimax convergence rate of the regression coefficients in $\ell _1$- and $\ell _2$-norms, respectively. In addition, strong model selection consistency and a semi-parametric Bernstein–von Mises theorem are proven under slightly stronger conditions.

Manifold regularization based on Nyström type subsampling

In this paper, we study the Nystr{\"o}m type subsampling for large scale kernel methods to reduce the computational complexities of big data. We discuss the multi-penalty regularization scheme based on Nystr{\"o}m type subsampling which is motivated from well-studied manifold regularization schemes. We develop a theoretical analysis of multi-penalty least-square regularization scheme under the general source condition in vector-valued function setting, therefore the results can also be applied to multi-task learning problems. We achieve the optimal minimax convergence rates of multi-penalty regularization using the concept of effective dimension for the appropriate subsampling size. We discuss an aggregation approach based on linear function strategy to combine various Nystr{\"o}m approximants. Finally, we demonstrate the performance of multi-penalty regularization based on Nystr{\"o}m type subsampling on Caltech-101 data set for multi-class image classification and NSL-KDD benchmark data set for intrusion detection problem.

Robust adaptive efficient estimation for semi-Markov nonparametric regression models

We consider the nonparametric robust estimation problem for regression models in continuous time with semi-Markov noises. An adaptive model selection procedure is proposed. Under general moment conditions on the noise distribution a sharp non-asymptotic oracle inequality for the robust risks is obtained and the robust efficiency is shown. It turns out that for semi-Markov models the robust minimax convergence rate may be faster or slower than the classical one.

Optimal Penalized Function-on-Function Regression Under a Reproducing Kernel Hilbert Space Framework

ABSTRACTMany scientific studies collect data where the response and predictor variables are both functions of time, location, or some other covariate. Understanding the relationship between these functional variables is a common goal in these studies. Motivated from two real-life examples, we present in this article a function-on-function regression model that can be used to analyze such kind of functional data. Our estimator of the 2D coefficient function is the optimizer of a form of penalized least squares where the penalty enforces a certain level of smoothness on the estimator. Our first result is the representer theorem which states that the exact optimizer of the penalized least squares actually resides in a data-adaptive finite-dimensional subspace although the optimization problem is defined on a function space of infinite dimensions. This theorem then allows us an easy incorporation of the Gaussian quadrature into the optimization of the penalized least squares, which can be carried out through standard numerical procedures. We also show that our estimator achieves the minimax convergence rate in mean prediction under the framework of function-on-function regression. Extensive simulation studies demonstrate the numerical advantages of our method over the existing ones, where a sparse functional data extension is also introduced. The proposed method is then applied to our motivating examples of the benchmark Canadian weather data and a histone regulation study. Supplementary materials for this article are available online.

Robust estimation of high-dimensional covariance and precision matrices.

High-dimensional data are often most plausibly generated from distributions with complex structure and leptokurtosis in some or all components. Covariance and precision matrices provide a useful summary of such structure, yet the performance of popular matrix estimators typically hinges upon a sub-Gaussianity assumption. This paper presents robust matrix estimators whose performance is guaranteed for a much richer class of distributions. The proposed estimators, under a bounded fourth moment assumption, achieve the same minimax convergence rates as do existing methods under a sub-Gaussianity assumption. Consistency of the proposed estimators is also established under the weak assumption of bounded 2 + ε moments for ε ∈ (0, 2). The associated convergence rates depend on ε.

Prediction of weakly locally stationary processes by auto-regression

In this contribution we introduce weakly locally stationary time series through the local approximation of the non-stationary covariance structure by a stationary one. This allows us to define autoregression coefficients in a non-stationary context, which, in the particular case of a locally stationary Time Varying Autoregressive (TVAR) process, coincide with the generating coefficients. We provide and study an estimator of the time varying autoregression coefficients in a general setting. The proposed estimator of these coefficients enjoys an optimal minimax convergence rate under limited smoothness conditions. In a second step, using a bias reduction technique, we derive a minimax-rate estimator for arbitrarily smooth time-evolving coefficients, which outperforms the previous one for large data sets. In turn, for TVAR processes, the predictor derived from the estimator exhibits an optimal minimax prediction rate.

Quantile estimation for a hybrid model of functional and varying coefficient regressions

We consider a hybrid of functional and varying-coefficient regression models for the analysis of mixed functional data. We propose a quantile estimation of this hybrid model as an alternative to the least square approach. Under regularity conditions, we establish the asymptotic normality of the proposed estimator. We show that the estimated slope function can attain the minimax convergence rate as in functional linear regression. A Monte Carlo simulation study and a real data application suggest that the proposed estimation is promising.

Inferences in panel data with interactive effects using large covariance matrices

We consider efficient estimation of panel data models with interactive effects, which relies on a high-dimensional inverse covariance matrix estimator. By using a consistent estimator of the error covariance matrix, we can take into account both cross-sectional correlations and heteroskedasticity. In the presence of cross-sectional correlations, the proposed estimator eliminates the cross-sectional correlation bias, and is more efficient than the existing methods. The rate of convergence is also improved. In addition, we find that when the statistical inference involves estimating a high-dimensional inverse covariance matrix, the minimax convergence rate on large covariance estimations is not sufficient for inferences. To address this issue, a new “doubly weighted convergence” result is developed. The proposed method is applied to the US divorce rate data. We find that our more efficient estimator identifies the significant effects of divorce-law reforms on the divorce rate, and provides tighter confidence intervals than existing methods. This provides a confirmation for the empirical findings of Wolfers (2006) under more general unobserved heterogeneity.

Optimal classification and nonparametric regression for functional data

We establish minimax convergence rates for classification of functional data and for nonparametric regression with functional design variables. The optimal rates are of logarithmic type under smoothness constraints on the functional density and the regression mapping, respectively. These asymptotic properties are attainable by conventional kernel procedures. The bandwidth selector does not require knowledge of the smoothness level of the target mapping. In this work, the functional data are considered as realisations of random variables which take their values in a general Polish metric space. We impose certain metric entropy constraints on this space; but no algebraic properties are required.

Minimax convergence rates for kernel CCA

Consistency of kernel canonical correlation analysis (kernel CCA) has been established while its optimal convergence rate remains unknown. In this paper we derive rigorous upper and lower bounds for the convergence rate of the weight functions in kernel CCA. In particular the optimal convergence rate is shown to only depend on the rate of decay of the eigenvalues of the covariance operators.

Minimax Rate-optimal Estimation of High-dimensional Covariance Matrices with Incomplete Data.

Missing data occur frequently in a wide range of applications. In this paper, we consider estimation of high-dimensional covariance matrices in the presence of missing observations under a general missing completely at random model in the sense that the missingness is not dependent on the values of the data. Based on incomplete data, estimators for bandable and sparse covariance matrices are proposed and their theoretical and numerical properties are investigated. Minimax rates of convergence are established under the spectral norm loss and the proposed estimators are shown to be rate-optimal under mild regularity conditions. Simulation studies demonstrate that the estimators perform well numerically. The methods are also illustrated through an application to data from four ovarian cancer studies. The key technical tools developed in this paper are of independent interest and potentially useful for a range of related problems in high-dimensional statistical inference with missing data.