Nonparametric Regression Problem Research Articles

Orthogonal systems based stable estimator for non parametric regression problems

. In this work, we study a univariate as well as a multivariate Jacobi polynomial-based estimator for solving the fundamental problem of non parametric (NP) regression. The proposed NP regression estimator aims to approximate a possibly very low regular regression function from the knowledge of its noised values given at some n random sampling points. We first study the univariate version of the proposed estimator, then extend this study to the general multivariate case with random sampling vectors drawn according to an unknown distribution. Our proposed estimator has the advantage of being stable with reasonable time complexity. Moreover, it has an optimal convergence rate under the weak condition that the regression function has a certain low Sobolev smoothness property. Although the proposed estimators are studied under the hypothesis that the random sampling vectors are supported on the d dimensional unit hypercube, by applying trivial transformations, the results of this work are still valid in the more general setting of any d dimensional hyperrectangle. Finally, the theoretical results of this work are illustrated by some numerical simulations on synthetic as well as real datasets.

Phase transitions in nonparametric regressions

When the unknown regression function of a single variable is known to have derivatives up to the (γ+1)th order bounded in absolute values by a common constant everywhere or a.e. (i.e., (γ+1)th degree of smoothness), the minimax optimal rate of the mean integrated squared error (MISE) is stated as 1n2γ+22γ+3 in the literature. This paper shows that: (i) if n≤γ+12γ+3, the minimax optimal MISE rate is lognnlog(logn) and the optimal degree of smoothness to exploit is roughly maxlogn2loglogn,1; (ii) if n>γ+12γ+3, the minimax optimal MISE rate is 1n2γ+22γ+3 and the optimal degree of smoothness to exploit is γ+1.The fundamental contribution of this paper is a set of metric entropy bounds we develop for smooth function classes. Some of our bounds are original, and some of them improve and/or generalize the ones in the literature (e.g., (Kolmogorov and Tikhomirov, 1959)). Our metric entropy bounds allow us to show phase transitions in the minimax optimal MISE rates associated with some commonly seen smoothness classes as well as non-standard smoothness classes, and can also be of independent interest outside the nonparametric regression problems.

A fuzzy nonparametric regression model based on an extended center and range method

When dealing with nonparametric regression problems, kernel-based methods are often employed. In this paper, we investigate nonparametric regression problems with fuzzy responses and exact predictors by implementing an extended center and range method within a three-stage procedure. For this purpose, we utilize the Nadaraya–Watson estimator for estimating the unknown fuzzy smooth function. In each stage, the unknown bandwidth is determined with the help of a generalized cross validation criterion. To evaluate and to compare the performance of our proposed fuzzy nonparametric regression model with established regression models, different goodness-of-fit criteria are considered. Finally, we investigate the practical applicability and show the superiority of the proposed regression model by means of a simulation study and real data applications.

Wavelet shrinkage in nonparametric regression models with positive noise

ABSTRACT Wavelet shrinkage estimators are widely applied to analyse datasets in wavelet domain from several fields of science. They typically act by reducing magnitudes of empirical coefficients in a discrete wavelet transformation to estimate wavelet coefficients. In nonparametric regression problems, most of the shrinkage rules are derived from models composed of an unknown function with additive Gaussian noise. Although Gaussian noise assumption is reasonable in several real data analysis, mainly for large sample sizes, it is not general. Contaminated data with positive noise can occur in practice and nonparametric regression models with positive noise bring challenges in wavelet shrinkage point of view. In this sense, this work proposes bayesian shrinkage rules to estimate wavelet coefficients from a nonparametric regression framework with additive and strictly positive noise under exponential and lognormal distributions. Computational aspects are discussed and simulation studies to analyse the performances of the proposed shrinkage rules and compare them with standard techniques are done. An application for winning times Boston Marathon dataset is also provided.

A nonparametric instrumental approach to confounding in competing risks models.

This paper discusses nonparametric identification and estimation of the causal effect of a treatment in the presence of confounding, competing risks and random right-censoring. Our identification strategy is based on an instrumental variable. We show that the competing risks model generates a nonparametric quantile instrumental regression problem. Quantile treatment effects on the subdistribution function can be recovered from the regression function. A distinguishing feature of the model is that censoring and competing risks prevent identification at some quantiles. We characterize the set of quantiles for which exact identification is possible and give partial identification results for other quantiles. We outline an estimation procedure and discuss its properties. The finite sample performance of the estimator is evaluated through simulations. We apply the proposed method to the Health Insurance Plan of Greater New York experiment.

Bayesian Trend Filtering via Proximal Markov Chain Monte Carlo

Proximal Markov chain Monte Carlo is a novel construct that lies at the intersection of Bayesian computation and convex optimization, which helped popularize the use of nondifferentiable priors in Bayesian statistics. Existing formulations of proximal MCMC, however, require hyperparameters and regularization parameters to be prespecified. In this article, we extend the paradigm of proximal MCMC through introducing a novel new class of nondifferentiable priors called epigraph priors. As a proof of concept, we place trend filtering, which was originally a nonparametric regression problem, in a parametric setting to provide a posterior median fit along with credible intervals as measures of uncertainty. The key idea is to replace the nonsmooth term in the posterior density with its Moreau-Yosida envelope, which enables the application of the gradient-based MCMC sampler Hamiltonian Monte Carlo. The proposed method identifies the appropriate amount of smoothing in a data-driven way, thereby automating regularization parameter selection. Compared with conventional proximal MCMC methods, our method is mostly tuning free, achieving simultaneous calibration of the mean, scale and regularization parameters in a fully Bayesian framework. Supplementary materials for this article are available online.

Posterior contraction and testing for multivariate isotonic regression

We consider the nonparametric regression problem with multiple predictors and an additive error, where the regression function is assumed to be coordinatewise nondecreasing. We propose a Bayesian approach to make inferences on the multivariate monotone regression function, obtain the posterior contraction rate, and construct a universally consistent Bayesian testing procedure for multivariate monotonicity. To facilitate posterior analysis, we temporarily set aside the shape restrictions, and endow a prior on blockwise constant regression functions with independently normally distributed heights. The unknown variance of the error term is either estimated by the marginal maximum likelihood estimate, or is equipped with an inverse-gamma prior. Then the unrestricted block-heights are a posteriori also independently normally distributed given the error variance, by conjugacy. To comply with the shape restrictions, we project samples from the unrestricted posterior onto the class of multivariate monotone functions, inducing the “projection-posterior distribution”, to be used for making an inference. Under an L1-metric, we show that the projection-posterior based on n independent samples contracts around the true monotone regression function at the optimal rate n−1∕(2+d). Then we construct a Bayesian test for multivariate monotonicity based on the posterior probability of a shrinking neighborhood of the class of multivariate monotone functions. We show that the test is universally consistent, that is, the level of the Bayesian test goes to zero, and the power at any fixed alternative goes to one. Moreover, we show that for a smooth alternative function, power goes to one as long as its distance to the class of multivariate monotone functions is at least of the order of the estimation error for a smooth function. To the best of our knowledge, no other test for multivariate monotonicity is available in the Bayesian or the frequentist literature.

Minimax estimation in multi-task regression under low-rank structures

This study investigates the minimaxity of a multi-task nonparametric regression problem. We formulate a simultaneous function estimation problem based on information pooling across multiple experiments under a low-dimensional structure. A nonparametric reduced rank regression estimator based on the nuclear norm penalisation scheme is proposed to incorporate the low-dimensional structure in the estimation process. A rank of a set of functions is defined in terms of their Fourier coefficients to formally characterise the dependence structure among functions. Minimax upper and lower bounds are established under various asymptotic scenarios to examine the role of the low-rank structure in determining optimal rates of convergence. The results confirm that exploiting the low-rank structure can significantly improve the convergence rate for the simultaneous estimation of multiple functions. The results also imply that the proposed estimator is rate optimal in the minimax sense for the rank-constraint Sobolev class of vector-valued functions.

A scalable Matérn Gaussian process for learning spatial curves distributions

Data points on Riemannian manifolds are fundamental objects in many applications and fields. Representations include shapes from biology and medical imaging, directions and rotations from robots. This paper addresses the problem of nonparametric regression on shapes when only few observations are available. In particular, we consider the problem of classifying unobserved 3D open parametrized curves using a continuous stochastic process to overcome the discrete nature of observations. The proposed method has a practical objective, characterizing populations of cochlear curves. The numerical solution is geometrically simpler, extensible and can be generalized for other applications. We illustrate and discuss the successful behavior of the proposed approach with various and multiple experimental results.

Improvement on LASSO-type estimator in nonparametric regression

ABSTRACT This paper is concerned with nonparametric regression with multi-dimensional, possibly high-dimensional, explanatory variables. A new -penalisation approach is proposed on the basis of the idea of slicing off waste restriction. The new approach can implement the variable selection and estimate the regression function simultaneously. The approach also develops the consistency of the variable selection and the asymptotic theory of regression estimator, thus highlighting the advantages of the proposed penalisation approach. Simulation results show that the proposed approaches for variable selection work efficiently, and the associated regression estimators perform well. Applications to some real data sets also reveal that the proposed methods yield reasonable and stable solutions for nonparametric regression problems.

The Temporal Overfitting Problem with Applications in Wind Power Curve Modeling

This article is concerned with a nonparametric regression problem in which the input variables and the errors are autocorrelated in time. The motivation for the research stems from modeling wind power curves. Using existing model selection methods, like cross-validation, results in model overfitting in presence of temporal autocorrelation. This phenomenon is referred to as temporal overfitting, which causes loss of performance while predicting responses for a time domain different from the training time domain. We propose a Gaussian process (GP)-based method to tackle the temporal overfitting problem. Our model is partitioned into two parts—a time-invariant component and a time-varying component, each of which is modeled through a GP. We modify the inference method to a thinning-based strategy, an idea borrowed from Markov chain Monte Carlo sampling, to overcome temporal overfitting and estimate the time-invariant component. We extensively compare our proposed method with both existing power curve models and available ideas for handling temporal overfitting on real wind turbine datasets. Our approach yields significant improvement when predicting response for a time period different from the training time period.

Nonconvex Sparse Regularization for Deep Neural Networks and Its Optimality.

Recent theoretical studies proved that deep neural network (DNN) estimators obtained by minimizing empirical risk with a certain sparsity constraint can attain optimal convergence rates for regression and classification problems. However, the sparsity constraint requires knowing certain properties of the true model, which are not available in practice. Moreover, computation is difficult due to the discrete nature of the sparsity constraint. In this letter, we propose a novel penalized estimation method for sparse DNNs that resolves the problems existing in the sparsity constraint. We establish an oracle inequality for the excess risk of the proposed sparse-penalized DNN estimator and derive convergence rates for several learning tasks. In particular, we prove that the sparse-penalized estimator can adaptively attain minimax convergence rates for various nonparametric regression problems. For computation, we develop an efficient gradient-based optimization algorithm that guarantees the monotonic reduction of the objective function.

Optimal learning with anisotropic Gaussian SVMs

This paper investigates the nonparametric regression problem using SVMs with anisotropic Gaussian RBF kernels. Under the assumption that the target functions are resided in certain anisotropic Besov spaces, we establish the almost optimal learning rates, more precisely, optimal up to some logarithmic factor, presented by the effective smoothness. By taking the effective smoothness into consideration, our almost optimal learning rates are faster than those obtained with the underlying RKHSs being certain anisotropic Sobolev spaces. Moreover, if the target function depends only on fewer dimensions, faster learning rates can be further achieved.

A unified framework for closed-form nonparametric regression, classification, preference and mixed problems with Skew Gaussian Processes

Skew-Gaussian Processes (SkewGPs) extend the multivariate Unified Skew-Normal distributions over finite dimensional vectors to distribution over functions. SkewGPs are more general and flexible than Gaussian processes, as SkewGPs may also represent asymmetric distributions. In a recent contribution, we showed that SkewGP and probit likelihood are conjugate, which allows us to compute the exact posterior for non-parametric binary classification and preference learning. In this paper, we generalize previous results and we prove that SkewGP is conjugate with both the normal and affine probit likelihood, and more in general, with their product. This allows us to (i) handle classification, preference, numeric and ordinal regression, and mixed problems in a unified framework; (ii) derive closed-form expression for the corresponding posterior distributions. We show empirically that the proposed framework based on SkewGP provides better performance than Gaussian processes in active learning and Bayesian (constrained) optimization. These two tasks are fundamental for design of experiments and in Data Science.

Statistical inference in sparse high-dimensional additive models

In this paper, we discuss the estimation of a nonparametric component f1 of a nonparametric additive model Y=f1(X1)+⋯+fq(Xq)+ϵ. We allow the number q of additive components to grow to infinity and we make sparsity assumptions about the number of nonzero additive components. We compare this estimation problem with that of estimating f1 in the oracle model Z=f1(X1)+ϵ, for which the additive components f2,…,fq are known. We construct a two-step presmoothing-and-resmoothing estimator of f1 and state finite-sample bounds for the difference between our estimator and a corresponding smoothing estimator fˆ1(oracle) in the oracle model. In an asymptotic setting, these bounds can be used to show asymptotic equivalence of our estimator and the oracle estimator; the paper thus shows that, asymptotically, under strong enough sparsity conditions, knowledge of f2,…,fq has no effect on estimation accuracy. Our first step is to estimate f1 with an undersmoothed estimator based on near-orthogonal projections with a group Lasso bias correction. In the second step, we construct pseudo responses Yˆ by evaluating this undersmoothed estimator of f1 at the design points and then apply the smoothing method of the oracle estimator fˆ1(oracle) to the nonparametric regression problem with “responses” Yˆ and covariates X1. Our mathematical exposition centers primarily on establishing properties of the presmoothing estimator. We present simulation results demonstrating close-to-oracle performance of our estimator in practical applications.

Log-Linear Bayesian Additive Regression Trees for Multinomial Logistic and Count Regression Models

We introduce Bayesian additive regression trees (BART) for log-linear models including multinomial logistic regression and count regression with zero-inflation and overdispersion. BART has been applied to nonparametric mean regression and binary classification problems in a range of settings. However, existing applications of BART have been mostly limited to models for Gaussian “data,” either observed or latent. This is primarily because efficient MCMC algorithms are available for Gaussian likelihoods. But while many useful models are naturally cast in terms of latent Gaussian variables, many others are not—including models considered in this article. We develop new data augmentation strategies and carefully specified prior distributions for these new models. Like the original BART prior, the new prior distributions are carefully constructed and calibrated to be flexible while guarding against overfitting. Together the new priors and data augmentation schemes allow us to implement an efficient MCMC sampler outside the context of Gaussian models. The utility of these new methods is illustrated with examples and an application to a previously published dataset. Supplementary materials for this article are available online.

Multivariate extensions of isotonic regression and total variation denoising via entire monotonicity and Hardy–Krause variation

We consider the problem of nonparametric regression when the covariate is d dimensional, where d≥1. In this paper, we introduce and study two nonparametric least squares estimators (LSEs) in this setting—the entirely monotonic LSE and the constrained Hardy–Krause variation LSE. We show that these two LSEs are natural generalizations of univariate isotonic regression and univariate total variation denoising, respectively, to multiple dimensions. We discuss the characterization and computation of these two LSEs obtained from n data points. We provide a detailed study of their risk properties under the squared error loss and fixed uniform lattice design. We show that the finite sample risk of these LSEs is always bounded from above by n−2/3 modulo logarithmic factors depending on d; thus these nonparametric LSEs avoid the curse of dimensionality to some extent. We also prove nearly matching minimax lower bounds. Further, we illustrate that these LSEs are particularly useful in fitting rectangular piecewise constant functions. Specifically, we show that the risk of the entirely monotonic LSE is almost parametric (at most 1/n up to logarithmic factors) when the true function is well approximable by a rectangular piecewise constant entirely monotone function with not too many constant pieces. A similar result is also shown to hold for the constrained Hardy–Krause variation LSE for a simple subclass of rectangular piecewise constant functions. We believe that the proposed LSEs yield a novel approach to estimating multivariate functions using convex optimization that avoid the curse of dimensionality to some extent.

ENCOLHIMENTO BAYESIANO DE COEFICIENTES DE ONDALETAS SOB PRIORI DE CHAMPERNOWNE COM APLICAÇÕES

Métodos estatísticos bayesianos para encolhimento de coeficientes de ondaletas (wavelet shrinkage) têm sido amplamente utilizados em diversas áreas para redução de ruídos em dados. Neste trabalho, propomos uma mistura da função delta de Dirac com a distribuição de Champernowne como distribuição probabilística a priori para os coeficientes das ondaletas em um problema de regressão não paramétrica. A regra bayesiana de encolhimento associada possui parâmetros de fácil interpretação e seu desempenho em estudos de simulação foi superior aos métodos disponíveis na literatura utilizados para comparação na maioria dos cenários considerados. Aplicações do método em dados de potenciais de ação neuronais e do índice da bolsa de valores de São Paulo (IBOVESPA) são feitas.

Strongly universally consistent nonparametric regression and classification with privatised data

In this paper we revisit the classical problem of nonparametric regression, but impose local differential privacy constraints. Under such constraints, the raw data (X1,Y1),...,(Xn,Yn), taking values in Rd×R, cannot be directly observed, and all estimators are functions of the randomised output from a suitable privacy mechanism. The statistician is free to choose the form of the privacy mechanism, and here we add Laplace distributed noise to a discretisation of the location of a feature vector Xi and to the value of its response variable Yi. Based on this randomised data, we design a novel estimator of the regression function, which can be viewed as a privatised version of the well-studied partitioning regression estimator. The main result is that the estimator is strongly universally consistent, and we further establish an upper bound on the rate of convergence. Our methods and analysis also give rise to a strongly universally consistent binary classification rule for locally differentially private data.

Posterior contraction and credible regions for level sets

For a given function f on a multivariate domain, the level sets, given by {x:f(x)=c} for different values of c, provide important geometrical insights about the underlying function of interest. The distance on level sets of two functions may be measured by the Hausdorff metric or a metric based on the Lebesgue measure of a discrepancy, both of which can be linked with the L∞-distance on the underlying functions. In a Bayesian framework, we derive posterior contraction rates and optimal sized credible sets with assured frequentist coverage for level sets in some nonparametric settings by extending some univariate L∞-posterior contraction rates to the corresponding multivariate settings. For the multivariate Gaussian white noise model, adaptive Hausdorff and Lebesgue contraction rates for levels sets of the signal function and its mixed order partial derivatives are derived using a wavelet series prior on the function. Assuming a known smoothness level of the signal function, an optimal sized credible region for a level set with assured frequentist coverage is derived based on a multidimensional trigonometric series prior. For the nonparametric regression problem, adaptive rates for level sets of the function and its mixed partial derivatives are obtained using a multidimensional wavelet series prior. When the smoothness level is given, optimal sized credible regions with assured frequentist coverage are obtained using a finite random series prior based on tensor products of B-splines. We also derive Hausdorff and Lebesgue contraction rates of a multivariate density function under a known smoothness setting.