Nonparametric Regression Estimation Research Articles

Nonparametric regression estimation for functional stationary ergodic data with missing at random

In this paper, we investigate the asymptotic properties of the estimator for the regression function operator whenever the functional stationary ergodic data with missing at random (MAR) are considered. Concretely, we construct the kernel type estimator of the regression operator for functional stationary ergodic data with the responses MAR, and some asymptotic properties such as the convergence rate in probability as well as the asymptotic normality of the estimator are obtained under some mild conditions respectively. As an application, the asymptotic (1−ζ) confidence interval of the regression operator is also presented for 0<ζ<1. Finally, a simulation study is carried out to compare the finite sample performance based on mean square error between the classical functional regression in complete case and the functional regression with MAR.

Methodology for Non-Parametric Deconvolution When the Error Distribution is Unknown

Summary In the non-parametric deconvolution problem, to estimate consistently a density or distribution from a sample of data contaminated by additive random noise, it is often assumed that the noise distribution is completely known or that an additional sample of replicated or validation data is available. Methods also have been suggested for estimating the scale of the error distribution, but they require somewhat restrictive smoothness assumptions on the signal distribution, which can be difficult to verify in practice. We take a completely new approach to the problem, not requiring extra data of any type. We argue that data rarely come from a simple regular distribution, and that this can be exploited to estimate the signal distributions by using a simple procedure. Our method can be extended to other problems involving errors in variables, such as non-parametric regression estimation. Its performance in practice is remarkably good, often equalling (even unexpectedly) the performance of techniques that use additional data to estimate the unknown error distribution.

Adaptive estimation in the functional nonparametric regression model

In this paper, we consider nonparametric regression estimation when the predictor is a functional random variable (typically a curve) and the response is scalar. Starting from a classical collection of kernel estimates, the bias–variance decomposition of a pointwise risk is investigated to understand what can be expected at best from adaptive estimation. We propose a fully data-driven local bandwidth selection rule in the spirit of the Goldenshluger and Lepski method. The main result is a nonasymptotic risk bound which shows the optimality of our tuned estimator from the oracle point of view. Convergence rates are also derived for regression functions belonging to Hölder spaces and under various assumptions on the rate of decay of the small ball probability of the explanatory variable. A simulation study also illustrates the good practical performances of our estimator.

Optimal difference-based variance estimation in heteroscedastic nonparametric regression

Estimating the residual variance is an important question in nonparametric regression. Among the existing estimators, the optimal difference-based variance estimation proposed in Hall, Kay and Titterington (1990) is widely used in practice. Their method is restricted to the situation when the errors are independent and identically distributed. In this paper, we propose the optimal difference-based variance estimation in heteroscedastic nonparametric regression under settings of uncorrelated errors and correlated errors. The proposed estimators are shown to be asymptotically unbiased and their mean squared errors are derived. Simulation studies indicate that the proposed estimators perform better than existing competitors in finite sample settings. In addition, data examples are analyzed to demonstrate the practical usefulness of the proposed method. The proposed method has many applications and we apply it to the nonparametric regression model with repeated measurements and the semiparametric partially linear model.

Robust estimating equation-based sufficient dimension reduction

In this paper, from the estimating equation-based sufficient dimension reduction method in the literature, its robust version is proposed to alleviate the impact from outliers. To achieve this, a robust nonparametric regression estimator is suggested. The estimator is plugged in the estimating equation of the semiparametric sufficient dimension reduction to obtain robust estimator for the central subspace. The asymptotic properties and robustness of the estimator are investigated. Numerical simulation and real data analysis are conducted to examine the performance of the estimators.

Additive kernel estimates of returns to schooling

In this paper, we employ a partially linear nonparametric additive regression estimator, with recent U.S. Current Population Survey data, to analyze returns to schooling. Similar to previous research, we find that blacks and Hispanics have higher rates of return on average. However, for married males, while non-Hispanic whites have lower returns on average, they typically possess the highest returns in the sample. For females, we are able to show that Hispanics have uniformly higher returns over non-Hispanic whites for the full sample. When we restrict our analysis to females whose highest level of education is a high school diploma, we find average, but no longer uniformly higher returns. However, these uniformly higher returns resurface for college graduates.

UNIFORM CONSISTENCY FOR NONPARAMETRIC ESTIMATORS IN NULL RECURRENT TIME SERIES

This paper establishes uniform consistency results for nonparametric kernel density and regression estimators when time series regressors concerned are nonstationary null recurrent Markov chains. Under suitable regularity conditions, we derive uniform convergence rates of the estimators. Our results can be viewed as a nonstationary extension of some well-known uniform consistency results for stationary time series.

Adaptive function estimation in nonparametric regression with one-sided errors

We consider the model of nonregular nonparametric regression where smoothness constraints are imposed on the regression function $f$ and the regression errors are assumed to decay with some sharpness level at their endpoints. The aim of this paper is to construct an adaptive estimator for the regression function $f$. In contrast to the standard model where local averaging is fruitful, the nonregular conditions require a substantial different treatment based on local extreme values. We study this model under the realistic setting in which both the smoothness degree $\beta>0$ and the sharpness degree $\mathfrak{a}\in(0,\infty)$ are unknown in advance. We construct adaptation procedures applying a nested version of Lepski’s method and the negative Hill estimator which show no loss in the convergence rates with respect to the general $L_{q}$-risk and a logarithmic loss with respect to the pointwise risk. Optimality of these rates is proved for $\mathfrak{a}\in(0,\infty)$. Some numerical simulations and an application to real data are provided.

ADAPTIVE NONPARAMETRIC REGRESSION WITH CONDITIONAL HETEROSKEDASTICITY

In this paper, we study adaptive nonparametric regression estimation in the presence of conditional heteroskedastic error terms. We demonstrate that both the conditional mean and conditional variance functions in a nonparametric regression model can be estimated adaptively based on the local profile likelihood principle. Both the one-step Newton–Raphson estimator and the local profile likelihood estimator are investigated. We show that the proposed estimators are asymptotically equivalent to the infeasible local likelihood estimators [e.g., Aerts and Claeskens (1997) Journal of the American Statistical Association 92, 1536–1545], which require knowledge of the error distribution. Simulation evidence suggests that when the distribution of the error term is different from Gaussian, the adaptive estimators of both conditional mean and variance can often achieve significant efficiency over the conventional local polynomial estimators.

THE INTEGRATED MEAN SQUARED ERROR OF SERIES REGRESSION AND A ROSENTHAL HILBERT-SPACE INEQUALITY

This paper develops uniform approximations for the integrated mean squared error (IMSE) of nonparametric series regression estimators, including both least-squares and averaging least-squares estimators. To develop these approximations, we also generalize an important probability inequality of Rosenthal (1970,Israel Journal of Mathematics8, 273–303; 1972,Sixth Berkeley Symposium on Mathematical Statistics and Probability, vol. 2, pp. 149–167. University of California Press) to the case of Hilbert-space valued random variables.

Block Thresholding on the Sphere

The aim of this paper is to study nonparametric regression estimators on the sphere based on needlet block thresholding. The block thresholding procedure proposed here follows the method introduced by Hall et al. (Ann. Stat. 26, 922–942, 1998, Stat. Sin. 9, 33–49, 1999), which we modify to exploit the properties of spherical needlets. We establish convergence rates, and we show that they attain adaptivity over Besov balls in the regular region. This work is strongly motivated by issues arising in Cosmology and Astrophysics, concerning in particular the analysis of Cosmic rays.

Semiparametric estimation of average treatment effect through a random coefficient dummy endogenous variable model

This paper provides an estimation procedure for average treatment effect through a random coefficient dummy endogenous variable model. A leading example of the model is estimating the effect of a training program on earnings. The model is composed of two equations: an outcome equation and a decision equation. Given the linear restriction in outcome and decision equations, Chen (1999) provided a distribution-free estimation procedure under conditional symmetric error distributions. In this paper we extend Chen’s estimator by relaxing the linear index into a nonparametric function, which greatly reduces the risk of model misspecification. A two-step approach is proposed: the first step uses a nonparametric regression estimator for the decision variable, and the second step uses an instrumental variables approach to estimate average treatment effect in the outcome equation. The proposed estimator is shown to be consistent and asymptotically normally distributed. Furthermore, we investigate the finite performance of our estimator by a Monte Carlo study and also use our estimator to study the return of college education in different periods of China. The estimates seem more reasonable than those of other commonly used estimators.

Empirical Comparison of Nonparametric Regression Estimates on Real Data

The performance of nine different nonparametric regression estimates is empirically compared on ten different real datasets. The number of data points in the real datasets varies between 7, 900 and 18, 000, where each real dataset contains between 5 and 20 variables. The nonparametric regression estimates include kernel, partitioning, nearest neighbor, additive spline, neural network, penalized smoothing splines, local linear kernel, regression trees, and random forests estimates. The main result is a table containing the empirical L2 risks of all nine nonparametric regression estimates on the evaluation part of the different datasets. The neural networks and random forests are the two estimates performing best. The datasets are publicly available, so that any new regression estimate can be easily compared with all nine estimates considered in this article by just applying it to the publicly available data and by computing its empirical L2 risks on the evaluation part of the datasets.

A plug-in the number of knots selector for polynomial spline regression

A plug-in the number of interior knots (NIKs) selector is proposed for polynomial spline estimation in nonparametric regression. The existence and properties of the optimal NIKs for spline regression are established by minimising the weighted mean integrated squared error. We obtain plug-in formulae for the optimal NIKs based on the theoretical results of asymptotic optimality, and develop strategies for choosing the NIKs of the spline estimator. The proposed NIKs selection method is tested on our simulated data with quite satisfactory performance, and is illustrated by analysing a fossil data set.

Generalized nonparametric smoothing with mixed discrete and continuous data

The nonparametric smoothing technique with mixed discrete and continuous regressors is considered. It is generally admitted that it is better to smooth the discrete variables, which is similar to the smoothing technique for continuous regressors but using discrete kernels. However, such an approach might lead to a potential problem which is linked to the bandwidth selection for the continuous regressors due to the presence of the discrete regressors. Through the numerical study, it is found that in many cases, the performance of the resulting nonparametric regression estimates may deteriorate if the discrete variables are smoothed in the way previously addressed, and that a fully separate estimation without any smoothing of the discrete variables may provide significantly better results both for bias and variance. As a solution, it is suggested a simple generalization of the nonparametric smoothing technique with both discrete and continuous data to address this problem and to provide estimates with more robust performance. The asymptotic theory for the new nonparametric smoothing method is developed and the finite sample behavior of the proposed generalized approach is studied through extensive Monte-Carlo experiments as well an empirical illustration.

Performance criteria and discrimination of extreme undersmoothing in nonparametric regression

The prediction error (average squared error) is the most commonly used performance criterion for the assessment of nonparametric regression estimators. However, there has been little investigation of the properties of the criterion itself. This paper shows that in certain situations the prediction error can be very misleading because it fails to discriminate an extreme undersmoothed estimate from a good estimate. For spline smoothing, we show, using asymptotic analysis and simulations, that there is poor discrimination of extreme undersmoothing in the following situations: small sample size or small error variance or a function with high curvature. To overcome this problem, we propose using the Sobolev error criterion. For spline smoothing, it is shown asymptotically and by simulations that the Sobolev error is significantly better than the prediction error in discriminating extreme undersmoothing. Similar results hold for other nonparametric regression estimators and for multivariate smoothing. For thin-plate smoothing splines, the prediction error׳s poor discrimination of extreme undersmoothing becomes significantly worse with increasing dimension.

Nonparametric Kernel Methods with Errors‐in‐Variables: Constructing Estimators, Computing them, and Avoiding Common Mistakes

SummaryEstimating a curve nonparametrically from data measured with error is a difficult problem that has been studied by many authors. Constructing a consistent estimator in this context can sometimes be quite challenging, and in this paper we review some of the tools that have been developed in the literature for kernel‐based approaches, founded on the Fourier transform and a more general unbiased score technique. We use those tools to rederive some of the existing nonparametric density and regression estimators for data contaminated by classical or Berkson errors, and discuss how to compute these estimators in practice. We also review some mistakes made by those working in the area, and highlight a number of problems with an existing R package decon.

EFFICIENT NON‐PARAMETRIC ESTIMATION OF THE SPECTRAL DENSITY IN THE PRESENCE OF MISSING OBSERVATIONS

The problem of non‐parametric spectral density estimation for discrete‐time series in the presence of missing observations has a long history. In particular, the first consistent estimators of the spectral density have been developed at about the same time as consistent estimators for non‐parametric regression. On the other hand, while for now, the theory of efficient (under the minimax mean integrated squared error criteria) and adaptive nonparametric regression estimation with missing data is well developed, no similar results have been proposed for the spectral density of a time series whose observations are missed according to an unknown stochastic process. This article develops the theory of efficient and adaptive estimation for a class of spectral densities that includes classical causal autoregressive moving‐average time series. The developed theory shows how a missing mechanism affects the estimation and what penalty it imposes on the risk convergence. In particular, given costs of a single observation in time series with and without missing data and a desired accuracy of estimation, the theory allows one to choose the cost‐effective time series. A numerical study confirms the asymptotic theory.

Diurnal patterns and associations among salivary cortisol, DHEA and alpha-amylase in older adults

BackgroundCortisol and dehydroepiandrosterone (DHEA) are considered to be valuable markers of the hypothalamus–pituitary–adrenal (HPA) axis, while salivary alpha-amylase (sAA) reflects the autonomic nervous system. Past studies have found certain diurnal patterns among these biomarkers, with some studies reporting results that differ from others. Also, some past studies have found an association among these three biomarkers while other studies have not. This study investigates these patterns and associations in older adults by taking advantage of modern statistical methods for dealing with non-normality, outliers and curvature. Basic characteristics of the data are reported as well, which are relevant to understanding the nature of any patterns and associations. MethodsBoxplots were used to check on the skewness and presence of outliers, including the impact of using simple transformations for dealing with non-normality. Diurnal patterns were investigated using recent advances aimed at comparing medians. When studying associations, the initial step was to check for curvature using a non-parametric regression estimator. Based on the resulting fit, a robust regression estimator was used that is designed to deal with skewed distributions and outliers. ResultsBoxplots indicated highly skewed distributions with outliers. Simple transformations (such as taking logs) did not deal with this issue in an effective manner. Consequently, diurnal patterns were investigated using medians and found to be consistent with some previous studies but not others. A positive association between awakening cortisol levels and DHEA was found when DHEA is relatively low; otherwise no association was found. The nature of the association between cortisol and DHEA was found to change during the course of the day. Upon awakening, cortisol was found to have no association with sAA when DHEA levels are relatively low, but otherwise there is a negative association. DHEA was found to have a positive association with sAA upon awakening. Shortly after awakening and for the remainder of the day, no association was found between DHEA and sAA ignoring cortisol. For DHEA and cortisol (taken as the independent variables) versus sAA (the dependent variable), again an association is found only upon awakening.

Probability estimation with machine learning methods for dichotomous and multicategory outcome: Theory

Probability estimation for binary and multicategory outcome using logistic and multinomial logistic regression has a long-standing tradition in biostatistics. However, biases may occur if the model is misspecified. In contrast, outcome probabilities for individuals can be estimated consistently with machine learning approaches, including k-nearest neighbors (k-NN), bagged nearest neighbors (b-NN), random forests (RF), and support vector machines (SVM). Because machine learning methods are rarely used by applied biostatisticians, the primary goal of this paper is to explain the concept of probability estimation with these methods and to summarize recent theoretical findings. Probability estimation in k-NN, b-NN, and RF can be embedded into the class of nonparametric regression learning machines; therefore, we start with the construction of nonparametric regression estimates and review results on consistency and rates of convergence. In SVMs, outcome probabilities for individuals are estimated consistently by repeatedly solving classification problems. For SVMs we review classification problem and then dichotomous probability estimation. Next we extend the algorithms for estimating probabilities using k-NN, b-NN, and RF to multicategory outcomes and discuss approaches for the multicategory probability estimation problem using SVM. In simulation studies for dichotomous and multicategory dependent variables we demonstrate the general validity of the machine learning methods and compare it with logistic regression. However, each method fails in at least one simulation scenario. We conclude with a discussion of the failures and give recommendations for selecting and tuning the methods. Applications to real data and example code are provided in a companion article (doi:10.1002/bimj.201300077).