Nonparametric Regression Estimation Research Articles

Optimal rates of convergence for nonparametric regression estimation under anisotropic Hölder condition

In the multidimensional setting, we consider the nonparametric regression estimation with errors-in-variables. Both ordinary smooth noise and super smooth one are assumed for errors in the covariates. An anisotropic kernel estimator is provided based on a deconvolution technique. We study the pointwise estimation and obtain the optimal rates of convergence under the anisotropic Hölder condition.

Estimation of a Function of Low Local Dimensionality by Deep Neural Networks

Deep neural networks (DNNs) achieve impressive results for complicated tasks like object detection on images and speech recognition. Motivated by this practical success, there is now a strong interest in showing good theoretical properties of DNNs. To describe for which tasks DNNs perform well and when they fail, it is a key challenge to understand their performance. The aim of this paper is to contribute to the current statistical theory of DNNs. We apply DNNs on high dimensional data and we show that the least squares regression estimates using DNNs are able to achieve dimensionality reduction in case that the regression function has locally low dimensionality. Consequently, the rate of convergence of the estimate does not depend on its input dimension <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$d$ </tex-math></inline-formula> , but on its local dimension <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$d^{*}$ </tex-math></inline-formula> and the DNNs are able to circumvent the curse of dimensionality in case that <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$d^{*}$ </tex-math></inline-formula> is much smaller than <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$d$ </tex-math></inline-formula> . In our simulation study we provide numerical experiments to support our theoretical result and we compare our estimate with other conventional nonparametric regression estimates. The performance of our estimates is also validated in experiments with real data.

Uniform consistency and uniform in number of neighbors consistency for nonparametric regression estimates and conditional U-statistics involving functional data

Uniform consistency and uniform in number of neighbors consistency for nonparametric regression estimates and conditional U-statistics involving functional data

A critical review of univariate non-parametric estimation of first derivatives

This paper gives new guidance for selection of univariate non-parametric derivative estimators with non-iid errors in finite samples. It is shown via an extensive set of Monte Carlo simulations that the generalized criterion of Charnigo et al. (A generalized criterion for derivative estimation. Technometrics. 2011;53:238–253.) with spline smoothing performs the best in situations with minimal noise. For increased noise, generalized cross-validation of Craven and Wahba (Smoothing noisy data with spline functions. Numer Math. 1978;31:377–403) with P spline smoothing and the improved Akaike Information Criterion of Hurvich et al. (Smoothing parameter selection in nonparametric regression using an improved Akaike information criterion. J R Stat Soc Ser B. 1998;60:271–293.) with P spline smoothing are preferred. In the class of kernel smoothing and local regression methods, the local-cubic estimator of Henderson et al. (Gradient-based smoothing parameter selection for nonparametric regression estimation. J Econom. 2015;184:233–241.) generally outperforms its competitors. An internal meta-analysis separately favours the generalized method and P spline smoothing. The empirical example given provides support for use of the local-cubic estimator.

Uniform convergence rates for nonparametric estimators smoothed by the beta kernel

AbstractThis paper provides a set of uniform consistency results with rates for nonparametric density and regression estimators smoothed by the beta kernel having support on the unit interval. Weak and strong uniform convergence is explored on the basis of expanding compact sets and general sequences of smoothing parameters. The results in this paper are useful for asymptotic analysis of two‐step semiparametric estimation using a first‐step kernel estimate as a plug‐in. We provide simulations and a real data example illustrating attractive properties of the estimators.

Function Approximation by Deep Neural Networks with Parameters {0,pm frac{1}{2}, pm 1, 2}

In this paper, it is shown that C_beta -smooth functions can be approximated by deep neural networks with ReLU activation function and with parameters {0,pm frac{1}{2}, pm 1, 2}. The l_0 and l_1 parameter norms of considered networks are thus equivalent. The depth, the width and the number of active parameters of the constructed networks have, up to a logarithmic factor, the same dependence on the approximation error as the networks with parameters in [-1,1]. In particular, this implies that the nonparametric regression estimation with constructed networks achieves, up to logarithmic factors, the same minimax convergence rates as with sparse networks with parameters in [-1,1].

Potential Trade Vulnerability of Bangladesh: An Exploratory Analysis

Bangladesh's integration in the global economy has hitherto proven to be an important force behind its economic growth. However, exclusively trade dependent growth is also trade vulnerable growth. This paper explores the factors that may potentially affect the trade vulnerability of Bangladesh. Results of the bivariate non-parametric local linear kernel regression estimation reveal that the minimum wage of readymade garment workers, manufacturing cost, utilities cost, crude oil prices, and climate change are positively related to trade volatility, while improved political stability is negatively related to trade volatility. Multivariate kernel regressions with composite indicators show that macroeconomic factors, such as exchange rate, foreign exchange reserves and budget balance, have a statistically significant impact on the trade volatility of Bangladesh. In view of these preliminary results, a number of policy recommendations are suggested to deal with the short run and long run trade vulnerability of Bangladesh.

Nonparametric regression in nonstandard spaces

A nonparametric regression setting is considered with a real-valued covariate and responses from a metric space. One may approach this setting via Fréchet regression, where the value of the regression function at each point is estimated via a Fréchet mean calculated from an estimated objective function. A second approach is geodesic regression, which builds upon fitting geodesics to observations by a least squares method. These approaches are applied to transform two of the most important nonparametric regression estimators in statistics to the metric setting – the local linear regression estimator and the orthogonal series projection estimator. The resulting procedures consist of known estimators as well as new methods. We investigate their rates of convergence in a general setting and compare their performance in a simulation study on the sphere.

Robust Gaussian process regression with a bias model

This paper presents a new approach to a robust Gaussian process regression, creating a non-parametric Bayesian regression estimate robust to outliers. Most existing approaches replace an outlier-prone Gaussian likelihood with a non-Gaussian likelihood induced from a heavy tail distribution, such as the Laplace distribution and Student-t distribution. However, the use of a non-Gaussian likelihood would incur the need for a computationally expensive Bayesian approximate computation in the posterior inferences. The proposed approach models an outlier as a noisy and biased observation of an unknown regression function, and accordingly, the likelihood contains bias terms to explain the degree of deviations from the regression function. We introduce two bias models that handle the bias terms differently, treating a bias as an unknown and fixed quantity or treating a bias as a random quantity. We entail how the biases can be estimated accurately with other hyperparameters by a regularized maximum likelihood estimation. Conditioned on the bias estimates, the robust GP regression can be reduced to a standard GP regression problem with analytical forms of the predictive mean and variance estimates. Therefore, the proposed approach is simple and very computationally attractive. It also gives a very robust and accurate GP estimate for many tested scenarios. For the numerical evaluation, we perform a comprehensive simulation study to evaluate the proposed approach with the comparison to the existing robust GP approaches under various simulated scenarios of different outlier proportions and different noise levels. The approach is applied to data from two measurement systems, where the predictors are based on robust environmental parameter measurements and the response variables utilize more complex chemical sensing methods that contain a certain percentage of outliers. The utility of the measurement systems and value of the environmental data are improved through the computationally efficient GP regression and bias model.

Interval Censored Recursive Forests

We propose interval censored recursive forests (ICRF), an iterative tree ensemble method for interval censored survival data. This nonparametric regression estimator addresses the splitting bias problem of existing tree-based methods and iteratively updates survival estimates in a self-consistent manner. Consistent splitting rules are developed for interval censored data, convergence is monitored using out-of-bag samples, and kernel-smoothing is applied. The ICRF is uniformly consistent and displays high prediction accuracy in both simulations and applications to avalanche and national mortality data. An R package icrf is available on CRAN. Supplementary files for this article are available online.

Two-time-scale nonparametric recursive regression estimator for independent functional data

In this paper, we propose and investigate a new kernel regression estimators based on the two-time-scale stochastic approximation algorithm in the case of independent functional data. We study the properties of the proposed recursive estimators and compare them with the recursive estimators based on single-time-scale stochastic algorithm proposed by Slaoui and to the non-recursive estimator proposed by Slaoui. It turns out that, with an adequate choice of the parameters, the proposed two-time-scale estimators perform better than the recursive estimators constructed using single-time-scale stochastic algorithm. We corroborate these theoretical results through some simulations and two real datasets.

Nonparametric Pointwise Estimation for a Regression Model with Multiplicative Noise

In this paper, we consider a general nonparametric regression estimation model with the feature of having multiplicative noise. We propose a linear estimator and nonlinear estimator by wavelet method. The convergence rates of those regression estimators under pointwise error over Besov spaces are proved. It turns out that the obtained convergence rates are consistent with the optimal convergence rate of pointwise nonparametric functional estimation.

Bootstrap based goodness-of-fit tests for binary multivariate regression models

We consider a binary multivariate regression model where the conditional expectation of a binary variable given a higher-dimensional input variable belongs to a parametric family. Based on this, we introduce a model-based bootstrap (MBB) for higher-dimensional input variables. This test can be used to check whether a sequence of independent and identically distributed observations belongs to such a parametric family. The approach is based on the empirical residual process introduced by Stute (Ann Statist 25:613–641, 1997). In contrast to Stute and Zhu’s approach (2002) Stute & Zhu (Scandinavian J Statist 29:535–545, 2002), a transformation is not required. Thus, any problems associated with non-parametric regression estimation are avoided. As a result, the MBB method is much easier for users to implement. To illustrate the power of the MBB based tests, a small simulation study is performed. Compared to the approach of Stute & Zhu (Scandinavian J Statist 29:535–545, 2002), the simulations indicate a slightly improved power of the MBB based method. Finally, both methods are applied to a real data set.

A Heteroscedastic Method for Comparing Regression Lines at Specified Design Points When Using a Robust Regression Estimator

It is well known that the ordinary least squares (OLS) regression estimator is not robust. Many robust regression estimators have been proposed and inferential methods based on these estimators have been derived. However, for two independent groups, let θj (X) be some conditional measure of location for the jth group, given X, based on some robust regression estimator. An issue that has not been addressed is computing a 1 − confidence interval for θ1(X) − θ2(X) in a manner that allows both within group and between group hetereoscedasticity. The paper reports the finite sample properties of a simple method for accomplishing this goal. Simulations indicate that, in terms of controlling the probability of a Type I error, the method performs very well for a wide range of situations, even with a relatively small sample size. In principle, any robust regression estimator can be used. The simulations are focused primarily on the Theil-Sen estimator, but some results using Yohai’s MM-estimator, as well as the Koenker and Bas sett quantile regression estimator, are noted. Data from the Well Elderly II study, dealing with measures of meaningful activity using the cortisol awakening response as a covariate, are used to illustrate that the choice between an extant method based on a nonparametric regression estimator, and the method suggested here, can make a practical difference.

Unpaid housework and super-exploitation of labor: a suggested model and empirical evidence from Mexico and Colombia

The objective of this paper is to suggest a mathematical model of unpaid housework and empirically test its main predictions using data from Mexico (2014) and Colombia (2017) time-use surveys. The model, based upon the Marxist-feminist approach, suggests that the magnitude of unpaid housework is determined by the super-exploitation of labor, i.e. the gap between wages and the value of labor-power. The outcome is an equation that relates positively the magnitude of unpaid housework with the super-exploitation of labor. The parametric and nonparametric regression estimates applied in both countries show preliminary empirical support for the theoretical model. The theoretical and empirical findings have several implications for Marxist-feminist literature as well as for empirical research on unpaid housework.

An Analysis about Fourier Series Estimator in Nonparametric Regression for Longitudinal Data

Fourier series is a function that is often used Mathematically and Statistically especially for modeling. Here, Fourier series can be constructed as an estimator in nonparametric regression. Nonparametric regression is not only using cross section data, but also longitudinal data. Some of nonparametric regression estimators have been developed for longitudinal data case, such as kernel, and spline. In this study, we concentrate to develop an inference analysis that related to Fourier series estimator in nonparametric regression for longitudinal data. Nonparametric regression based on Fourier series is capable to model data relationship with fluctuation or oscillation pattern that represents with sine and cosine functions. For point estimation analysis, Penalized Weighted Least Square (PWLS) is used to determine an estimator for parameter vector in nonparametric regression. Different with previous studies, PWLS is used to get smooth estimator. The result is an estimator for nonparametric regression curve for longitudinal data based on Fourier series approach. In addition, this study also investigated the asymptotic properties of the nonparametric regression curve estimators using the Fourier series approach for longitudinal data, especially linearity and consistency. Some study cases based on previous research and a new study case is given to make sure that Fourier series estimator in nonparametric regression has good performance in longitudinal data modeling. This study is important in order to develop further inferences Statistics, such as interval estimation and test hypothesis that related nonparametric regression with Fourier series estimator for longitudinal data.

An Analysis of the Areas Occupied by Vessels in the Ocular Surface of Diabetic Patients: An Application of a Nonparametric Tilted Additive Model.

(1) Background: As diabetes melllitus (DM) can affect the microvasculature, this study evaluates different clinical parameters and the vascular density of ocular surface microvasculature in diabetic patients. (2) Methods: In this cross-sectional study, red-free conjunctival photographs of diabetic individuals aged 30–60 were taken under defined conditions and analyzed using a Radon transform-based algorithm for vascular segmentation. The Areas Occupied by Vessels (AOV) images of different diameters were calculated. To establish the sum of AOV of different sized vessels. We adopt a novel approach to investigate the association between clinical characteristics as the predictors and AOV as the outcome, that is Tilted Additive Model (TAM). We use a tilted nonparametric regression estimator to estimate the nonlinear effect of predictors on the outcome in the additive setting for the first time. (3) Results: The results show Age (p-value = 0.019) and Mean Arterial Pressure (MAP) have a significant linear effect on AOV (p-value = 0.034). We also find a nonlinear association between Body Mass Index (BMI), daily Urinary Protein Excretion (UPE), Hemoglobin A1C, and Blood Urea Nitrogen (BUN) with AOV. (4) Conclusions: As many predictors do not have a linear relationship with the outcome, we conclude that the TAM will help better elucidate the effect of the different predictors. The highest level of AOV can be seen at Hemoglobin A1C of 9% and AOV increases when the daily UPE exceeds 600 mg. These effects need to be considered in future studies of ocular surface vessels of diabetic patients.

Extrapolation-based tuning parameters selection in massive data analysis

Many statistical modeling procedures involve one or more tuning parameters tocontrol the model complexity. These tuning parameters can be the bandwidth in thekernel smoothing method in the nonparametric regression and density estimation orbe the regularization parameter in the regularization method for feature selectionin the high dimensional modeling. Tuning parameter selection plays critical rolesin the statistical modeling and machine learning. For the massive data analysis,commonly-used methods such as grid-point search with information criteriabecome prohibitively costly in computation. Their feasibility isquestionable even with modern parallel computing platforms.This paper aims to develop a fast algorithm to efficientlyapproximate the best tuning parameters. The algorithm entails (a) assuming aparametric model to describe the trend between the best tuning parameters andsample sizes, (b) establishing the trend via fitting the model with subsamplingdata, and (c) extrapolating this trend to the case of huge sample size. Todetermine the subsampling sample sizes to be taken, we derive optimaldesigns for settings that allow a constraint on the budget of total computational cost.We show that the proposed designs possess an asymptotic optimality 性质.Our numerical studies demonstrate that with a simple two-parameter polynomial model, the proposed algorithm performsalmost equivalently to the procedure using the full data setin several different statistical settings, while ithas a significant reduction in computing time and storage.

Bias reduction by projection on parametric models in Hilbertian nonparametric regression

In this paper we discuss a general way of improving the bias properties of nonparametric kernel regression estimators. The procedure involves choosing a parametric model and constructing a semiparametric estimator that consists of a parametric component and a nonparametric adjustment, where the parametric component is picked from the parametric model in such a way that the resulting estimator has the best bias performance. We study the method for response variables taking values in a general Hilbert space and for local linear smoother. We show that the procedure always improves the bias of the local linear estimator regardless of the choice of parametric model. We also illustrate the method via a real data example where the response variable is a random density.

Estimation of the bandwidth parameter in Nadaraya-Watson kernel non-parametric regression based on universal threshold level

This paper proposes a new improvement of the Nadaraya-Watson kernel non-parametric regression estimator and the bandwidth of this new improvement is obtained depending on universal threshold level with wavelet of kernel function instead of using fixed bandwidth and variable bandwidth for geometric, arithmetic mean, range and median measurements. A simulation study is presented, including comparisons between the proposed method and five others Nadaraya-Watson kernel estimators (classical methods), as well as using real data depending on a program written in MATLAB language which was designed for this purpose. It was concluded that the proposed method is more accurate than all classical methods for all simulations and real data based on MSE criterion.