Nonparametric Regression Estimation Research Articles

Deep regression learning with optimal loss function

In this paper, we develop a novel efficient and robust nonparametric regression estimator under a framework of a feedforward neural network (FNN). There are several interesting characteristics for the proposed estimator. First, the loss function is built upon an estimated maximum likelihood function, which integrates the information from observed data as well as the information from the data distribution. Consequently, the resulting estimator has desirable optimal properties, such as efficiency. Second, different from the traditional maximum likelihood estimation (MLE), the proposed method avoids the specification of the distribution, making it adaptable to various distributions such as heavy tails and multimodal or heterogeneous distributions. Third, the proposed loss function relies on probabilities rather than direct observations as in least square loss, contributing to the robustness of the proposed estimator. Finally, the proposed loss function involves a nonparametric regression function only. This enables the direct application of the existing packages, simplifying the computational and programming requirements. We establish the large sample property of the proposed estimator in terms of its excess risk and minimax near-optimal rate. The theoretical results demonstrate that the proposed estimator is equivalent to the true MLE where the density function is known in terms of excess risk. Our simulation studies show that the proposed estimator outperforms the existing methods based on prediction accuracy, efficiency and robustness. Particularly, it is comparable to the MLE with the known density and even gets slightly better as the sample size increases. This implies that the adaptive and data-driven loss function from the estimated density may offer an additional avenue for capturing valuable information. We further apply the proposed method to four real data examples, resulting in significantly reduced out-of-sample prediction errors compared to existing methods.

Nonparametric regression estimator of multivariable Fourier Series for categorical data

In recent years, Fourier Series estimators in nonparametric regression for quantitative data have received significant attention. However, in reality, there is often a relationship between response and predictor, where the response is categorical data. Some methods developed today to address the case of qualitative response data use only certain approaches. No Fourier Series estimator can handle categorical response data. This paper introduces a new method that uses response variable in the form of categorical data. This study aimed to develop a multivariable Fourier Series nonparametric regression estimator for categorical data. The research methods used are literature and theoretical studies. To apply this method, we used two application data. The results obtained indicate that the nonparametric regression of the Fourier Series provides significantly better estimation results and accuracy for both data applications, due to the small deviance value and larger AUC and Press'Q values. The highlights of this research are summarized below.•The Fourier Series method for categorical data assumes a relationship between the logit function and predictor variables that has a repeating pattern.•The estimator was obtained through Maximum Likelihood Estimation and Newton–Raphson method.•The Fourier Series nonparametric regression method provides better estimation than binary logistic regression.

Relative error prediction from censored data under α-mixing condition

In this paper, we address the case of a randomly right-censored model when the data exhibit some kind of dependency. We build and study a new nonparametric regression estimator by using the mean squared relative error as a loss function. Under classical conditions, we establish the uniform consistency with rate and asymptotic normality of the estimator suitably normalized.

Selection and Aggregation of Conformal Prediction Sets

Conformal prediction is a generic methodology for finite-sample valid distribution-free prediction. This technique has garnered a lot of attention in the literature partly because it can be applied with any machine learning algorithm that provides point predictions to yield valid prediction regions. Of course, the efficiency (width/volume) of the resulting prediction region depends on the performance of the machine learning algorithm. In the context of point prediction, several techniques (such as cross-validation) exist to select one of many machine learning algorithms for better performance. In contrast, such selection techniques are seldom discussed in the context of set prediction (or prediction regions). In this article, we consider the problem of obtaining the smallest conformal prediction region given a family of machine learning algorithms. We provide two general-purpose selection algorithms and consider coverage as well as width properties of the final prediction region. The first selection method yields the smallest width prediction region among the family of conformal prediction regions for all sample sizes but only has an approximate coverage guarantee. The second selection method has a finite sample coverage guarantee but only attains close to the smallest width. The approximate optimal width property of the second method is quantified via an oracle inequality. As an illustration, we consider the use of aggregation of nonparametric regression estimators in the split conformal method with the absolute residual conformal score. Supplementary materials for this article are available online, including a standardized description of the materials available for reproducing the work.

Difference-based covariance matrix estimation in time series nonparametric regression with application to specification tests

Summary Long-run covariance matrix estimation is the building block of time series inference. The corresponding difference-based estimator, which avoids detrending, has attracted considerable interest due to its robustness to both smooth and abrupt structural breaks and its competitive finite sample performance. However, existing methods mainly focus on estimators for the univariate process, while their direct and multivariate extensions for most linear models are asymptotically biased. We propose a novel difference-based and debiased long-run covariance matrix estimator for functional linear models with time-varying regression coefficients, allowing time series nonstationarity, long-range dependence, state heteroscedasticity and combinations thereof. We apply the new estimator to (i) the structural stability test, overcoming the notorious nonmonotonic power phenomena caused by piecewise smooth alternatives for regression coefficients, and (ii) the nonparametric residual-based tests for long memory, improving the performance via the residual-free formula of the proposed estimator. The effectiveness of the proposed method is justified theoretically and demonstrated by superior performance in simulation studies, while its usefulness is elaborated via real data analysis. Our method is implemented in the R package mlrv.

Linear and nonlinear signal detection and estimation in high-dimensional nonparametric regression under weak sparsity

Linear and nonlinear signal detection and estimation in high-dimensional nonparametric regression under weak sparsity

Application of the evolutionary approach to structural and parametric identification of dynamic objects

The paper examines the evolutionary approach to structural and parametric identification of dynamic systems in the form of differential equations. The approach is based on a genetic programming algorithm to determine a structure of the equation and differential evolution method for parameters selection. The author proposed approach based on such evolutionary algorithms as genetic programming and differential evolution. The search for the structure is carried out by genetic programming. The selection of numerical parameters and initial conditions is implemented by a method of differential evolution. The problem of finding a model that describes changes in the efficiency of a hydraulic system is solved with the help of this approach. The proposed approach is compared with a recurrent neural network and a nonparametric kernel regression estimation.

Wavelet estimation for the nonparametric additive model in random design and long-memory dependent errors

We investigate the nonparametric additive regression estimation in random design and long-memory errors and construct adaptive thresholding estimators based on wavelet series. The proposed approach achieves asymptotically near-optimal convergence rates when the unknown function and its univariate additive components belong to Besov space. We consider the problem under two noise structures; (1) homoskedastic Gaussian long memory errors and (2) heteroskedastic Gaussian long memory errors. In the homoskedastic long-memory error case, the estimator is completely adaptive with respect to the long-memory parameter. In the heteroskedastic long-memory case, the estimator may not be adaptive with respect to the long-memory parameter unless the heteroskedasticity is of polynomial form. In either case, the convergence rates depend on the long-memory parameter only when long-memory is strong enough, otherwise, the rates are identical to those under i.i.d. errors. In addition, convergence rates are free from the curse of dimensionality.

Regression Estimation with Errors in the Variables via the Laplace Transform

This paper considers nonparametric regression estimation with errors in the variables. It is a standard assumption that the characteristic function of the covariate error does not vanish on the real line. This assumption is rather strong. In this paper, we assume the covariate error distribution is a convolution of uniform distributions, the characteristic function of which contains zeros on the real line. Our regression estimator is constructed via the Laplace transform. We prove its strong consistency and show its convergence rate. It turns out that zeros in the characteristic function have no effect on the convergence rate of our estimator.

Selective machine learning of doubly robust functionals

Summary While model selection is a well-studied topic in parametric and nonparametric regression or density estimation, selection of possibly high-dimensional nuisance parameters in semiparametric problems is far less developed. In this paper, we propose a selective machine learning framework for making inferences about a finite-dimensional functional defined on a semiparametric model, when the latter admits a doubly robust estimating function and several candidate machine learning algorithms are available for estimating the nuisance parameters. We introduce a new selection criterion aimed at bias reduction in estimating the functional of interest based on a novel definition of pseudo risk inspired by the double robustness property. Intuitively, the proposed criterion selects a pair of learners with the smallest pseudo risk, so that the estimated functional is least sensitive to perturbations of a nuisance parameter. We establish an oracle property for a multi-fold cross-validation version of the new selection criterion that states that our empirical criterion performs nearly as well as an oracle with a priori knowledge of the pseudo risk for each pair of candidate learners. Finally, we apply the approach to model selection of a semiparametric estimator of average treatment effect given an ensemble of candidate machine learners to account for confounding in an observational study that we illustrate in simulations and a data application.

Impact analysis of correlated and non-normal errors in nonparametric regression estimation: A simulation study

In nonparametric regression, the correlation of errors can have important consequences for the statistical properties of the estimators, but the focus is on the on the identification of the effect on Average Mean Squared Error (AMSE). This is performed by a Monte Carlo experiment where we use two types of correlation structures and examine them with different correlation points/levels and different error distributions with different sample sizes. We concluded that if errors are correlated, then the distribution of errors is important with correlation structures, but correlation points/levels have a less significant effect, comparatively. When errors are uniformly distributed, AMSE is the smallest, followed by any other distribution, and if errors follow the Laplace distribution, then AMSE is the largest, followed by other distributions. Laplace also has some alarming effects. More specifically, the kernel estimator is robust in the case of a simple correlation structure, and AMSEs attain their minimum when errors are uncorrelated.

On Modelling Sea State Bias of Jason-2 Altimeter Data Based on Significant Wave Heights and Wind Speeds

Altimeter data processing is very important to improve the quality of sea surface height (SSH) measurements. Sea state bias (SSB) correction is a relatively uncertain error correction due to the lack of a clear theoretical model. At present, the commonly used methods for SSB correction are polynomial models (parametric models) and non-parametric models. The non-parametric model usually was constructed by collinear data. However, the amount of collinear data was enormous, and it contained redundant information. In this study, the non-parametric regression estimation model was optimized by using the parameter replacement method of ascending and descending tracks based on the crossover data. In this method, significant wave heights from the Jason-2 altimeter data during cycles 200–301 and wind speed from the ERA5 reanalysis data were used. The non-parametric regression estimation model of Jason-2 was constructed by combining it with local linear regression, Epanechnikov kernel function and local window width. At the same time, based on the significant wave height and wind speed at the crossover points, the SSB polynomial model containing six parameters was constructed by using the Taylor series expansion, and the model was optimized. By comparing polynomial model construction with different parameters, the optimized model was obtained. The SSH of the crossover points and the tide gauge records were used to validate these results derived from two models and GDR. Compared with the crossover discrepancies of SSH corrected by the polynomial model, the RMS of the crossover discrepancies of SSH corrected by the non-parametric regression estimation model was reduced by 7.9%. Compared with the crossover discrepancies of SSH corrected by the conventional non-parametric model from GDR, the RMS of the crossover discrepancies of SSH corrected by the non-parametric regression estimation model was reduced by 4.1%. This shows that the precision of the SSHs derived by after the SSB correction, as calculated by the non-parametric regression estimation model, was better than that of the polynomial model and the SSB correction from GDR. Using the Jason-2 altimeter data, the along-track geoid gradient and the sea level change rate of the global ocean were determined by using two models to correct the SSB. By comparing the results of the two models, the accuracy of the geoid gradient along the orbit that was obtained by the non-parametric regression estimation model was better than that of the polynomial model and GDR. The global average sea level change rate after the non-parametric regression estimation model correction was 3.47 ± 0.09 mm/y, which was the closest to the average sea level change rate that has been published in the international literature within this field.

Robust nonparametric regression: Review and practical considerations

Nonparametric regression models offer a way to understand and quantify relationships between variables without having to identify an appropriate family of possible regression functions. Although many estimation methods for these models have been proposed in the literature, most of them can be highly sensitive to the presence of a small proportion of atypical observations in the training set. A review of outlier robust estimation methods for nonparametric regression models is provided, paying particular attention to practical considerations. Since outliers can also influence negatively the regression estimator by affecting the selection of bandwidths or smoothing parameters, a discussion of robust alternatives for this task is also included. Using many of the “classical” nonparametric regression estimators (and their robust counterparts) can be very challenging in settings with a moderate or large number of explanatory variables, so recently proposed robust nonparametric regression methods that scale well with a growing number of covariates are also discussed.

KNN robustification equivariant nonparametric regression estimators for functional ergodic data

We discuss in this paper the robust equivariant nonparametric regression estimators for ergodic data with the k Nearst Neighbour (kNN) method. We consider a new robust regression estimator when the scale parameter is unknown. The principal aim is to prove the almost complete convergence (with rate) for the proposed estimator. Furthermore, a comparison study based on simulated data is also provided to illustrate the finite sample performances and the usefulness of the kNN approach and to prove the highly sensitive of the kNN approach to the presence of even a small proportion of outliers in the data.

ON THE ACCURACY BE THE METRIC OF THE DENSITY ESTIMATION CONSTRUCTED BY DEPENDENT OBSERVATIONS

ON THE ACCURACY BE THE METRIC OF THE DENSITY ESTIMATION CONSTRUCTED BY DEPENDENT OBSERVATIONS

Nonparametric multiple regression by projection on non-compactly supported bases

We study the nonparametric regression estimation problem with a random design in $${\mathbb{R}}^{p}$$ with $$p\ge 2$$ . We do so by using a projection estimator obtained by least squares minimization. Our contribution is to consider non-compact estimation domains in $${\mathbb {R}}^{p}$$ , on which we recover the function, and to provide a theoretical study of the risk of the estimator relative to a norm weighted by the distribution of the design. We propose a model selection procedure in which the model collection is random and takes into account the discrepancy between the empirical norm and the norm associated with the distribution of design. We prove that the resulting estimator automatically optimizes the bias-variance trade-off in both norms, and we illustrate the numerical performance of our procedure on simulated data.

Pemodelan Produk Domestik Regional Bruto di Indonesia dengan Regresi Nonparametrik Menggunakan Estimator Spline

The implementation of PPKM results in a decrease in community activities, hampers mobility and ultimately disrupts overall economic activity. So it is necessary to have a study that can explain the extent of economic development in Indonesia due to COVID-19 with a model. The modeling used in this research is nonparametric regression modeling with spline estimator. The purpose of this study is to determine the descriptive statistics of GRDP data in all provinces in Indonesia as an indicator of the rate of economic growth in Indonesia and to model the GRDP in Indonesia using nonparametric regression estimator spline. The results obtained in this study are the best nonparametric spline regression model with three knot points with a GCV value of 5199284230 and an R2 of 98.61 percent.

Weak consistency for the nonparametric kernel regression estimator based on negatively associated random errors

In this article, we mainly study the weak consistency for the Gasser and Müller (G-M) kernel estimator (a weighted kernel estimator with integral form) in a nonparametric regression model based on negatively associated random errors under some suitable conditions. In addition, a simulation to study the numerical performance of the consistency for the G-M estimator is provided.

Nonparametric Regression Modeling Based on Spline Truncated Estimator on Simulation Data

Regression analysis is one of the statistical analysis used to estimate the pattern of the relationship between predictor variables and response variables . In general, the approach to estimating the regression function is the parametric regression, the nonparametric regression and the semiparametric regression. The approach with parametric regression is used if the shape of the regression curve is assumed to follow a certain pattern such as linear, quadratic, cubic and so on, but in fact there is an unknown pattern of relationship between predictor variables and response variables, so nonparametric regression is used. Then the combination of parametric and nonparametric regression is semiparametric regression. One of the well-known nonparametric regression estimators is the spline truncated. This study was conducted by simulating the relationship pattern of the response variable and the predictor variable that not have specific pattern by following a trigonometric function that formed a regression curve with a standard deviation of 0,05 and 0,25 which formed a different distribution of data, then will be approached with parametric regression (linear, quadratic, cubic) and nonparametric regression (spline truncated linear). Based on the coefficient of determination of each standard deviation, it will shows that the nonparametric regression approach has high flexibility so that it is able to adjust the form of regression curve estimation by itself

Non-parametric Multivariate Kernel Regression Estimation to Describe Cognitive Processes and Mental Representations

In this research paper, we set forward a non-parametric multivariate recursive kernel regression estimator under missing data using the propensity score approach in order to describe writing word production. Our main objective is to explore cognitive processes and mental representations mobilized when a human being prepares to write a word according to the idea developed in Perret and Olive (2019). We investigate the asymptotic properties of the proposed recursive estimator and compare them to the well known Nadaraya-Watson’s regression estimator. We calculate the bias and the variance of the proposed estimator which depend on the choice of some parameters such as the stepsize and the bandwidth. We examine some data-driven procedures to select these parameters. Thus, we demonstrate that, under some optimal choices of these parameters, the MSE (Mean Squared Error) of the proposed estimator can be smaller than the one obtained by using Nadaraya Watson’s regression estimator. The elaborated estimator is then applied to the behavioral data to classify some participants in groups. This classification may stand for a departure point to tackle written behavior variations.