Nadaraya-Watson Regression Research Articles

Medical Data for the Nadaraya-Watson Kernel Method in Nonparametric Regression Models Based on Hyperbolic Secant Function

Nonparametric kernel estimates used in this work aim to compare different treatment options by examining the recorded medical data. The following use of the suggested strategy depends on a kernel function and a parameter called bandwidth. The Nadaraya-Watson kernel (NWK) estimation is a necessary nonparametric kernel estimator used in regression models. A new Nadaraya-Watson regression estimate depends on the hyperbolic secant kernel (HSK) with fixed bandwidth (FNW) and Variable Bandwidth (VNW) is proposed.. We calculated some properties of the unknown regression function estimator, including bias, variance, optimal bandwidth, and a global measure of error criterion mean square error. Finally, simulation and three real data sets are used to evaluate its performance. Results from simulation and real data showed that the VNW using HSK is more effective than the FNW based on Average Mean Square Error Criterion. Also, Nadaraya-Watson using HSK function is more effective than Nadaraya-Watson using the Gaussian kernel density function.

Weak convergence of the conditional single index $ U $-statistics for locally stationary functional time series

<abstract><p>In recent years, there has been a notable shift in focus towards the analysis of non-stationary time series, driven largely by the complexities associated with delineating significant asymptotic behaviors inherent to such processes. The genesis of the theory of locally stationary processes arises from the quest for asymptotic inference grounded in nonparametric statistics. This paper endeavors to formulate a comprehensive framework for conducting inference within the realm of locally stationary functional time series by harnessing the conditional $ U $-statistics methodology as propounded by W. Stute in 1991. The proposed methodology extends the Nadaraya-Watson regression function estimations. Within this context, a novel estimator was introduced for the single index conditional $ U $-statistics operator, adept at accommodating the non-stationary attributes inherent to the data-generating process. The primary objective of this paper was to establish the weak convergence of conditional $ U $-processes within the domain of locally stationary functional mixing data. Specifically, the investigation delved into scenarios of weak convergence involving functional explanatory variables, considering both bounded and unbounded sets of functions while adhering to specific moment requirements. The derived findings emanate from broad structural specifications applicable to the class of functions and models under scrutiny. The theoretical insights expounded in this study constitute pivotal tools for advancing the domain of functional data analysis.</p></abstract>

Adaptive Nadaraya-Watson kernel regression estimators utilizing some non-traditional and robust measures: a numerical application of British food data

In nonparametric regression research, estimation of regression function is a prime concern. Recently, researchers developed some modified Nadaraya-Watson (N-W) regression estimators utilizing robust mean, median and harmonic mean. In this paper, we propose to utilize the additive combination of non-traditional measures i.e. (Hodges-Lehmann, Mid-Range, Tri-Mean, Quartile-Deviation) with the robust minimum covariance determinant (MCD) scale estimator in (N-W) regression estimator. Utilizing these measures, we get some new versions of (N-W) regression estimator. We also attempted to derive the properties of the proposed versions, such as bias, variance, mean square error (MSE), and mean integrated square error (MISE). The proposed estimators are compared with some of the existing estimators available in literature through a simulation study, utilizing two artificial populations. We also incorporated real-life application by taking British food data set denoted as Engel95, and assess the predictive ability of nonparametric regression, based on proposed and existing N-W estimators.

Predicting student performance using machine learning tools

Digital assistants are increasingly penetrating various areas of human activity, including education. Today, they are no longer just automated systems or web applications that support and automate certain processes, including educational processes. Now they are more intelligent and more autonomous systems. Digital assistants play a special role in a student’s life, in a sense replacing the dean’s office, mentor, tutor, representatives of other university services and other elements of educational infrastructure. The digital support for the student is important and useful, especially in the first year during his adaptation to the environment of higher education, which is significantly different from the school one. It is at this point that the largest amount of students dropouts occurs due to academic failure. According to the authors, a digital assistant in the form of a mobile application that can predict learning outcomes and inform about it in time, can provide important support for the student and help him/her orient and adjust his/her behavior in case of a threat of a negative result. To solve the problems of creating a predictive model of student learning outcomes and a mobile application that implements it, as well as to conduct a pre-project study, the following methods and tools of mathematical statistics were used: k-means method, Kendall correlation method, Friedman’ test with Durbin—Conover posterior test, linear regression, logistic regression, categorical Bayesian classifier, random forest method, neural network (multilayer perceptron), non-parametric estimation of the Nadaraya—Watson regression function, STATISTICA 10.0 and Jamovi 2.2.5, Python libraries. As a result of the study, a mathematical model for predicting learning outcomes in disciplines based on current performance in e-learning courses was created. The accuracy of the model depends on the week of training in which it is applied and reaches 92,6 %. In the early stages (e. g., for week 7), the accuracy is at least 85 % and varies depending on the contingent of the student population and disciplines. As a result of the study, a mobile application was developed that implements a predictive model and other related functions to inform the student about his/her estimated educational success. The created predictive model is based on current performance data obtained from electronic courses and is capable of making accurate predictions, which allows it to be applied in practice online and through the mobile application to inform students.

Heterogeneous Treatment Effect with Trained Kernels of the Nadaraya–Watson Regression

A new method for estimating the conditional average treatment effect is proposed in this paper. It is called TNW-CATE (the Trainable Nadaraya–Watson regression for CATE) and based on the assumption that the number of controls is rather large and the number of treatments is small. TNW-CATE uses the Nadaraya–Watson regression for predicting outcomes of patients from control and treatment groups. The main idea behind TNW-CATE is to train kernels of the Nadaraya–Watson regression by using a weight sharing neural network of a specific form. The network is trained on controls, and it replaces standard kernels with a set of neural subnetworks with shared parameters such that every subnetwork implements the trainable kernel, but the whole network implements the Nadaraya–Watson estimator. The network memorizes how the feature vectors are located in the feature space. The proposed approach is similar to transfer learning when domains of source and target data are similar, but the tasks are different. Various numerical simulation experiments illustrate TNW-CATE and compare it with the well-known T-learner, S-learner, and X-learner for several types of control and treatment outcome functions. The code of proposed algorithms implementing TNW-CATE is publicly available.

Improved Anomaly Detection by Using the Attention-Based Isolation Forest

A new modification of the isolation forest called the attention-based isolation forest (ABIForest) is proposed for solving the anomaly detection problem. It incorporates an attention mechanism in the form of Nadaraya–Watson regression into the isolation forest to improve the solution of the anomaly detection problem. The main idea underlying the modification is the assignment of attention weights to each path of trees with learnable parameters depending on the instances and trees themselves. Huber’s contamination model is proposed to be used to define the attention weights and their parameters. As a result, the attention weights are linearly dependent on learnable attention parameters that are trained by solving a standard linear or quadratic optimization problem. ABIForest can be viewed as the first modification of the isolation forest to incorporate an attention mechanism in a simple way without applying gradient-based algorithms. Numerical experiments with synthetic and real datasets illustrate that the results of ABIForest outperform those of other methods. The code of the proposed algorithms has been made available.

The stochastic approximation method for semi-recursive multivariate kernel-type regression estimation

In this research paper, we elaborate an extension of the semi-recursive kernel-type regression function estimator. We investigate the asymptotic properties of this estimator and compare them with non-recursive Nadaraya Watson regression estimator. From this perspective, we first calculate the bias and the variance of the proposed estimator which strongly depend on the choice of three parameters, namely the stepsizes (βn) and (γn) as well as the bandwidth (hn) chosen using one of the best methods of bandwidth selection, the bootstrap approach compared to the plug-in method. An appropriate choice of those parameters yields that, under some conditions, the MSE (Mean Squared Error) of the proposed estimator can be smaller than that of Nadaraya Watson's estimator. We corroborate our theoretical results through simulations studies and by considering two real dataset applications, the French Hospital Data of COVID-19 epidemic as well as the Plasmodium Falciparum Parasite Load (PL).

Random Survival Forests Incorporated by the Nadaraya-Watson Regression

An attention-based random survival forest (Att-RSF) is presented in the paper. The first main idea behind this model is to adapt the Nadaraya-Watson kernel regression to the random survival forest so that the regression weights or kernels can be regarded as trainable attention weights under important condition that predictions of the random survival forest are represented in the form of functions, for example, the survival function and the cumulative hazard function. Each trainable weight assigned to a tree and a training or testing example is defined by two factors: by the ability of corresponding tree to predict and by the peculiarity of an example which falls into a leaf of the tree. The second main idea behind Att-RSF is to apply the Huber's contamination model to represent the attention weights as the linear function of the trainable attention parameters. The Harrell's C-index (concordance index) measuring the prediction quality of the random survival forest is used to form the loss function for training the attention weights. The C-index jointly with the contamination model lead to the standard quadratic optimization problem for computing the weights, which has many simple algorithms for its solution. Numerical experiments with real datasets containing survival data illustrate Att-RSF.

A Comparative Study of Nearest Neighbor Regression and Nadaraya Watson Regression

Two non-parametric statistical methods are studied in this work. These are the nearest neighbor regression and the Nadaraya Watson kernel smoothing technique. We have proven that under a precise circumstance, the nearest neighborhood estimator and the Nadaraya Watson smoothing produce a smoothed data with a same error level, which means they have the same performance. Another result of the paper is that nearest neighborhood estimator performs better locally, but it graphically shows a weakness point when a large data set is considered on a global scale.

New non-parametric tests for independence

ABSTRACTIn this paper, we consider the problem of testing independence against stochastically increasing property. For the construction of new statistical tests, we employ Nadaraya–Watson regression estimator. We examine their asymptotic properties under the null and an alternative hypothesis. The performance of the tests is studied via power study. For this purpose, the bootstrap versions of proposed tests are utilized. The finite sample results indicate their competitiveness compared to existing statistical methods.

Determination of the order of stochastically linear dynamic systems by using non-parametric estimation of a regression function

In order to determine the accurate order of the model of discrete linear dynamic system, nonparametric algorithm is investigated. The algorithm is based on the application of the rule of selection of significant variables depending on a bandwidth. Selected variables are included in the Nadaraya–Watson regression estimation. The investigated dynamic system was simulated under diverse working conditions. The accuracy of the models estimated with the non-parametric methods is investigated. Computational experiments showed that using the non-parametric algorithms, the model orders can be determined. Because of the large number of cases investigated, the probability of the general validity of this nonparametric algorithm is high.

On asymptotic behavior of Nadaraya–Watson regression estimator

ABSTRACTThe random field (Xi, Yi), is dependent and assumed to satisfy some mild mixing conditions. In this article, a Nadaraya–Watson regression estimator (N. W. R. E.) is established and investigated. We show that the distribution of the N. W. R. E. is asymptotically normal under some commonly used conditions.

Estimating additive models with missing responses

ABSTRACTFor multivariate regressors, the Nadaraya–Watson regression estimator suffers from the well-known curse of dimensionality. Additive models overcome this drawback. To estimate the additive components, it is usually assumed that we observe all the data. However, in many applied statistical analysis missing data occur. In this paper, we study the effect of missing responses on the additive components estimation. The estimators are based on marginal integration adapted to the missing situation. The proposed estimators turn out to be consistent under mild assumptions. A simulation study allows to compare the behavior of our procedures, under different scenarios.

Classification of wood using differential thermogravimetric analysis

The aim of this study is to propose an alternative methodology to classify wood species using the first (DTG), second (2DTG), and third (3DTG) derivatives of the thermogravimetric curves (TG). Accordingly, the main contribution of this new procedure consists on classifying materials (wood) taking into account the mass loss rate and acceleration with respect to temperature. In our research, each TG curve is firstly smoothed using the local polynomial regression estimator, and the first, second, and third derivatives are estimated. The application of the local polynomial regression estimator provides a reliable way to obtain the TG derivatives, overcoming the noise problem in the TG derivative estimation. Then, using these estimated curves, the different wood classes are discriminated employing a nonparametric functional data analysis (NPFDA) technique, based on the Bayes rule and the Nadaraya-Watson regression estimator, and also novel functional generalized additive models (GAM). The latter allows to classify materials using simultaneously more than one type of thermal curves. The results are compared with those obtained using classical and machine learning multivariate supervised classification methods, such as Linear discriminant analysis, Quadratic classification, Naive Bayes, Logistic regression, \(k\) Nearest neighbors, Neural networks, and Support vector machines. A regression model consisting of the mixture of the first derivatives of four generalized logistic components, one per principal wood constituent (water, hemicellulose, cellulose, and lignin), is applied to fit the DTG curves. The resulting 16 parameters from this fit characterize each curve and are used as datasets to apply the multivariate supervised classification methods. The use of the TG derivatives jointly with the TG curves has proved to be an optimal discriminating feature, when the new functional GAM techniques are employed.

A sampling algorithm for bandwidth estimation in a nonparametric regression model with a flexible error density

The unknown error density of a nonparametric regression model is approximated by a mixture of Gaussian densities with means being the individual error realizations and variance a constant parameter. Such a mixture density has the form of a kernel density estimator of error realizations. An approximate likelihood and posterior for bandwidth parameters in the kernel-form error density and the Nadaraya–Watson regression estimator are derived, and a sampling algorithm is developed. A simulation study shows that when the true error density is non-Gaussian, the kernel-form error density is often favored against its parametric counterparts including the correct error density assumption. The proposed approach is demonstrated through a nonparametric regression model of the Australian All Ordinaries daily return on the overnight FTSE and S&P 500 returns. With the estimated bandwidths, the one-day-ahead posterior predictive density of the All Ordinaries return is derived, and a distribution-free value-at-risk is obtained. The proposed algorithm is also applied to a nonparametric regression model involved in state-price density estimation based on S&P 500 options data.

Central limit theorem for quadratic errors of Nadaraya–Watson regression estimator under dependence

In this paper, we derive the central limit theorem (CLT) for quadratic error where the Nadaraya-Watson regression estimator is employed under dependence. Our results may be applicable for various nonparametric regression models including nonlinear auto-regressive model. In addition, we discuss the usefulness of our results for obtaining test statistics specifying a regression model under dependence. The proof uses CLT for variable degenerate U -statistics due to Kim and Luo (in press).

Functional nonparametric classification of wood species from thermal data

In this study, thermogravimetric (TG) and differential scanning calorimetry (DSC) curves, obtained by means of a simultaneous TG/DSC analyzer, and statistical functional nonparametric methods are used to classify different wood species. The temperature ranges, where the highest probability of correct classification is reached, are also computed. As each observation is a curve, a nonparametric functional discriminant technique based on the Bayes rule and the Nadaraya–Watson regression estimator is used. It assigns a future observation to the highest probability predefined class (supervised classification). The smoothing parameter needed in this nonparametric method is selected according to the cross-validation technique. The method proposed is applied to a sample of 49 wood items (7 per wood class) and also to classify between hardwoods and softwoods. In all the cases, the samples have been successfully classified, obtaining better results with the TG curves. The results are compared with those obtained with other nonparametric methods based on boosting algorithm. A discussion about the relation of the obtained results with the referenced wood component degradation temperature ranks is presented.

Strong consistency of kernel estimates of regression function under dependence

By a classic Tauberian theorem and moment inequalities, strong L r -consistency ( 1 ≤ r < ∞ ) of Nadaraya–Watson regression estimates is established under conditions only on moments together with ρ -mixing or order of α -mixing. A similar L 1 -consistency result concerns Rosenblatt–Parzen density estimates, weakening conditions in previous literature.

Asymptotic normality of kernel type regression estimators for random fields

The asymptotic normality of the Nadaraya–Watson regression estimator is studied for α - mixing random fields. The infill-increasing setting is considered, that is when the locations of observations become dense in an increasing sequence of domains. This setting fills the gap between continuous and discrete models. In the infill-increasing case the asymptotic normality of the Nadaraya–Watson estimator holds, but with an unusual asymptotic covariance structure. It turns out that this covariance structure is a combination of the covariance structures that we observe in the discrete and in the continuous case.

A Monte Carlo Study of the Effect of Item Characteristic Curve Estimation on the Accuracy of Three Person-Fit Statistics

To date, there have been no studies comparing parametric and nonparametric Item Characteristic Curve (ICC) estimation methods on the effectiveness of Person-Fit Statistics (PFS). The primary aim of this study was to determine if the use of ICCs estimated by nonparametric methods would increase the accuracy of item response theory—based PFS for small sample sizes. Using three recognized PFS (lz, ECI2 z, and ECI4z), four estimation methods were compared: two parametric methods (the two-parameter logistic model and three-parameter logistic model) and two nonparametric methods (Nadaraya-Watson's regression and the local logistic regression). Finally, matrices of 100- and 1,000-answer vectors were generated for this Monte Carlo study. For both large and small sample sizes, the accuracy of the PFS was greater when used with the parametric models.