Nadaraya-Watson Kernel Regression Research Articles

Adaptive Nadaraya-Watson kernel regression estimators utilizing some non-traditional and robust measures: a numerical application of British food data

In nonparametric regression research, estimation of regression function is a prime concern. Recently, researchers developed some modified Nadaraya-Watson (N-W) regression estimators utilizing robust mean, median and harmonic mean. In this paper, we propose to utilize the additive combination of non-traditional measures i.e. (Hodges-Lehmann, Mid-Range, Tri-Mean, Quartile-Deviation) with the robust minimum covariance determinant (MCD) scale estimator in (N-W) regression estimator. Utilizing these measures, we get some new versions of (N-W) regression estimator. We also attempted to derive the properties of the proposed versions, such as bias, variance, mean square error (MSE), and mean integrated square error (MISE). The proposed estimators are compared with some of the existing estimators available in literature through a simulation study, utilizing two artificial populations. We also incorporated real-life application by taking British food data set denoted as Engel95, and assess the predictive ability of nonparametric regression, based on proposed and existing N-W estimators.

Regional forecasting of PM2.5 concentrations: A novel model based on the empirical orthogonal function analysis and Nadaraya–Watson kernel regression estimator

Predicting regional air pollutant concentrations quickly and accurately is still a challenge when meteorological data that affect pollutant concentrations are not available. To avoid modeling single points, which is tedious, it is necessary to explore more efficient and reliable regional methods with fewer input data. In this study, the principle and procedure of a new hybrid model, EOF-NWKRE, based on empirical orthogonal function (EOF) decomposition and the N–W kernel regression estimator (NWKRE) are proposed, then, the PM2.5 concentrations of 13 cities in Jiangsu Province is simulated by the EOF-NWKRE. The results show that the average prediction accuracy of EOF-NWKRE model is 74.38%, with more than 92% of the cumulative variance. The model is completely self-driven by PM2.5 data without covariate data input, which reduces the computational burden, improves the computational efficiency, and results in better prediction accuracy. It is suitable for the prediction of pollutant concentrations in a large area.

Attention and self-attention in random forests

New models of random forests jointly using the attention and self-attention mechanisms are proposed for solving the regression problem. The models can be regarded as extensions of the attention-based random forest whose idea stems from applying a combination of the Nadaraya–Watson kernel regression and the Huber’s contamination model to random forests. The self-attention aims to capture dependencies of the tree predictions and to remove noise or anomalous predictions in the random forest. The self-attention module is trained jointly with the attention module for computing weights. It is shown that the training process of attention weights is reduced to solving a single quadratic or linear optimization problem. Three modifications of the self-attention are proposed and compared. A specific multi-head self-attention for the random forest is also considered. Heads of the self-attention are obtained by changing its tuning parameters including the kernel parameters and the contamination parameter of models. The proposed modifications of the attention and self-attention combinations are verified and compared with other random forest models by using several datasets. The code implementing the corresponding algorithms is publicly available.

Investigation of financial applications with blockchain technology

Aims: This article investigates recent advancements in machine learning and blockchain technology for cryptocurrency price prediction. The study presents a ML system using various techniques applied to six different datasets. The findings highlight that simpler models can outperform complex ones in predicting cryptocurrency prices. Methods: The methods used in this study include applying diverse ML techniques such as LSTM, CNN, SVM, KNN, XGBoost, Astro ML, LASSO, RIDGE, linear regression, DT, and GP on six cryptocurrency datasets to predict prices. Results: The research evaluated various machine learning techniques for predicting cryptocurrency prices and reported the following RMSE values: Bitcoin prediction using Nadaraya-Watson kernel regression yielded an RMSE of 0.17, while Dogecoin prediction with linear regression resulted in an RMSE of 0.032. Ethereum price prediction using Gaussian regression achieved an RMSE of 0.02. For USD Coin, a combination of XGBoost, Gaussian regression, and Ridge techniques led to an RMSE of 0.014. Binance Coin price prediction using Gaussian regression had an RMSE of 0.032, and finally, Cardano Coin prediction employing LSTM reached an RMSE of 0.059. Conclusion: This study demonstrated the effectiveness of various machine learning techniques in predicting cryptocurrency prices. It revealed that simpler models can outperform complex ones in certain cases. The research contributes valuable insights to the field and can guide future work in cryptocurrency price prediction. The proposed model achieved promising results as evaluated by the RMSE metric.

Using method of Nadaraya-Watson kernel regression to detection outliers in multivariate data fusion

In this paper, the researcher discussed a developed approach to the detection of outliers that is suited to multivariate data fusion. The challenge in outlier detection when dealing with multivariate data it is the detection of the outlier with more than two dimensions. To address this issue, the researcher developed a method to detect anomalies using methods based on local density including comparing a specific observations density with the densities of its neighboring observations. To make such comparisons, the researcher often employs an outlier score. In this study, various density estimation functions and distance metrics were utilized. Nadaraya-Watson kernel regression for multivariate data considered the KNN with multivariate data. Finally, the estimate of the Volcano kernel method is an essential method for outliers detection. In the simulation experiments of multivariate data with (4,6,8) variables and (60,120,180) observations, the results of simulation experiments by using the criterion of the precision evaluation showed that the N-W method is better than the VOL method in outlier detection in multivariate data.

Very short-term probabilistic prediction of PV based on multi-period error distribution

• Empirical mode decomposition based on long short-term memory and backpropagation neural network (EEMD-LSTM-BP) is proposed. • EEMD is utilized to study the characteristic of wave behaviors in different frequency domains. • LSTM and BP are used to determine intrinsic mode functions (IMFs) and remaining components. • Based on the dataset of PV stations in Ningxia Province, the case studies verify the method's feasibility. As the penetration rate of photovoltaic (PV) in the grid increases, enormous challenges have been brought into power grid dispatcher's operation. Efficient and accurate PV power prediction is the key to solve this problem. Considering multi-period error distribution (MPED), a novel probabilistic prediction approach via ensemble empirical mode decomposition based on long short-term memory and backpropagation neural network (EEMD-LSTM-BP) is proposed. EEMD is utilized to study the characteristic of wave behaviors in different frequency domains. LSTM and BP are used to determine intrinsic mode functions (IMFs) and remaining components, respectively. Afterward, based on the prediction errors, PV power output fluctuation in different periods is analysed. The segment points are determined by Nadaraya-Watson (N-W) kernel regression. The bounds of prediction intervals (PIs) are quantified based on the error probability distribution. Based on the dataset of PV stations in Ningxia Province, the case studies verify the method's feasibility.

Random Survival Forests Incorporated by the Nadaraya-Watson Regression

An attention-based random survival forest (Att-RSF) is presented in the paper. The first main idea behind this model is to adapt the Nadaraya-Watson kernel regression to the random survival forest so that the regression weights or kernels can be regarded as trainable attention weights under important condition that predictions of the random survival forest are represented in the form of functions, for example, the survival function and the cumulative hazard function. Each trainable weight assigned to a tree and a training or testing example is defined by two factors: by the ability of corresponding tree to predict and by the peculiarity of an example which falls into a leaf of the tree. The second main idea behind Att-RSF is to apply the Huber's contamination model to represent the attention weights as the linear function of the trainable attention parameters. The Harrell's C-index (concordance index) measuring the prediction quality of the random survival forest is used to form the loss function for training the attention weights. The C-index jointly with the contamination model lead to the standard quadratic optimization problem for computing the weights, which has many simple algorithms for its solution. Numerical experiments with real datasets containing survival data illustrate Att-RSF.

Attention-based random forest and contamination model

A new approach called ABRF (the attention-based random forest) and its modifications for applying the attention mechanism to the random forest (RF) for regression and classification are proposed. The main idea behind the proposed ABRF models is to assign attention weights with trainable parameters to decision trees in a specific way. The attention weights depend on the distance between an instance, which falls into a corresponding leaf of a tree, and training instances, which fall in the same leaf. This idea stems from representation of the Nadaraya–Watson kernel regression in the form of a RF. Three modifications of the general approach are proposed. The first one is based on applying the Huber’s contamination model and on computing the attention weights by solving quadratic or linear optimization problems. The second and the third modifications use the gradient-based algorithms for computing an extended set of the attention trainable parameters. Numerical experiments with various regression and classification datasets illustrate the proposed method. The code implementing the approach is publicly available.

Random Forests with Attentive Nodes

An extension of the attention-based random forest called ABRF-E for incorporating the attention mechanism into the random forest (RF) for improving the machine learning model performance is proposed. The main idea behind the proposed ABRF-E model is based on the Nadaraya-Watson kernel regression model. The attention weights with trainable parameters are assigned to each node of decision trees. The weights depend on the distance between an instance, which falls into a corresponding node of a tree, and instances, which fall in the same node. The attention weight are computed by using the Huber's contamination model which provides the linear relationship between the attention weights and their trainable parameters. The linear relationship leads to solving a quadratic optimization problem and allows us to avoid gradient-based algorithms for training attention parameters and attention weights. Numerical experiments with real datasets illustrate the proposed model.

A Comparison Between New Modification of ANWK and Classical ANWK Methods in Nonparametric Regression

Nonparametric kernel estimators are mostly used in a variety of statistical research fields. Nadaraya-Watson kernel estimator (NWK) is one of the most important nonparametric kernel estimator that is often used in regression models with a fixed bandwidth. In this article, we consider the four new Proposed Adaptive Nadaraya-Watson Kernel Regression Estimators (Interquartile Range, Standard Deviation, Mean Absolute Devotion, and Median Absolute Deviation) rather than (Fixed Bandwidth, Adaptive Geometric, Adaptive Mean, Adaptive Range, and Adaptive Median). The outcomes in both simulation and actual data in Leukemia Cancer show that the four new ANW Kernel Estimators (Interquartile Range, Standard Deviation, Mean Absolute devotion, and Median Absolute Deviation) is more effective than the kernel estimations with fixed bandwidth in previous studies using Mean Square Error (MSE) Criterion.

Novel kernel density estimator based on ensemble unbiased cross-validation

Unbiased cross-validation (UCV) is a commonly-used method to calculate the optimal bandwidth for the kernel density estimator (KDE), which estimates the underlying probability density function (PDF) for a given data set. Since the UCV method was proposed, there have been few studies that have pointed out its instability when determining the KDE bandwidth. Following the principle of stability improvement, this paper presents a novel ensemble UCV based KDE (EUCV-KDE), which determines the expectation of an estimated PDF using an ensemble of data-block based UCVs rather than a single data-point based UCV. To derive the optimal bandwidth, a novel objective function is designed for EUCV-KDE by considering the empirical and structural risk of KDE together. We validate the rationality and effectiveness of EUCV-KDE on 10 probability distributions. The experimental results show that EUCV-KDE is convergent as the number of data-block based UCVs increases and can obtain a more stable and better prediction performance than the classical UCV-KDE and the revisited cross-validation (RCV) based KDE (RCV-KDE). In addition, a real-world application based on UK climate data is provided to further validate the effectiveness of EUCV-KDE by determining the optimal bandwidth for Nadaraya-Watson kernel regression estimator.

Kernel Regression Residual Decomposition Method to Detect Rolling Element Bearing Faults

The raw vibration signal carries a great deal of information representing the mechanical equipment's health conditions. However, in the working condition, the vibration response signals of faulty components are often characterized by the presence of different kinds of impulses, and the corresponding fault features are always immersed in heavy noises. Therefore, signal denoising is one of the most important tasks in the fault detection of mechanical components. As a time-frequency signal processing technique without the support of the strictly mathematical theory, empirical mode decomposition (EMD) has been widely applied to detect faults in mechanical systems. Kernel regression (KR) is a well-known nonparametric mathematical tool to construct a prediction model with good performance. Inspired by the basic idea of EMD, a new kernel regression residual decomposition (KRRD) method is proposed. Nonparametric Nadaraya–Watson KR and a standard deviation (SD) criterion are employed to generate a deep cascading framework including a series of high-frequency terms denoted by residual signals and a final low-frequency term represented by kernel regression signal. The soft thresholding technique is then applied to each residual signal to suppress noises. To illustrate the feasibility and the performance of the KRRD method, a numerical simulation and the faulty rolling element bearings of well-known open access data as well as the experimental investigations of the machinery simulator are performed. The fault detection results show that the proposed method enables the recognition of faults in mechanical systems. It is expected that the KRRD method might have a similar application prospect of EMD.

An Upper Bound of the Bias of Nadaraya-Watson Kernel Regression under Lipschitz Assumptions

The Nadaraya-Watson kernel estimator is among the most popular nonparameteric regression technique thanks to its simplicity. Its asymptotic bias has been studied by Rosenblatt in 1969 and has been reported in several related literature. However, given its asymptotic nature, it gives no access to a hard bound. The increasing popularity of predictive tools for automated decision-making surges the need for hard (non-probabilistic) guarantees. To alleviate this issue, we propose an upper bound of the bias which holds for finite bandwidths using Lipschitz assumptions and mitigating some of the prerequisites of Rosenblatt’s analysis. Our bound has potential applications in fields like surgical robots or self-driving cars, where some hard guarantees on the prediction-error are needed.

Estimation of a Finite Population Mean under Random Nonresponse Using Kernel Weights

Nonresponse is a potential source of errors in sample surveys. It introduces bias and large variance in the estimation of finite population parameters. Regression models have been recognized as one of the techniques of reducing bias and variance due to random nonresponse using auxiliary data. In this study, it is assumed that random nonresponse occurs in the survey variable in the second stage of cluster sampling, assuming full auxiliary information is available throughout. Auxiliary information is used at the estimation stage via a regression model to address the problem of random nonresponse. In particular, auxiliary information is used via an improved Nadaraya–Watson kernel regression technique to compensate for random nonresponse. The asymptotic bias and mean squared error of the estimator proposed are derived. Besides, a simulation study conducted indicates that the proposed estimator has smaller values of the bias and smaller mean squared error values compared to existing estimators of a finite population mean. The proposed estimator is also shown to have tighter confidence interval lengths at 95% coverage rate. The results obtained in this study are useful for instance in choosing efficient estimators of a finite population mean in demographic sample surveys.

On the Nadaraya–Watson kernel regression estimator for irregularly spaced spatial data

We investigate the asymptotic normality of the Nadaraya–Watson kernel regression estimator for irregularly spaced data collected on a finite region of the lattice Zd where d is a positive integer. The results are stated for strongly mixing random fields in the sense of Rosenblatt (1956) and for weakly dependent random fields in the sense of Wu (2005). Only minimal conditions on the bandwidth parameter and simple conditions on the dependence structure of the data are assumed.

Curve-Fitting Method for Implied Volatility

Curve-fitting methods are widely used in derivatives markets for construction of the implied volatility surface (IVS). Here we discuss the goodness of fit, smoothness, and economic implications of 12 distinctive curve-fitting methods. The choice of method relies on specific requirements. When fitting the Chicago Board Options Exchange data, three interpolation methods were found to provide the best goodness, whereas quadratic regression, the Nadaraya–Watson kernel regression, and the theoretical Carr–Wu model generate the smoothest surfaces. Because of the irregular nature of the emerging options market data, we propose a transformation method to improve three statistical methods to satisfy the Lee’s condition. Empirically, quadratic regression provides the best goodness when fitting the China 50ETF options data. In addition, the Carr–Wu model is a very good alternative because it natively satisfies the Lee’s condition and has economic implications.

Complexity of Products: The Effect of Data Regularisation.

Among several developments, the field of Economic Complexity (EC) has notably seen the introduction of two new techniques. One is the Bootstrapped Selective Predictability Scheme (SPSb), which can provide quantitative forecasts of the Gross Domestic Product of countries. The other, Hidden Markov Model (HMM) regularisation, denoises the datasets typically employed in the literature. We contribute to EC along three different directions. First, we prove the convergence of the SPSb algorithm to a well-known statistical learning technique known as Nadaraya-Watson Kernel regression. The latter has significantly lower time complexity, produces deterministic results, and it is interchangeable with SPSb for the purpose of making predictions. Second, we study the effects of HMM regularization on the Product Complexity and logPRODY metrics, for which a model of time evolution has been recently proposed. We find confirmation for the original interpretation of the logPRODY model as describing the change in the global market structure of products with new insights allowing a new interpretation of the Complexity measure, for which we propose a modification. Third, we explore new effects of regularisation on the data. We find that it reduces noise, and observe for the first time that it increases nestedness in the export network adjacency matrix.

APPROXIMATELY SINGULAR WAVELET

The problem of approximation is relevant for most engineering applications. In this connection, the universal methods of approximation are of interest. The method of nonparametric approximation is developing in the paper – the method of singular wavelets. The method includes an effective numerical algorithm based on the summation of a recursive sequence of functions. The universal algorithm of approximation makes it possible to apply it to approximate one-dimensional and multidimensional functions, in decision support systems, in the processing of stochastic information, pattern recognition, and solution of boundary-value problems.The introduction explain the idea of the method of singular wavelets – to combine the theory of wavelets with the Nadaraya-Watson kernel regression estimator. Usually, Nadaraya-Watson kernel regression are considered as an example of non- parametric estimation. However, one parameter, the smoothing parameter, is still present in the traditional kernel regression algorithm. The choice of the optimal value of this parameter is a complex mathematical problem, and numerous studies have been devoted to this question. In the approximation by the method of singular wavelets, summation of Nadaraya-Watson kernel regression estimates with the smoothing parameter takes place, which solves the problem of the optimal choice of this parameter.In the main part of the paper theorems are formulated that determine the properties of the regularized wavelet transform. Sufficient conditions for uniform convergence of the wavelet series are obtained for the first time. To illustrate the effectiveness of the numerical approximation algorithm, we consider an example of the quasi-interpolation of the Runge function by wavelets with a uniform distribution of interpolation nodes.

The mean shift algorithm and its relation to kernel regression

We investigate the connection between the asymptotic bias of the well-known Nadaraya–Watson kernel regression and the mean shift (MS) vector with the Gaussian kernel. We first show that the asymptotic bias for the univariate Nadaraya–Watson kernel regression can be estimated using the MS scalar. Then, we investigate the general D-dimensional case (D > 1) and derive a formula for the asymptotic bias as a function of the MS vector. We show that when the regression function is a linear function of its entries, then the asymptotic bias can be represented as a linear function of the MS vector. The MS algorithm for the univariate and the linear cases can be used to find points where the Nadaraya–Watson kernel regression gives an unbiased estimate. Furthermore, this connection suggests that exploiting the theoretical properties of the asymptotic bias of the Nadaraya–Watson kernel regression may be helpful to show the convergence of the MS algorithm. Through the simulations, we show that how the given theoretical results can be used to estimate the bias of the estimated clean signal by just observing the noisy signal. Having access to the bias of the estimator, enables us to evaluate the accuracy of the estimated clean data, which later will be used to design a quantizer.

Parameter Optimization for Nadaraya-Watson Kernel Regression Method with Small Samples

Many current regression algorithms have unsatisfactory prediction accuracy with small samples. To solve this problem, a regression algorithm based on Nadaraya-Watson kernel regression (NWKR) is proposed. The proposed method advocates parameter selection directly from the standard deviation of training data, optimized with leave-one-out cross- validation (LOO-CV). Good generalization performance of the proposed parameter selection is demonstrated empirically using small sample regression problems with Gaussian noise. The results show that proposed parameter optimization method is more robust and accurate than other methods for different noise levels and different sample sizes, and indicate the importance of Vapnik’s e-insensitive loss for regression problems with small samples.