Nonparametric Regression Function Research Articles

Deep regression learning with optimal loss function

In this paper, we develop a novel efficient and robust nonparametric regression estimator under a framework of a feedforward neural network (FNN). There are several interesting characteristics for the proposed estimator. First, the loss function is built upon an estimated maximum likelihood function, which integrates the information from observed data as well as the information from the data distribution. Consequently, the resulting estimator has desirable optimal properties, such as efficiency. Second, different from the traditional maximum likelihood estimation (MLE), the proposed method avoids the specification of the distribution, making it adaptable to various distributions such as heavy tails and multimodal or heterogeneous distributions. Third, the proposed loss function relies on probabilities rather than direct observations as in least square loss, contributing to the robustness of the proposed estimator. Finally, the proposed loss function involves a nonparametric regression function only. This enables the direct application of the existing packages, simplifying the computational and programming requirements. We establish the large sample property of the proposed estimator in terms of its excess risk and minimax near-optimal rate. The theoretical results demonstrate that the proposed estimator is equivalent to the true MLE where the density function is known in terms of excess risk. Our simulation studies show that the proposed estimator outperforms the existing methods based on prediction accuracy, efficiency and robustness. Particularly, it is comparable to the MLE with the known density and even gets slightly better as the sample size increases. This implies that the adaptive and data-driven loss function from the estimated density may offer an additional avenue for capturing valuable information. We further apply the proposed method to four real data examples, resulting in significantly reduced out-of-sample prediction errors compared to existing methods.

Evaluation of the Performance of Kernel Non-parametric Regression and Ordinary Least Squares Regression

Researchers need to understand the differences between parametric and nonparametric regression models and how they work with available information about the relationship between response and explanatory variables and the distribution of random errors. This paper proposes a new nonparametric regression function for the kernel and employs it with the Nadaraya-Watson kernel estimator method and the Gaussian kernel function. The proposed kernel function (AMS) is then compared to the Gaussian kernel and the traditional parametric method, the ordinary least squares method (OLS). The objective of this study is to examine the effectiveness of nonparametric regression and identify the best-performing model when employing the Nadaraya-Watson kernel estimator method with the proposed kernel function (AMS), the Gaussian kernel, and the ordinary least squares (OLS) method. Additionally, it determines which method yields the most accurate results when analyzing nonparametric regression models and provides valuable insights for practitioners looking to apply these techniques in real-world scenarios. However, criteria such as generalized cross-validation (GCV), mean square error (MSE), and coefficient determination are used to select the most efficient estimated model. Simulated data was used to evaluate the performance and efficiency of estimators using different sample sizes. The results favorable the simulation illustrate that the Nadaraya-Watson kernel estimator using the proposed kernel function (AMS) exhibited favorable and superior performance compared to other methods. The coefficients of determination indicate that the highest values attained were 98%, 99%, and 99%. The proposed function (AMS) yielded the lowest MSE and GCV values across all samples. Therefore, this suggests that the model can generate precise predictions and enhance the performance of the focused data.

Strong Consistency of Wavelet Estimator for Biased Nonparametric Regression Function Under Strong Mixing

Strong Consistency of Wavelet Estimator for Biased Nonparametric Regression Function Under Strong Mixing

Properties of Test Statistics for Nonparametric Cointegrating Regression Functions Based on Subsamples

Nonparametric cointegrating regression models have been extensively used in financial markets, stock prices, heavy traffic, climate datasets, and energy markets. Models with parametric regression functions can be more appealing in practice compared to nonparametric forms, but do result in potential functional misspecification. Thus, there exists a vast literature on developing a model specification test for parametric forms of regression functions. In this article, we develop two test statistics which are applicable for the endogenous regressors driven by long memory and semi-long memory input shocks in the regression model. The limit distributions of the test statistics under these two scenarios are complicated and cannot be effectively used in practice. To overcome this difficulty, we use the subsampling method and compute the test statistics on smaller blocks of the data to construct their empirical distributions. Throughout, Monte Carlo simulation studies are used to illustrate the properties of test statistics. We also provide an empirical example of relating gross domestic product to total output of carbon dioxide in two European countries. Supplementary materials for this article are available online.

Comparative Analysis of Predictive Performance in Nonparametric Functional Regression: A Case Study of Spectrometric Fat Content Prediction

Objective: This research aims to compare two nonparametric functional regression models, the Kernel Model and the K-Nearest Neighbor (KNN) Model, with a focus on predicting scalar responses from functional covariates. Two semi-metrics, one based on second derivatives and the other on Functional Principle Component Analysis, are employed for prediction. The study assesses the accuracy of these models by computing Mean Square Errors (MSE) and provides practical applications for illustration.
 Method: The study delves into the realm of nonparametric functional regression, where the response variable (Y) is scalar, and the covariate variable (x) is a function. The Kernel Model, known as funopare.kernel.cv, and the KNN Model, termed funopare.knn.gcv, are used for prediction. The Kernel Model employs automatic bandwidth selection via Cross-Validation, while the KNN Model employs a global smoothing parameter. The performance of both models is evaluated using MSE, considering two different semi-metrics.
 Results: The results indicate that the KNN Model outperforms the Kernel Model in terms of prediction accuracy, as supported by the computed MSE. The choice of semi-metric, whether based on second derivatives or Functional Principle Component Analysis, impacts the model's performance. Two real-world applications, Spectrometric Data for predicting fat content and Canadian Weather Station data for predicting precipitation, demonstrate the practicality and utility of the models.
 Conclusion: This research provides valuable insights into nonparametric functional regression methods for predicting scalar responses from functional covariates. The KNN Model, when compared to the Kernel Model, offers superior predictive performance. The selection of an appropriate semi-metric is essential for model accuracy. Future research may explore the extension of these models to cases involving multivariate responses and consider interactions between response components.

Semi-parametric single-index predictive regression models with cointegrated regressors

This paper considers the estimation of a semi-parametric single-index regression model that allows for nonlinear predictive relationships. This model is useful for predicting financial asset returns, whose observed behaviour resembles a stationary process, if the multiple nonstationary predictors are cointegrated. The presence of cointegrated regressors imposes a single-index structure in the model, and this structure not only balances the nonstationarity properties of the multiple predictors with the stationarity properties of asset returns but also avoids the curse of dimensionality associated with nonparametric regression function estimation. An orthogonal series expansion is used to approximate the unknown link function for the single-index component. We consider the constrained nonlinear least squares estimator of the single-index (or the cointegrating) parameters and the plug-in estimator of the link function, and derive their asymptotic properties. In an empirical application, we find some evidence of in-sample nonlinear predictability of U.S. stock returns using cointegrated predictors. We also find that the single-index model in general produces better out-of-sample forecasts than both the historical average benchmark and the linear predictive regression model.

Detecting change structures of nonparametric regressions

This research investigates detecting change points of general nonparametric regression functions by introducing a novel criterion. It is based on the moving sums of conditional expectation to avoid both computationally expensive algorithms, exhaustive search methods need, and false positives hypothesis testing-based approaches encounter. This new criterion can simultaneously and consistently, in a certain sense, detect multiple change points and their locations even when, as the sample size goes to infinity, the number of changes grows up to infinity, and some changes tend to zero. Further, because of its visualization nature, in practice, the locations can be relatively more easily identified, by plotting its signal statistic, than existing methods in the literature. Numerical studies are conducted to examine its performance in finite sample scenarios, and a real data example is analyzed for illustration.

Disparities in sleep-wake patterns by labor force status: Population-based findings

ABSTRACT Sleep disturbances have been associated with unemployment, but variation in sleep-wake patterns by labor force status has rarely been examined. With a population-based sample, we investigated differences in sleep-wake patterns by labor force status (employed, unemployed, and not-in-the-labor-force) and potential disparities by sociodemographic variables. The analysis included 130,602 adults aged 25–60 y, who participated in the American Time Use Survey between 2003 and 2019. Individual sleep-wake pattern was extracted from time use logs in a strict 24-h period (04:00 h–03:59 h). Functional nonparametric regression models based on dimensionality reduction and neighborhood matching were applied to model the relationship between sleep-wake patterns and labor force status. Specifically, we predicted changes in intra-person sleep-wake patterns under hypothetical changes of labor force status from employed to unemployed or not-in-the-labor-force. We then studied moderations of this association by gender, race/ethnicity and educational attainment. In comparison to the employed state, unemployed and not-in-the-labor-force states were predicted to have later wake-times, later bedtimes, and higher tendency for taking midday naps. Changes in labor force status led to more apparent shifts in wake-times than in bedtimes. Additionally, sleep schedules of Hispanics and those with higher education level were more vulnerable to the change of labor force status from employed to unemployed.

A spline-assisted semiparametric approach to nonparametric measurement error models

A spline-assisted approach is proposed to handle the measurement error problem in treating the pollution and asthma data. It is well known that the minimax rate of convergence in nonparametric regression function estimation of a random variable measured with error is much slower than the rate in the error free case. A different problem is considered. It is shown that if one is willing to impose a relatively mild assumption in requiring that the error-prone variable has a compact support, then standard nonparametric results are obtainable for measurement error models. New and constructive methods to take full advantage of the compact support assumption via spline-assisted semiparametric methods are proposed. It is proven that the new estimator achieves the usual nonparametric rate as if there were no measurement error. Furthermore it is shown that similar spline approach can be used to assess the probability density function on a compact set, using observations with error, and the resulting estimator differs from the true density function only by a constant scale. In addition, it retains the nonparametric density convergence properties of the error free case. The performance of the new methods is demonstrated through simulations and the methods are implemented to analyze the relation between asthma and pollution.

Generalized kernel regularized least squares estimator with parametric error covariance

A two-step estimator of a nonparametric regression function via Kernel regularized least squares (KRLS) with parametric error covariance is proposed. The KRLS, not considering any information in the error covariance, is improved by incorporating a parametric error covariance, allowing for both heteroskedasticity and autocorrelation, in estimating the regression function. A two step procedure is used, where in the first step, a parametric error covariance is estimated by using KRLS residuals and in the second step, a transformed model using the error covariance is estimated by KRLS. Theoretical results including bias, variance, and asymptotics are derived. Simulation results show that the proposed estimator outperforms the KRLS in both heteroskedastic errors and autocorrelated errors cases. An empirical example is illustrated with estimating an airline cost function under a random effects model with heteroskedastic and correlated errors. The derivatives are evaluated, and the average partial effects of the inputs are determined in the application.

A Functional Approach for Analyzing Time-Dependent Driver Response Behavior to Real-World Connected Vehicle Warnings

Understanding driver response behavior to Connected Vehicle (CV) warnings is one of the fundamental tasks in improving road safety. However, previous studies often used aggregated safety measures and simulation environments to accomplish this task. Consequently, time-dependent driver response behavior under real-world conditions has rarely been investigated. This study proposes a new functional data analysis (FDA) approach to analyze time-dependent driver response behavior to CV warnings obtained from the New York City CV Pilot Deployment (NYC CVPD) project. Sparse functional design that can account for irregularly spaced functional measurements is adopted to accommodate the potential noise contaminated real-world CV data. Functional principal component analysis and nonparametric functional linear regression are used to smooth raw driver behavior profile and estimate time-dependent effect of driver response behavior, respectively. The speed compliance application implemented in the NYC CVPD project is used as the case study. By modeling time series of speed data after receiving warnings as continuous functions, the proposed FDA approach reveals new patterns of driver response behavior over time, such as the diminishing effect of drivers’ speed reduction and the increase in variability of driver response behavior over time, which have rarely been explored using traditional approaches. Moreover, the proposed FDA approach supports the extraction of certain traditional safety measures and has the potential to be generalized to analyze various CV applications. The findings of this study can support the calibration of detailed driver behavior in CV environments and facilitate better CV application design and further investigation of the benefits of CVs.

Distributional data analysis of accelerometer data from the NHANES database using nonparametric survey regression models

Abstract The aim of this paper is twofold. First, a new functional representation of accelerometer data of a distributional nature is introduced to build a complete individualized profile of each subject’s physical activity levels. Second, we extend two nonparametric functional regression models, kernel smoothing and kernel ridge regression, to handle survey data and obtain reliable conclusions about the influence of physical activity. The advantages of the proposed distributional representation are demonstrated through various analyses performed on the NHANES cohort, which possesses a complex sampling design.

THE USE OF PENALIZED WEIGHTED LEAST SQUARE TO OVERCOME CORRELATIONS BETWEEN TWO RESPONSES

The non-parametric regression model can consider two correlated responses. However, for these conditions, we cannot use the usual estimation process because there are violations of assumptions. To solve this problem, we use a penalized weighted least square involving knots, smoothing parameters, and weighting in the estimation criteria simultaneously. The estimation process involves a weighted criteria matrix in the estimation criteria. Estimation results show that the estimated two-response non-parametric regression function with penalized spline is a linear estimation class in y response observation and depends on the knot point and smoothing parameter. Furthermore, the use of the model on toddler growth data shows some changes in the pattern of weight and height gain. The pattern segmentation that experienced a gradual increase was age 7-43 months for weight and age 6-54 months for height

Nonparametric Functional Data Analysis for Forecasting Container Throughput: The Case of Shanghai Port

Transportation is one of the major carbon sources in China. Container throughput is one of the main influencing factors of ports’ carbon emission budget, and accurate prediction of container throughput is of great significance to the study of carbon emissions. Time series methods are key techniques and frequently used for container throughput. However, the existing time series methods treat container throughput data as discrete points and ignore the functional characteristics of the data. There has recently been interest in developing new statistical methods to predict time series by taking into account a continuous set of past values as predictors. In addition, to eliminate the linear constraint in the functional time series prediction approach, we propose a functional version of a nonparametric model that allows using a continuous path in the past to predict future values of the process, including functional nonparametric regression and functional conditional quantile and functional conditional mode models, to forecast the container throughput of Shanghai Port. For the purpose better forecasting, an experiment was conducted to compare our functional data analysis approaches with other forecasting methods. The results indicated that nonparametric functional forecasting methods exhibit more significant performance than other classical models, including the functional linear regression model, nonparametric regression model, and autoregressive integrated moving-average model. At the same time, we also compared the prediction accuracy of the three nonparametric functional methods.

The trimmed mean in non-parametric regression function estimation

This article studies a trimmed version of the Nadaraya–Watson estimator for the unknown non-parametric regression function. The characterization of the estimator through the minimization problem is established, and its pointwise asymptotic distribution is derived. The robustness property of the proposed estimator is also studied through the breakdown point. Moreover, similar to the trimmed mean in the location model, and for a wide range of trimming proportion, the proposed estimator possesses good efficiency and high breakdown point, which is out of the ordinary properties for any estimator. Furthermore, the usefulness of the proposed estimator is shown for two benchmark real data and various simulated data.

Nonparametric Testing for Hammerstein Systems

We examine the problem of testing a parametric form of the nonlinearity of Hammerstein systems against a nonparametric alternative. This article proposes test statistics relying on nonparametric orthogonal series regression function estimates. An asymptotic theory of the proposed tests is established including the asymptotic normality under the null hypothesis that the parametric form of the system nonlinearity is correct. Moreover, it is shown that under nonparametric alternatives, the probability of detecting that the parametric model is incorrect tends to one. The rate of detecting local alternatives is also established.

Nonparametric inverse-probability-weighted estimators based on the highly adaptive lasso.

Inverse-probability-weighted estimators are the oldest and potentially most commonly used class of procedures for the estimation of causal effects. By adjusting for selection biases via a weighting mechanism, these procedures estimate an effect of interest by constructing a pseudopopulation in which selection biases are eliminated. Despite their ease of use, these estimators require the correct specification of a model for the weighting mechanism, are known to be inefficient, and suffer from the curse of dimensionality. We propose a class of nonparametric inverse-probability-weighted estimators in which the weighting mechanism is estimated via undersmoothing of the highly adaptive lasso, a nonparametric regression function proven to converge at nearly -rate to the true weighting mechanism. We demonstrate that our estimators are asymptotically linear with variance converging to the nonparametric efficiency bound. Unlike doubly robust estimators, our procedures require neither derivation of the efficient influence function nor specification of the conditional outcome model. Our theoretical developments have broad implications for the construction of efficient inverse-probability-weighted estimators in large statistical models and a variety of problem settings. We assess the practical performance of our estimators in simulation studies and demonstrate use of our proposed methodology with data from a large-scale epidemiologicstudy.

Testing stochastic dominance with many conditioning variables

We propose tests of the conditional first- and second-order stochastic dominance in the presence of growing numbers of covariates. Our approach builds on a semiparametric location-scale model, where the conditional distribution of the outcome given the covariates is characterized by nonparametric mean and skedastic functions with independent innovations from an unknown distribution. The nonparametric regression functions are estimated by utilizing the ℓ1-penalized nonparametric series estimation with thresholding. Deviation bounds for the regression functions and series coefficients estimates are obtained allowing for the time series dependence. We propose test statistics, which are the maximum (integrated) deviation of a composite of the estimated regression functions and the residual empirical distribution, and introduce a smooth stationary bootstrap to compute p-values. We investigate the finite sample performance of the bootstrap critical values by a set of Monte Carlo simulations. Finally, our method is illustrated by an application to stochastic dominance among portfolio returns given all the past information.

Estimation and Model Selection for Nonparametric Function-on-Function Regression

Regression models with a functional response and functional covariate have received significant attention recently. While various nonparametric and semiparametric models have been developed, there is an urgent need for model selection and diagnostic methods. In this article, we develop a unified framework for estimation and model selection in nonparametric function-on-function regression. We propose a general nonparametric functional regression model with the model space constructed through smoothing spline analysis of variance (SS ANOVA). The proposed model reduces to some of the existing models when selected components in the SS ANOVA decomposition are eliminated. We propose new estimation procedures under either L 1 or L 2 penalty and show that the combination of the SS ANOVA decomposition and L 1 penalty provides powerful tools for model selection and diagnostics. We establish consistency and convergence rates for estimates of the regression function and each component in its decomposition under both the L 1 and L 2 penalties. Simulation studies and real examples show that the proposed methods perform well. Technical details and additional simulation results are available in online supplementary materials.

K-Nearest Neighbor Estimation of Functional Nonparametric Regression Model under NA Samples

Functional data, which provides information about curves, surfaces or anything else varying over a continuum, has become a commonly encountered type of data. The k-nearest neighbor (kNN) method, as a nonparametric method, has become one of the most popular supervised machine learning algorithms used to solve both classification and regression problems. This paper is devoted to the k-nearest neighbor (kNN) estimators of the nonparametric functional regression model when the observed variables take values from negatively associated (NA) sequences. The consistent and complete convergence rate for the proposed kNN estimator is first provided. Then, numerical assessments, including simulation study and real data analysis, are conducted to evaluate the performance of the proposed method and compare it with the standard nonparametric kernel approach.