Local Linear Smoothing Research Articles

Mean and covariance estimation for discretely observed high-dimensional functional data: Rates of convergence and division of observational regimes

Estimation of the mean and covariance parameters for functional data is a critical task, with local linear smoothing being a popular choice. In recent years, many scientific domains are producing multivariate functional data for which p, the number of curves per subject, is often much larger than the sample size n. In this setting of high-dimensional functional data, much of developed methodology relies on preliminary estimates of the unknown mean functions and the auto- and cross-covariance functions. This paper investigates the convergence rates of local linear estimators in terms of the maximal error across components and pairs of components for mean and covariance functions, respectively, in both L2 and uniform metrics. The local linear estimators utilize a generic weighting scheme that can adjust for differing numbers of discrete observations Nij across curves j and subjects i, where the Nij vary with n. Particular attention is given to the equal weight per observation (OBS) and equal weight per subject (SUBJ) weighting schemes. The theoretical results utilize novel applications of concentration inequalities for functional data and demonstrate that, similar to univariate functional data, the order of the Nij relative to p and n divides high-dimensional functional data into three regimes (sparse, dense, and ultra-dense), with the high-dimensional parametric convergence rate of log(p)/n1/2 being attainable in the latter two.

Generalized single index modeling of longitudinal data with multiple binary responses.

In health and clinical research, medical indices (eg, BMI) are commonly used for monitoring and/or predicting health outcomes of interest. While single-index modeling can be used to construct such indices, methods to use single-index models for analyzing longitudinal data with multiple correlated binary responses are underdeveloped, although there are abundant applications with such data (eg, prediction of multiple medical conditions based on longitudinally observed disease risk factors). This article aims to fill the gap by proposing a generalized single-index model that can incorporate multiple single indices and mixed effects for describing observed longitudinal data of multiple binary responses. Compared to the existing methods focusing on constructing marginal models for each response, the proposed method can make use of the correlation information in the observed data about different responses when estimating different single indices for predicting response variables. Estimation of the proposed model is achieved by using a local linear kernel smoothing procedure, together with methods designed specifically for estimating single-index models and traditional methods for estimating generalized linear mixed models. Numerical studies show that the proposed method is effective in various cases considered. It is also demonstrated using a dataset from the English Longitudinal Study of Aging project.

Acceleration and age in soccer

This paper considers how player acceleration changes in soccer relative to age. A plot of average maximum acceleration versus age is produced. The construction of the plot is based on methods from functional data analysis and the availability of tracking data from the 2019 season of the Chinese Super League. For an individual player, we calculate his maximum acceleration for each single match of the 2019 season. Since the players’ maximum accelerations are observed only on a single season instead of their entire careers, we treat them as incomplete functional data, called functional snippets. The average maximum acceleration, i.e., the mean function of the functional snippets rather than full curves is estimated by a local linear smoothing method. The most important observation is that the shape of the acceleration curve closely resembles curves of soccer performance versus age. This observation has implications for predicting future performance since acceleration is more easily and more accurately measured than performance.

Estimation for single-index spatial autoregressive model with covariate measurement errors

This paper explores the estimators of parameters for a spatial data single-index model which has measurement errors of covariates in the nonparametric part. The related estimations are considered to combine a local-linear smoother based simulation-extrapolation (SIMEX) algorithm, the estimation equation and the estimation method for profile maximum likelihood. Under regular conditions, asymptotic properties of the link function and uncertain estimators are derived. As verified in simulations, the performance of the estimators is satisfactory. Finally, an application to a real dataset is illustrated.

Functional-Coefficient Quantile Regression for Panel Data with Latent Group Structure

This article considers estimating functional-coefficient models in panel quantile regression with individual effects, allowing the cross-sectional and temporal dependence for large panel observations. A latent group structure is imposed on the heterogeneous quantile regression models so that the number of nonparametric functional coefficients to be estimated can be reduced considerably. With the preliminary local linear quantile estimates of the subject-specific functional coefficients, a classic agglomerative clustering algorithm is used to estimate the unknown group structure and an easy-to-implement ratio criterion is proposed to determine the group number. The estimated group number and structure are shown to be consistent. Furthermore, a post-grouping local linear smoothing method is introduced to estimate the group-specific functional coefficients, and the relevant asymptotic normal distribution theory is derived with a normalization rate comparable to that in the literature. The developed methodologies and theory are verified through a simulation study and showcased with an application to house price data from U.K. local authority districts, which reveals different homogeneity structures at different quantile levels.

Asymptotic normality of the local linear estimator of the functional expectile regression

We are concerned with the nonparametric estimation of the expectile functional regression. More precisely, we build an estimator, by the local linear smoothing approach, of the conditional expectile. Then we establish the asymptotic distribution of the constructed estimator. Establishing this result requires the Bahadur representation of the conditional expectile. The latter is obtained under certain standard conditions which cover the functional aspect of the data as well as the nonparametric characteristic of the model. The real impact of this result in nonparametric functional statistics has been discussed and highlighted using artificial data.

Incorporating covariate into mean and covariance function estimation of functional data under a general weighing scheme

This paper develops the estimation method of mean and covariance functions of functional data with additional covariate information. With the strength of both local linear smoothing modeling and general weighing scheme, we are able to explicitly characterize the mean and covariance functions with incorporating covariate for irregularly spaced and sparsely observed longitudinal data, as typically encountered in engineering technology or biomedical studies, as well as for functional data which are densely measured. Theoretically, we establish the uniform convergence rates of the estimators in the general weighing scheme. Monte Carlo simulation is conducted to investigate the finite-sample performance of the proposed approach. Two applications including the children growth data and white matter tract dataset obtained from Alzheimer's Disease Neuroimaging Initiative study are also provided.

Simultaneous confidence bands and global inferences for extended partially linear single-index models

Substantial efforts have been made for the popular semiparametric single-index models in the last two decades. Extended partially linear single-index models as generalized single-index semiparametric models effectively reduce the problem of model misspecification by allowing the data to automatically choose the principal linear component and the principal nonlinear component. In this paper, we construct an oracle-efficient simultaneous confidence band as a global inference tool for the extended partially linear single-index models. Specifically, we apply a LASSO penalized local linear smoothing method to estimate the coefficient parameters and conduct variable selection to avoid model overfitting caused by the two appearances of covariates. It is shown that the local linear estimator for the nonparametric link function by employing the penalized coefficient estimates is oracle-efficient in the sense that it is uniformly as efficient as the ideal one obtained by utilizing the true coefficient parameters. Then by applying the oracle efficiency and the extreme value distribution theory of the local linear regression, an asymptotically accurate simultaneous confidence band for the nonparametric link function in the extended partially linear single-index models is established. Simulation experiments with commonly encountered sample sizes corroborate our theoretical findings. The proposed method is applied to analyze two engineering datasets: motor trend car road tests dataset and concrete slump tests dataset.

Dynamic Principal Component Analysis in High Dimensions

Principal component analysis is a versatile tool to reduce dimensionality which has wide applications in statistics and machine learning. It is particularly useful for modeling data in high-dimensional scenarios where the number of variables p is comparable to, or much larger than the sample size n. Despite an extensive literature on this topic, researchers have focused on modeling static principal eigenvectors, which are not suitable for stochastic processes that are dynamic in nature. To characterize the change in the entire course of high-dimensional data collection, we propose a unified framework to directly estimate dynamic eigenvectors of covariance matrices. Specifically, we formulate an optimization problem by combining the local linear smoothing and regularization penalty together with the orthogonality constraint, which can be effectively solved by manifold optimization algorithms. We show that our method is suitable for high-dimensional data observed under both common and irregular designs, and theoretical properties of the estimators are investigated under l q ( 0 ≤ q ≤ 1 ) sparsity. Extensive experiments demonstrate the effectiveness of the proposed method in both simulated and real data examples.

Nonparametric regression for interval-valued data based on local linear smoothing approach

In this paper, we propose an interval local linear method (ILLM) to fit the regression model with interval-valued explanatory and response variables. The proposed method has no restriction on the form of the regression function. Moreover, it reduces the boundary effect of interval kernel method. Some experimental studies including two simulations and four real datasets are examined to evaluate the proposed method. Experimental results show that our method has higher predictive accuracy than some existed methods, including the center and range method, the interval least absolute method, the interval kernel method, multi-output support vector regression and the interval multilayer perceptron.

Intrinsic Riemannian functional data analysis for sparse longitudinal observations

A new framework is developed to intrinsically analyze sparsely observed Riemannian functional data. It features four innovative components: a frame-independent covariance function, a smooth vector bundle termed covariance vector bundle, a parallel transport and a smooth bundle metric on the covariance vector bundle. The introduced intrinsic covariance function links estimation of covariance structure to smoothing problems that involve raw covariance observations derived from sparsely observed Riemannian functional data, while the covariance vector bundle provides a rigorous mathematical foundation for formulating such smoothing problems. The parallel transport and the bundle metric together make it possible to measure fidelity of fit to the covariance function. They also play a critical role in quantifying the quality of estimators for the covariance function. As an illustration, based on the proposed framework, we develop a local linear smoothing estimator for the covariance function, analyze its theoretical properties and provide numerical demonstration via simulated and real data sets. The intrinsic feature of the framework makes it applicable to not only Euclidean submanifolds but also manifolds without a canonical ambient space.

Eigen-Adjusted Functional Principal Component Analysis

Functional Principal Component Analysis (FPCA) has become a widely used dimension reduction tool for functional data analysis. When additional covariates are available, existing FPCA models integrate them either in the mean function or in both the mean function and the covariance function. However, methods of the first kind are not suitable for data that display second-order variation, while those of the second kind are time-consuming and make it difficult to perform subsequent statistical analyses on the dimension-reduced representations. To tackle these issues, we introduce an eigen-adjusted FPCA model that integrates covariates in the covariance function only through its eigenvalues. In particular, different structures on the covariate-specific eigenvalues—corresponding to different practical problems—are discussed to illustrate the model’s flexibility as well as utility. To handle functional observations under different sampling schemes, we employ local linear smoothers to estimate the mean function and the pooled covariance function, and a weighted least square approach to estimate the covariate-specific eigenvalues. The convergence rates of the proposed estimators are further investigated under the different sampling schemes. In addition to simulation studies, the proposed model is applied to functional Magnetic Resonance Imaging scans, collected within the Human Connectome Project, for functional connectivity investigation. Supplementary materials for this article are available online.

Inference and Computation for Sparsely Sampled Random Surfaces

Nonparametric inference for functional data over two-dimensional domains entails additional computational and statistical challenges, compared to the one-dimensional case. Separability of the covariance is commonly assumed to address these issues in the densely observed regime. Instead, we consider the sparse regime, where the latent surfaces are observed only at few irregular locations with additive measurement error, and propose an estimator of covariance based on local linear smoothers. Consequently, the assumption of separability reduces the intrinsically four-dimensional smoothing problem into several two-dimensional smoothers and allows the proposed estimator to retain the classical minimax-optimal convergence rate for two-dimensional smoothers. Even when separability fails to hold, imposing it can be still advantageous as a form of regularization. A simulation study reveals a favorable bias-variance tradeoff and massive speed-ups achieved by our approach. Finally, the proposed methodology is used for qualitative analysis of implied volatility surfaces corresponding to call options, and for prediction of the latent surfaces based on information from the entire dataset, allowing for uncertainty quantification. Our cross-validated out-of-sample quantitative results show that the proposed methodology outperforms the common approach of pre-smoothing every implied volatility surface separately. Supplementary materials for this article are available online.

A Novel Estimation Method in Generalized Single Index Models

The single index and generalized single index models have been demonstrated to be a powerful tool for studying nonlinear interaction effects of variables in the low-dimensional case. In this article, we propose a new estimation approach for generalized single index models E ( Y | θ ⊤ X ) = ψ ( g ( θ ⊤ X ) ) with ψ ( · ) known but g ( · ) unknown. Specifically, we first obtain a consistent estimator of the regression function by using a local linear smoother, and then estimate the parametric components by treating ψ ( g ̂ ( θ ⊤ X i ) ) as our continuous response. The resulting estimators of θ are asymptotically normal. The proposed procedure can substantially overcome convergence problems encountered in generalized linear models with discrete response variables when sparseness occurs and misspecification. We conduct simulation experiments to evaluate the numerical performance of the proposed methods and analyze a financial dataset from a peer-to-peer lending platform of China as an illustration.

Unified statistical inference for a nonlinear dynamic functional/longitudinal data model

Studying massive functional/longitudinal data, we adopt a flexible nonlinear dynamic regression method named the Semi-Varying Coefficient Additive Model, in which the response can be a functional/longitudinal variable, and the explanatory variables can be a mixture of functional/longitudinal and scalar variables. With the aid of an initial B-spline approximation, a local linear smoothing is proposed to estimate the unknown functional effects in the model. Existing methods of statistical inference for sparse data and dense data are significantly different. We therefore develop the asymptotic theories of the resultant pilot estimation based local linear estimators (PEBLLE) on a unified framework of sparse, dense and ultra-dense cases of data. Remarkably, we obtain the oracle properties as if other functions were known in advance. Extensive Monte Carlo simulation studies investigating the finite sample performance of the proposed methodologies confirm our asymptotic results. We further illustrate our methodologies by analyzing COVID-19 data from China.

An Outer-Product-of-Gradient Approach to Dimension Reduction and its Application to Classification in High Dimensional Space

Sufficient dimension reduction (SDR) has progressed steadily. However, its ability to improve general function estimation or classification has not been well received, especially for high-dimensional data. In this article, we first devise a local linear smoother for high dimensional nonparametric regression and then utilise it in the outer-product-of-gradient (OPG) approach of SDR. We call the method high-dimensional OPG (HOPG). To apply SDR to classification in high-dimensional data, we propose an ensemble classifier by aggregating results of classifiers that are built on subspaces reduced by the random projection and HOPG consecutively from the data. Asymptotic results for both HOPG and the classifier are established. Superior performance over the existing methods is demonstrated in simulations and real data analyses. Supplementary materials for this article are available online.

Simultaneous variable selection and structural identification for time‐varying coefficient models

Time‐varying coefficient models are important tools in time series analysis due to their flexibility to fit non‐stationary data. To improve the accuracy of these models, it is important to identify covariates with null, constant and time‐varying effects and to estimate their coefficients. This article proposes a combination of the local linear smoothing method and the adaptive group lasso penalty approach to achieve covariate identification and coefficient estimation. The penalty term consists of two parts. The first term penalizes the norm of the coefficient function, which is used to select relevant variables. The second term penalizes the norm of the derivative function, which assesses the constancy of the coefficient functions. The asymptotic properties of the proposed methodology are established. Performance of the proposed method is demonstrated using simulated data along with an application to the analysis of the air quality and health data in Hong Kong.

Time-varying additive model with autoregressive errors for locally stationary time series

In this article, we study the time-varying additive model with time-varying autoregression (tvAR) error in the locally stationary context, and propose the two-step estimation for it. B-spline method, which is computation efficient, is adopted to obtain the initial estimator of trend function and additive components. And then the structure of autoregression error is estimated by ULASSO, the consistency and asymptotical normality are proved. At last, with the initial estimator and the estimated error structure, the improved estimator of trend function and additive components is derived by local linear smoothing, and its asymptotic normality and oracle property are proved. Simulation studies validate the properties of the proposed estimators. A real data application illustrates the proposed model is applicable and more appropriate than the classical additive model in the presence of locally stationary regressors.

Statistical estimation for single-index varying-coefficient models with multiplicative distortion measurement errors

During the observation, various factors can cause variables to be contaminated by measurement errors. In this paper, we consider the estimation for single-index varying-coefficient models where both the response variable and covariates are measured with multiplicative distortion measurement errors. First, we use the conditional absolute mean calibration to calibrate the distorted variables. Based on the calibrated variables, we obtain the estimators of the parameters and coefficient functions by local linear smoothing and estimating equations. Furthermore, we prove that the obtained estimators have asymptotic normality and the estimator of parametric component is still root-n consistent. Finally, two simulation examples and the analysis of body fat data are given to illustrate that the proposed estimators are unbiased, robust and practical.

Computational aspects of the kNN local linear smoothing for some conditional models in high dimensional statistics

We consider the problem of nonparametric estimation of certain conditional models when the explanatory variable is functional. The models studied are the Conditional Expectation (CE), the Conditional Distribution Function (CDF) and the Conditional Probability Density (CPD). The estimators are constructed by combining the local linear method and the k-Nearest Neighbors (kNN) smoothing approach. For each of the three models, we define an optimal estimator associated with the best number of neighbors chosen using two bandwidth selection procedures which are the cross-validation criterion and the bootstrap smoothing rule. The asymptotic properties of these optimal kNN Local Linear Estimators ( NN- LLE) are established under some standard conditions. The performances of the finite samples of these estimators are then examined and compared through a Monte-carlo study. In addition, an application on the riboflavin content in the yogurt using the Near-infrared curves is carried out to demonstrate the usefulness of the models proposed in the point-wise prediction as well as for the predictive interval.