Nonparametric Function Estimation Research Articles

A Locally Adaptive Penalty for Estimation of Functions With Varying Roughness

We propose a new regularization method called Loco-Spline for nonparametric function estimation. Loco-Spline uses a penalty which is data driven and locally adaptive. This allows for more flexible estimation of the function in regions of the domain where it has more curvature, without over fitting in regions that have little curvature. This methodology is also transferred into higher dimensions via the Smoothing Spline ANOVA framework. General conditions for optimal MSE rate of convergence are given and the Loco-Spline is shown to achieve this rate. In our simulation study, the Loco-Spline substantially outperforms the traditional smoothing spline and the locally adaptive kernel smoother. Code to fit Loco-Spline models is included with the Supplemental Materials for this article which are available online.

Tilted Nonparametric Estimation of Volatility Functions with Empirical Applications, 2nd Version

This paper proposes a novel positive nonparametric estimator of the conditional variance function without reliance on logarithmic or other transformations. The estimator is based on an empirical likelihood modification of conventional local level nonparametric regression applied to squared mean regression residuals. The estimator is shown to be asymptotically equivalent to the local linear estimator in the case of unbounded support but, unlike that estimator, is restricted to be non-negative in finite samples. It is fully adaptive to the unknown conditional mean function. Simulations are conducted to evaluate the finite sample performance of the estimator. Two empirical applications are reported. One uses cross section data and studies the relationship between occupational prestige and income. The other uses time series data on Treasury bill rates to fit the total volatility function in a continuous-time jump diffusion model.

ROBUST LOCATION AND SPREAD MEASURES FOR NONPARAMETRIC PROBABILITY DENSITY FUNCTION ESTIMATION

Robustness against outliers is a desirable property of any unsupervised learning scheme. In particular, probability density estimators benefit from incorporating this feature. A possible strategy to achieve this goal is to substitute the sample mean and the sample covariance matrix by more robust location and spread estimators. Here we use the L1-median to develop a nonparametric probability density function (PDF) estimator. We prove its most relevant properties, and we show its performance in density estimation and classification applications.

Sharp adaptive estimation by a blockwise method

We consider a blockwise James–Stein estimator for nonparametric function estimation in suitable wavelet or Fourier bases. The estimator can be readily explained and implemented. We show that the estimator is asymptotically sharp adaptive in minimax risk over any Sobolev ball containing the true function. Further, for a moderately broad range of bounded sets in the Besov space our estimator is asymptotically nearly sharp adaptive in the sense that it comes within the Donoho–Liu constant, 1.24, of being exactly sharp adaptive. Other parameter spaces are also considered. The paper concludes with a Monte-Carlo study comparing the performance of our estimator with that of three other popular wavelet estimators. Our procedure generally (but not always) outperforms two of these and is overall comparable, or perhaps slightly superior, with the third.

Nonlinearity Recovering in Hammerstein System from Short Measurement Sequence

The problem of data pre-filtering for nonparametric identification of Hammerstein system from short (finite) data set is considered. The two-stage method is proposed. First, the linear dynamic block is identified using instrumental variables technique, and the inverse of the obtained model is used for output filtering. Next, the standard procedure of nonparametric regression function estimation (kernel-based, or using orthogonal series expansion) is applied, involving the filtered output sequence instead of the original one. It is shown, that for small and moderate number of data, the estimation error can be significantly reduced in comparison with standard nonparametric methods. The asymptotic properties of the method (consistency and rate of convergence) remain the same as in the classical versions of nonparametric algorithms.

A semi‐parametric threshold regression analysis of sexually transmitted infections in adolescent women

Time-to-event analysis of sexually transmitted infection data is often complicated by the existence of nonproportional hazards and nonlinear independent variable effects. Methods without the proportional hazards assumption, such as threshold regression models, have been successfully used in many applications. This paper seeks to extend the existing threshold regression models to accommodate the nonlinear independent variable effects. Specifically, we incorporated penalized and regression splines to the threshold regression models for added modeling flexibility. Cross validation methods were used for the selection of the number of knots and for the determination of smoothing parameters. Variance estimates were proposed for inference purposes. Simulation results showed that the proposed methods were able to achieve nonparametric function and parametric coefficient estimates that are close to their true values. Simulation also demonstrated satisfactory performance of variance estimates. Using the proposed methods, we analyzed time from sexual debut to the first infection with Chlamydia trachomatis infection in a group of young women. Analysis shows that the lifetime number of sexual partners has a nonlinear effect on the risk of C. trachomatis infection and the infection risks were differential by ethnicity and age of sexual debut.

Nonparametric regression with weakly dependent data: the discrete and continuous regressor case

Data-driven methods of bandwidth selection are necessary for the sound application of kernel methods, with benefits including but not limited to automatic dimensionality reduction in the presence of irrelevant regressors [P. Hall, Q. Li, and J.S. Racine, ‘Nonparametric estimation of regression functions in the presence of irrelevant regressors, Rev. Econ. Statist. 89 (2007), pp. 784–789] and the ability to handle the mix of discrete and continuous data often encountered in applied settings without resorting to sample splitting [J.S. Racine and Q. Li, Nonparametric estimation of regression functions with both categorical and continuous data, J. Econometrics 119(1) (2004), pp. 99–130]. Many existing results have been developed under the presumption of independence, which may not hold when one deals with time-series data. This paper develops the properties of data-driven kernel regression for weakly dependent mixed discrete and continuous data. Monte Carlo simulations are undertaken to examine the finite-sample properties of the estimator, and an illustrative application is presented.

Nonparametric inference and uniqueness for periodically observed progressive disease models

In many studies examining the progression of HIV and other chronic diseases, subjects are periodically monitored to assess their progression through disease states. This gives rise to a specific type of panel data which have been termed "chain-of-events data"; e.g. data that result from periodic observation of a progressive disease process whose states occur in a prescribed order and where state transitions are not observable. Using a discrete time semi-Markov model, we develop an algorithm for nonparametric estimation of the distribution functions of sojourn times in a J state progressive disease model. Issues of uniqueness for chain-of-events data are not well-understood. Thus, a main goal of this paper is to determine the uniqueness of the nonparametric estimators of the distribution functions of sojourn times within states. We develop sufficient conditions for uniqueness of the nonparametric maximum likelihood estimator, including situations where some but not all of its components are unique. We illustrate the methods with three examples.

Reliability Nonparametric Bayesian Estimation in Parallel Systems

Relevant results for (sub-)distribution functions related to parallel systems are discussed. The reverse hazard rate is defined using the product integral. Consequently, the restriction of absolute continuity for the involved distributions can be relaxed. The only restriction is that the sets of discontinuity points of the parallel distributions have to be disjointed. Nonparametric Bayesian estimators of all survival (sub-)distribution functions are derived. Dual to the series systems that use minimum life times as observations, the parallel systems record the maximum life times. Dirichlet multivariate processes forming a class of prior distributions are considered for the nonparametric Bayesian estimation of the component distribution functions, and the system reliability. For illustration, two striking numerical examples are presented.

Two-Step Extremum Estimation with Estimated Single-Indices

This paper studies two-step extremum estimation that involves the first step estimation of nonparametric functions of single-indices. First, this paper finds that under certain regularity conditions for conditional measures, linear functionals of conditional expectations are insensitive to the first order perturbation of the parameters in the conditioning variable. Applying this result to symmetrized nearest neighborhood estimation of the nonparametric functions, this paper shows that the influence of the estimated single-indices on the estimator of main interest is asymptotically negligible even when the estimated single-indices follow cube root asymptotics. As a practical use of this finding, this paper proposes a bootstrap method for conditional moment restrictions that are asymptotically valid in the presence of cube root-converging single-index estimators. Some results from Monte Carlo simulations are presented and discussed.

Nonparametric Functional Mapping of Quantitative Trait Loci

Functional mapping is a useful tool for mapping quantitative trait loci (QTL) that control dynamic traits. It incorporates mathematical aspects of biological processes into the mixture model-based likelihood setting for QTL mapping, thus increasing the power of QTL detection and the precision of parameter estimation. However, in many situations there is no obvious functional form and, in such cases, this strategy will not be optimal. Here we propose to use nonparametric function estimation, typically implemented with B-splines, to estimate the underlying functional form of phenotypic trajectories, and then construct a nonparametric test to find evidence of existing QTL. Using the representation of a nonparametric regression as a mixed model, the final test statistic is a likelihood ratio test. We consider two types of genetic maps: dense maps and general maps, and the power of nonparametric functional mapping is investigated through simulation studies and demonstrated by examples.

Bayesian Nonparametric Estimation of Continuous Monotone Functions with Applications to Dose–Response Analysis

In this article, we consider monotone nonparametric regression in a Bayesian framework. The monotone function is modeled as a mixture of shifted and scaled parametric probability distribution functions, and a general random probability measure is assumed as the prior for the mixing distribution. We investigate the choice of the underlying parametric distribution function and find that the two-sided power distribution function is well suited both from a computational and mathematical point of view. The model is motivated by traditional nonlinear models for dose-response analysis, and provides possibilities to elicitate informative prior distributions on different aspects of the curve. The method is compared with other recent approaches to monotone nonparametric regression in a simulation study and is illustrated on a data set from dose-response analysis.

Recurrent nonparametric estimation of functions from functionals of multidimensional density and their derivatives

Recurrent kernel estimates of functions are considered, which depend on functionals of the multidimensional distribution density and their derivatives, and also their piecewise smooth approximations. The principal part of a mean square error of estimates is found. The suggested approach enables estimating from unique positions, for example, the production function and its characteristics. The comparison is performed of recurrent and common estimates both in asymptotics and at finite volumes of samples.

Hammerstein system identification by non-parametric instrumental variables

A mixed, parametric–non-parametric routine for Hammerstein system identification is presented. Parameters of a non-linear characteristic and of ARMA linear dynamical part of Hammerstein system are estimated by least squares and instrumental variables assuming poor a priori knowledge about the random input and random noise. Both subsystems are identified separately, thanks to the fact that the unmeasurable interaction inputs and suitable instrumental variables are estimated in a preliminary step by the use of a non-parametric regression function estimation method. A wide class of non-linear characteristics including functions which are not linear in the parameters is admitted. It is shown that the resulting estimates of system parameters are consistent for both white and coloured noise. The problem of generating optimal instruments is discussed and proper non-parametric method of computing the best instrumental variables is proposed. The analytical findings are validated using numerical simulation results.

Minimum distance regression model checking with Berkson measurement errors

Lack-of-fit testing of a regression model with Berkson measurement error has not been discussed in the literature to date. To fill this void, we propose a class of tests based on minimized integrated square distances between a nonparametric regression function estimator and the parametric model being fitted. We prove asymptotic normality of these test statistics under the null hypothesis and that of the corresponding minimum distance estimators under minimal conditions on the model being fitted. We also prove consistency of the proposed tests against a class of fixed alternatives and obtain their asymptotic power against a class of local alternatives orthogonal to the null hypothesis. These latter results are new even when there is no measurement error. A simulation that is included shows very desirable finite sample behavior of the proposed inference procedures.

Accelerating Machine-Learning Algorithms on FPGAs using Pattern-Based Decomposition

Machine-learning algorithms are employed in a wide variety of applications to extract useful information from data sets, and many are known to suffer from super-linear increases in computational time with increasing data size and number of signals being processed (data dimension). Certain principal machine-learning algorithms are commonly found embedded in larger detection, estimation, or classification operations. Three such principal algorithms are the Parzen window-based, non-parametric estimation of Probability Density Functions (PDFs), K-means clustering and correlation. Because they form an integral part of numerous machine-learning applications, fast and efficient execution of these algorithms is extremely desirable. FPGA-based reconfigurable computing (RC) has been successfully used to accelerate computationally intensive problems in a wide variety of scientific domains to achieve speedup over traditional software implementations. However, this potential benefit is quite often not fully realized because creating efficient FPGA designs is generally carried out in a laborious, case-specific manner requiring a great amount of redundant time and effort. In this paper, an approach using pattern-based decomposition for algorithm acceleration on FPGAs is proposed that offers significant increases in productivity via design reusability. Using this approach, we design, analyze, and implement a multi-dimensional PDF estimation algorithm using Gaussian kernels on FPGAs. First, the algorithm's amenability to a hardware paradigm and expected speedups are predicted. After implementation, actual speedup and performance metrics are compared to the predictions, showing speedup on the order of 20× over a 3.2 GHz processor. Multi-core architectures are developed to further improve performance by scaling the design. Portability of the hardware design across multiple FPGA platforms is also analyzed. After implementing the PDF algorithm, the value of pattern-based decomposition to support reuse is demonstrated by rapid development of the K-means and correlation algorithms.

Bayesian Nonparametric Model for the Validation of Peptide Identification in Shotgun Proteomics

Tandem mass spectrometry combined with database searching allows high throughput identification of peptides in shotgun proteomics. However, validating database search results, a problem with a lot of solutions proposed, is still advancing in some aspects, such as the sensitivity, specificity, and generalizability of the validation algorithms. Here a Bayesian nonparametric (BNP) model for the validation of database search results was developed that incorporates several popular techniques in statistical learning, including the compression of feature space with a linear discriminant function, the flexible nonparametric probability density function estimation for the variable probability structure in complex problem, and the Bayesian method to calculate the posterior probability. Importantly the BNP model is compatible with the popular target-decoy database search strategy naturally. We tested the BNP model on standard proteins and real, complex sample data sets from multiple MS platforms and compared it with Peptide-Prophet, the cutoff-based method, and a simple nonparametric method (proposed by us previously). The performance of the BNP model was shown to be superior for all data sets searched on sensitivity and generalizability. Some high quality matches that had been filtered out by other methods were detected and assigned with high probability by the BNP model. Thus, the BNP model could be able to validate the database search results effectively and extract more information from MS/MS data.

On the asymptotics of marginal regression splines with longitudinal data

SUMMARY There have been studies on how the asymptotic efficiency of a nonparametric function estimator depends on the handling of the within-cluster correlation when nonparametric regression models are used on longitudinal or cluster data. In particular, methods based on smoothing splines and local polynomial kernels exhibit different behaviour. We show that the generalized estimation equations based on weighted least squares regression splines for the nonparametric function have an interesting property: the asymptotic bias of the estimator does not depend on the working correlation matrix, but the asymptotic variance, and therefore the mean squared error, is minimized when the true correlation structure is specified. This property of the asymptotic bias distinguishes regression splines from smoothing splines.

Nonparametric Estimation of Mean and Dispersion Functions in Extended Generalized Linear Models

In this paper the interest is in regression analysis for data that show possibly overdispersion or underdispersion. The starting point for modeling are generalized linear models in which we no longer admit a linear form for the mean regression function, but allow it to be any smooth function of the covariate(s). In view of analyzing overdispersed or underdispersed data, we additionally bring in an unknown dispersion function. The mean regression function and the dispersion function are then estimated using P-splines with difference type of penalty to prevent from overfitting.We discuss two approaches: one based on an extended quasi-likelihood idea and one based on a pseudo-likelihood approach. The choices of smoothing parameters and implementation issues are discussed. The performance of the estimation method is investigated via simulations and its use is illustrated on several data, including continuous data, counts and proportions.

A Bayesian Approach to Non-Parametric Monotone Function Estimation

SummaryThe paper proposes two Bayesian approaches to non-parametric monotone function estimation. The first approach uses a hierarchical Bayes framework and a characterization of smooth monotone functions given by Ramsay that allows unconstrained estimation. The second approach uses a Bayesian regression spline model of Smith and Kohn with a mixture distribution of constrained normal distributions as the prior for the regression coefficients to ensure the monotonicity of the resulting function estimate. The small sample properties of the two function estimators across a range of functions are provided via simulation and compared with existing methods. Asymptotic results are also given that show that Bayesian methods provide consistent function estimators for a large class of smooth functions. An example is provided involving economic demand functions that illustrates the application of the constrained regression spline estimator in the context of a multiple-regression model where two functions are constrained to be monotone.