Nonparametric Estimation Research Articles

The survival function NPMLE for combined right-censored and length-biased right-censored failure time data: properties and applications.

Many cohort studies in survival analysis have imbedded in them subcohorts consisting of incident cases and prevalent cases. Instead of analysing the data from the incident and prevalent cohorts alone, there are surely advantages to combining the data from these two subcohorts. In this paper, we discuss a survival function nonparametric maximum likelihood estimator (NPMLE) using both length-biased right-censored prevalent cohort data and right-censored incident cohort data. We establish the asymptotic properties of the survival function NPMLE and utilize the NPMLE to estimate the distribution for time spent in a Montreal area hospital.

Identification and estimation of panel semiparametric conditional heteroskedastic frontiers with dynamic inefficiency

We study a semiparametric panel stochastic frontier model with nonseparable unobserved heterogeneity, which allows for time-varying conditional heteroskedastic productivity components. It does not require distributional assumptions on random noise except conditional symmetry. We utilize conditional characteristic functions from Kotlarski’s Lemma to derive new moment conditions that yield the identification of the heteroskedastic variance parameters of inefficiency and random noise. Identification only requires a panel with three periods for serially correlated inefficiency. A nonparametric estimation procedure is also developed for the conditional variance of inefficiency, and its convergence rate is established. Monte Carlo simulation shows that the estimator is robust to misspecification of inefficiency distributions.

Nonparametric statistical inference for stochastic optimal control problems and its applications for financial investment

Against a backdrop of financial decision-making, this study examines nonparametric statistical inference problems in stochastic optimal control problems. First, we construct a nonparametric estimation model using the wavelet estimation method for drift and diffusion coefficients. In this study, we focus on stochastic linear–quadratic (SLQ) problems, solved using Riccati equations, and apply this to a cash management problem within the linear–quadratic framework. The results show the effectiveness of the wavelet method using a numerical case.

A hybrid nonparametric multivariate density estimator with applications to risk management

. Multivariate density estimation is plagued by the curse of dimensionality in theory and practice. We propose a hybrid density estimator of a multivariate density f that combines the strengths of the kernel estimator and the exponential series estimator. This estimator refines a preliminary kernel estimate f ̂ 0 with a multiplicative correction that estimates the ratio r = f / f ̂ 0 with an exponential series estimator. Thanks to the consistency of the pilot estimate, the coefficients of the series expansion tend to approach zero asymptotically. Accordingly, we design a thresholding method for basis function selection. A major obstacle of multivariate exponential series estimator is the calculation of its normalization factor. We resolve this difficulty with Monte Carlo integration, using the pilot kernel estimate as the trial density for importance sampling. This approach greatly enhances the practicality of the hybrid estimator. Numerical simulations demonstrate the good finite sample performance of the hybrid estimator. We present one empirical application in financial risk management.

Practical causal mediation analysis: extending nonparametric estimators to accommodate multiple mediators and multiple intermediate confounders.

Mediation analysis is appealing for its ability to improve understanding of the mechanistic drivers of causal effects, but real-world data complexities challenge its successful implementation, including (i) the existence of post-exposure variables that also affect mediators and outcomes (thus, confounding the mediator-outcome relationship), that may also be (ii) multivariate, and (iii) the existence of multivariate mediators. All three challenges are present in the mediation analysis we consider here, where our goal is to estimate the indirect effects of receiving a Section 8 housing voucher as a young child on the risk of developing a psychiatric mood disorder in adolescence that operate through mediators related to neighborhood poverty, the school environment, and instability of the neighborhood and school environments, considered together and separately. Interventional direct and indirect effects (IDE/IIE) accommodate post-exposure variables that confound the mediator-outcome relationship, but currently, no readily implementable nonparametric estimator for IDE/IIE exists that allows for both multivariate mediators and multivariate post-exposure intermediate confounders. The absence of such an IDE/IIE estimator that can easily accommodate both multivariate mediators and post-exposure confounders represents a significant limitation for real-world analyses, because when considering each mediator subgroup separately, the remaining mediator subgroups (or a subset of them) become post-exposure intermediate confounders. We address this gap by extending a recently developed nonparametric estimator for the IDE/IIE to allow for easy incorporation of multivariate mediators and multivariate post-exposure confounders simultaneously. We apply the proposed estimation approach to our analysis, including walking through a strategy to account for other, possibly co-occurring intermediate variables when considering each mediator subgroup separately.

Data variability in the imputation quality of missing data

Imputation methods were developed to define estimates for missing data and hence solve possible problems generated by the loss of this information. This study aims to assess whether data variability influences the results obtained after applying an imputation method. Incomplete databases were generated from complete real databases of experiments of tomato plants conducted using the randomized block design with three replications and 12 treatments by removing different amounts of data. The evaluated variables consisted of fruit weight per plant, number of fruits per plant, and average fruit length and width, forming eight balanced databases. Subsequently, the distribution-free multiple imputation method was applied, generating complete databases from imputation. The number of missing information influenced the accuracy measures for the data in this study. Data imputation was inadequate when there was high variability but more precise and accurate in cases of low variability. It confirmed the importance of assessing data variability before choosing to apply the imputation method.

Intraday volatility patterns from short-dated options

We propose a nonparametric estimator for the deterministic periodic component of volatility from short-dated options within an in-fill asymptotic setting. The estimator uses options with zero and one day to expiration sampled at high-frequency during a trading day. At each point in time, we aggregate the options to form nonparametric estimates of conditional risk-neutral expectations of future integrated return variation for the two available option tenors. A suitable ratio of these estimates removes the stochastic components of the conditional expectations of future volatility, up to asymptotically higher-order terms, and allows to form estimates of the deterministic periodic component of volatility. We derive a Central Limit Theorem for the estimator, with its rate of convergence determined from the mesh of the strike grid and the length of the time to expiration of the options. The newly-developed estimation procedure is applied to S&P 500 index options data.

A framework to select tuning parameters for nonparametric derivative estimation.

In this paper, we propose a general framework to select tuning parameters for the nonparametric derivative estimation. The new framework broadens the scope of the previously proposed generalized criterion by replacing the empirical derivative with any other linear nonparametric smoother. We provide the theoretical support of the proposed derivative estimation in a random design and justify it through simulation studies. The practical application of the proposed framework is demonstrated in the study of the age effect on hippocampal gray matter volume in healthy adults from the IXI dataset and the study of the effect of age and body mass index on blood pressure from the Pima Indiansdataset.

Particle filtering supported probability density estimation of mobility patterns

This paper presents a methodology that aims to enhance the accuracy of probability density estimation in mobility pattern analysis by integrating prior knowledge of system dynamics and contextual information into the particle filter algorithm. The quality of the data used for density estimation is often inadequate due to measurement noise, which significantly influences the distribution of the measurement data. Thus, it is crucial to augment the information content of the input data by incorporating additional sources of information beyond the measured position data. These other sources can include the dynamic model of movement and the spatial model of the environment, which influences motion patterns. To effectively combine the information provided by positional measurements with system and environment models, the particle filter algorithm is employed, which generates discrete probability distributions. By subjecting these discrete distributions to exploratory techniques, it becomes possible to extract more certain information compared to using raw measurement data alone. Consequently, this study proposes a methodology in which probability density estimation is not solely based on raw positional data but rather on probability-weighted samples generated through the particle filter. This approach yields more compact and precise modeling distributions. Specifically, the method is applied to process position measurement data using a nonparametric density estimator known as kernel density estimation. The proposed methodology is thoroughly tested and validated using information-theoretic and probability metrics. The applicability of the methodology is demonstrated through a practical example of mobility pattern analysis based on forklift data in a warehouse environment.

A novel nonparametric time-dependent precision-recall curve estimator for right-censored survival data.

In order to assess prognostic risk for individuals in precision health research, risk prediction models are increasingly used, in which statistical models are used to estimate the risk of future outcomes based on clinical and nonclinical characteristics. The predictive accuracy of a risk score must be assessed before it can be used in routine clinical decision making, where the receiver operator characteristic curves, precision-recall curves, and their corresponding area under the curves are commonly used metrics to evaluate the discriminatory ability of a continuous risk score. Among these the precision-recall curves have been shown to be more informative when dealing with unbalanced biomarker distribution between classes, which is common in rare event, even though except one, all existing methods are proposed for classic uncensored data. This paper is therefore to propose a novel nonparametric estimation approach for the time-dependent precision-recall curve and its associated area under the curve for right-censored data. A simulation is conducted to show the better finite sample property of the proposed estimator over the existing method and a real-world data from primary biliary cirrhosis trial is used to demonstrate the practical applicability of the proposedestimator.

Spectral density estimation for random processes with stationary increments

AbstractSpectral density analysis plays an important role in studying a stationary random process on a real line. In this paper, we extend this discussion for the random process with stationary increments. We investigate the properties of the method of moments structure function estimation, and propose a nonparametric spectral density function estimator. Our numerical results show that the proposed spectral density estimator performs comparable with the parametric counterpart when the underlying process is assumed to be band‐limited. Additionally, this method is applied to analyze US Housing Starts Data, where the hidden periodicities are detected, providing consistent conclusions with previous economic studies.

Fast Rates for the Regret of Offline Reinforcement Learning

We study the regret of offline reinforcement learning in an infinite-horizon discounted Markov decision process (MDP). While existing analyses of common approaches, such as fitted Q-iteration (FQI), suggest root-n convergence for regret, empirical behavior exhibits much faster convergence. In this paper, we present a finer regret analysis that exactly characterizes this phenomenon by providing fast rates for the regret convergence. First, we show that given any estimate for the optimal quality function, the regret of the policy it defines converges at a rate given by the exponentiation of the estimate’s pointwise convergence rate, thus speeding up the rate. The level of exponentiation depends on the level of noise in the decision-making problem, rather than the estimation problem. We establish such noise levels for linear and tabular MDPs as examples. Second, we provide new analyses of FQI and Bellman residual minimization to establish the correct pointwise convergence guarantees. As specific cases, our results imply one-over-n rates in linear cases and exponential-in-n rates in tabular cases. We extend our findings to general function approximation by extending our results to regret guarantees based on Lp-convergence rates for estimating the optimal quality function rather than pointwise rates, where L2 guarantees for nonparametric estimation can be ensured under mild conditions. Funding: This work was supported by the Division of Information and Intelligent Systems, National Science Foundation [Grant 1846210].

Statistical Spatially Inhomogeneous Diffusion Inference

Inferring a diffusion equation from discretely observed measurements is a statistical challenge of significant importance in a variety of fields, from single-molecule tracking in biophysical systems to modeling financial instruments. Assuming that the underlying dynamical process obeys a d-dimensional stochastic differential equation of the form dx_t = b(x_t)dt + \Sigma(x_t)dw_t, we propose neural network-based estimators of both the drift b and the spatially-inhomogeneous diffusion tensor D = \Sigma\Sigma^T/2 and provide statistical convergence guarantees when b and D are s-Hölder continuous. Notably, our bound aligns with the minimax optimal rate N^{-\frac{2s}{2s+d}} for nonparametric function estimation even in the presence of correlation within observational data, which necessitates careful handling when establishing fast-rate generalization bounds. Our theoretical results are bolstered by numerical experiments demonstrating accurate inference of spatially-inhomogeneous diffusion tensors.

Regression estimation for continuous-time functional data processes with missing at random response

In this paper, we are interested in nonparametric kernel estimation of a generalised regression function based on an incomplete sample ( X t , Y t , ζ t ) t ∈ [ 0 , T ] copies of a continuous-time stationary and ergodic process ( X , Y , ζ ) . The predictor X is valued in some infinite-dimensional space, whereas the real-valued process Y is observed when the Bernoulli process ζ = 1 and missing whenever ζ = 0 . Uniform almost sure consistency rate as well as the evaluation of the conditional bias and asymptotic mean square error are established. The asymptotic distribution of the estimator is provided with a discussion on its use in building asymptotic confidence intervals. To illustrate the performance of the proposed estimator, a first simulation is performed to compare the efficiency of discrete-time and continuous-time estimators. A second simulation is conducted to discuss the selection of the optimal sampling mesh in the continuous-time case. Then, a third simulation is considered to build asymptotic confidence intervals. An application to financial time series is used to study the performance of the proposed estimator in terms of point and interval prediction of the IBM asset price log-returns. Finally, a second application is introduced to discuss the usage of the initial estimator to impute missing household-level peak electricity demand.

On the role of Volterra integral equations in self-consistent, product-limit, inverse probability of censoring weighted, and redistribution-to-the-right estimators for the survival function.

This paper reconsiders several results of historical and current importance to nonparametric estimation of the survival distribution for failure in the presence of right-censored observation times, demonstrating in particular how Volterra integral equations help inter-connect the resulting estimators. The paper begins by considering Efron's self-consistency equation, introduced in a seminal 1967 Berkeley symposium paper. Novel insights provided in the current work include the observations that (i) the self-consistency equation leads directly to an anticipating Volterra integral equation whose solution is given by a product-limit estimator for the censoring survival function; (ii) a definition used in this argument immediately establishes the familiar product-limit estimator for the failure survival function; (iii) the usual Volterra integral equation for the product-limit estimator of the failure survival function leads to an immediate and simple proof that it can be represented as an inverse probability of censoring weighted estimator; (iv) a simple identity characterizes the relationship between natural inverse probability of censoring weighted estimators for the survival and distribution functions of failure; (v) the resulting inverse probability of censoring weighted estimators, attributed to a highly influential 1992 paper of Robins and Rotnitzky, were implicitly introduced in Efron's 1967 paper in its development of the redistribution-to-the-right algorithm. All results developed herein allow for ties between failure and/or censored observations.

A data-driven newsvendor model for elective-emergency admission control under uncertain inpatient bed capacity.

Elective-emergency admission control referred to allocating available inpatient bed capacity between elective and emergency hospitalization demand. Existing approaches for admission control often excluded several complex factors when making decisions, such as uncertain bed capacity and unknown true probability distributions of patient arrivals and departures. We aimed to create a data-driven newsvendor framework to study the elective-emergency admission control problem to achieve bed operational efficiency and effectiveness. We developed a data-driven approach that utilized the newsvendor framework to formulate the admission control problem. We also created approximation algorithms to generate a pool of candidate admission control solutions. Past observations and relevant emergency demand and bed capacity features were modeled in a newsvendor framework. Using approximation algorithmic approaches (sample average approximation, separated estimation and optimization, linear programing-LP, and distribution-free model) allowed us to derive computationally efficient data-driven solutions with tight bounds on the expected in-sample and out-of-sample cost guaranteed. Tight generalization bounds on the expected out-of-sample cost of the feature-based model were derived with respect to the LP and quadratic programing (QP) algorithms, respectively. Results showed that the optimal feature-based model outperformed the optimal observation-based model with respect to the expected cost. In a setting where the unit overscheduled cost was higher than the unit under-scheduled cost, scheduling fewer elective patients would replace the benefit of incorporating related features in the model. The tighter the available bed capacity for elective patients, the bigger the difference of the schedule cost between the feature-based model and the observation-based model. The study provides a reference for the theoretical study on bed capacity allocation between elective and emergency patients under the condition of the unknown true probability distribution of bed capacity and emergency demand, and it also proves that the approximate optimal policy has good performance.

Abstract P329: Lifetime Risk of Dementia: The Atherosclerosis Risk in Communities (ARIC) Study

Introduction: Prior studies suggest that 15-20% of adults will develop dementia after midlife. However, current lifetime risk estimates are based on limited dementia ascertainment, potentially resulting in underestimation. Methods: We conducted a prospective cohort analysis using data from the Atherosclerosis Risk in Communities Study (ARIC) (baseline 1990-1992). We used non-parametric cumulative incidence function estimation to calculate lifetime risk of dementia while accounting for the competing risk of death, overall and stratified by demographics and APOE ε 4 status. Dementia was ascertained until December 31, 2019 by: (1) cognitive testing at study visits, with diagnoses generated using a standardized computer algorithm and confirmed by a panel of experts; (2) semiannual telephone cognitive testing and Clinical Dementia Rating interviews with informants; and (3) ICD codes from hospital records and death certificates. Results: We included 13,358 participants (mean age: 57 years; 25% of Black race; 55% female sex; 31% carried at least one APOE ε 4 allele). Over a median follow-up of 24 (IQR: 20, 29) years, there were 2,812 incident cases of dementia. At age 55, the lifetime risk of dementia (up to age 95) was 43% (Fig A) . Adults with two copies of the APOE ε 4 allele had substantially higher lifetime risk (61%) of dementia compared to those with one copy (48%) and those none (40%), with differences beginning at age ~70 years (Fig B) . Lifetime risk was higher in women and Black adults (Fig C-D) . Conclusion: More than 4 in 10 adults will develop dementia after midlife. Comprehensive capture of dementia in the ARIC study results in lifetime risk estimate substantially exceeding prior research.

Changes in source specific PM2.5 from 2010 to 2019 in New York and New Jersey identified by dispersion normalized PMF

Since the Clean Air Act amendments of 1970, various efforts have been made in the United States to control ambient particulate matter and improve air quality. Although substantial progress had been made by the end of the 1990s, further reductions were needed to meet the National Ambient Air Quality Standards. To assess the effectiveness of the regulations and the impacts of economic drivers, we investigated the PM2.5 source trends at 11 sites in New York and New Jersey for the 2010–2019 period. Dispersion-normalized positive matrix factorization (DN-PMF) was used to reduce the meteorological impacts on the source apportionment. The Theil-Sen nonparametric estimator and piecewise linear regression were applied to the resolved source contributions to quantify the trends. While there is a consistent overall decrease of PM2.5 at all sites, increases were observed for some specific PM sources. Secondary Sulfate increased after 2017 at most sites based on the piecewise analysis. Similarly, recent-year increases were resolved for Diesel concentrations likely due to increasing vehicle miles traveled for heavy-duty vehicles. Back trajectory analysis of the OP-Rich source identified southeastern U.S. as a likely major source area, while Biomass Burning has a mixed source from local domestic heating as well as transported from proscribed and wildfires in the southeastern U.S.

A novel wind power deterministic and interval prediction framework based on the critic weight method, improved northern goshawk optimization, and kernel density estimation

Wind power prediction holds significant importance for the stable operation of power systems and the enhancement of energy utilization efficiency. Interval prediction, capable of providing more uncertain information, has garnered widespread attention. In this study we have designed a hybrid prediction framework for both deterministic and interval prediction. Initially, the original wind power data undergo cleaning using the quartile method and the fuzzy C-means (FCM) clustering algorithm. This approach not only eliminates scattered outliers but also identifies concentrated outliers more effectively. Subsequently, five distinct single prediction models were employed for point predictions. The critic weight method was applied to yield point forecasting results with higher prediction accuracy. Nonparametric kernel density estimation (KDE) and normal distribution (ND) were incorporated to calculate the prediction interval (PI) under varying confidence levels. Finally, an improved northern goshawk optimization (INGO) algorithm was proposed, integrating three enhancement methods: levy flight, sinusoidal mapping, and a reverse learning strategy. This optimization aims to fine-tune the weight coefficients of the aforementioned two interval prediction methods. The resulting PI overcomes the limitations of a single-interval prediction model. To effectively demonstrate the forecasting performance of the proposed prediction framework, two datasets from a wind turbine in northwest China were selected for simulation verification. In deterministic prediction, the designed hybrid forecast model demonstrates a reduction in root mean square error (RMSE), mean absolute percentage error (MAPE), and mean absolute error (MAE) by 8.5%, 16.98%, and 11.49%, respectively, compared to other single prediction models. In interval prediction, the prediction interval coverage probability (PICP) is enhanced, while the prediction interval normalized average width (PINAW) is effectively reduced. The designed interval prediction model can reduce the comprehensive evaluation index coverage width-based criterion (CWC) by 10.51%, 9.62%, and 12.95% on average at 95%, 90%, and 80% confidence levels, respectively. Simulation results verified the validity and feasibility of the proposed prediction framework.

Theory and practice of second-order expansions for moments of 100ρ% accelerated sequential stopping times in parametric and nonparametric estimation with arbitrary fractional ρ

Sampling strategies with fractional acceleration 0 < ρ < 1 achieve substantial operational savings compared with purely sequential counterparts. But, acceleration customarily yielded second-order (s.o.) lower and upper bounds for requisite characteristics when ρ − 1 was not an integer. First time in the literature, we have recently designed acceleration with asymptotic s.o. expansions in normal mean problems with ρ − 1 arbitrary. In this article, a general unified theory is now developed leading to asymptotic s.o. expansions for customarily studied characteristics with arbitrary ρ − 1 . That is, the previously known s.o. lower/upper bounds can be replaced with appropriate s.o. expansions in a variety of inference problems with prescribed accuracy. We emphasize the theoretical foundation and ensuing analyses with a series of interesting inference problems from (a) a number of non-normal distributions as well as (b) one- and two-sample distribution-free scenarios. These prove a desirable breadth of coverage under our proposed big tent.