Residual Bootstrap Research Articles

Nonparametric specification for non-stationary time series regression

We investigate the behavior of the Generalized Likelihood Ratio Test (GLRT) (Fan, Zhang and Zhang [Ann. Statist. 29 (2001) 153–193]) for time varying coefficient models where the regressors and errors are non-stationary time series and can be cross correlated. It is found that the GLRT retains the minimax rate of local alternative detection under weak dependence and non-stationarity. However, in general, the Wilks phenomenon as well as the classic residual bootstrap are sensitive to either conditional heteroscedasticity of the errors, non-stationarity or temporal dependence. An averaged test is suggested to alleviate the sensitivity of the test to the choice of bandwidth and is shown to be more powerful than tests based on a single bandwidth. An alternative wild bootstrap method is proposed and shown to be consistent when making inference of time varying coefficient models for non-stationary time series.

Simultaneous bootstrap for all three parameters in random coefficient autoregressive models

In this paper we consider autoregressive processes with random coefficients and develop bootstrap approaches that asymptotically work for the distribution of estimated autoregressive parameter as well as for the distribution of estimated variances of the innovation noise and the disturbance noise. We discuss how to obtain approximative residuals of the process and how to separate between the innovation and the disturbance noise in order to be able to extend the classical residual bootstrap for autoregressive processes to the situation considered in this paper. Thereafter, we propose a wild bootstrap procedure as a variation of the residual bootstrap that uses estimated densities of the innovation and the disturbance noise to generate bootstrap replicates of the data generating process. The consistency of the bootstrap approaches is established and their performance is illustrated by a simulation study.

Multilevel bootstrap analysis with assumptions violated.

Likelihood-based methods can work poorly when the residuals are not normally distributed and the variances across clusters are heterogeneous. The performance of two estimation methods, the non-parametric residual bootstrap (RB) and the restricted maximum likelihood (REML) for fitting multilevel models are compared through simulation studies in terms of bias, coverage, and precision. We find that (a) both methods produce unbiased estimates of the fixed parameters, but biased estimates of the random parameters, although the REML was more prone to give biased estimates for the variance components; (b) the RB method yields substantial reductions in the difference between nominal and actual confidence interval coverage, compared with the REML method; and (c) for the square root of the mean squared error (RMSE) of the fixed effects, the RB method performed slightly better than the REML method. For the variance components, however, the RB method did not offer a systematic improvement over the REML method in terms of RMSE. It can be stated that the RB method is, in general, superior to the REML method with violated assumptions.

Evaluation of bootstrap methods for estimating uncertainty of parameters in nonlinear mixed-effects models: a simulation study in population pharmacokinetics

Bootstrap methods are used in many disciplines to estimate the uncertainty of parameters, including multi-level or linear mixed-effects models. Residual-based bootstrap methods which resample both random effects and residuals are an alternative approach to case bootstrap, which resamples the individuals. Most PKPD applications use the case bootstrap, for which software is available. In this study, we evaluated the performance of three bootstrap methods (case bootstrap, nonparametric residual bootstrap and parametric bootstrap) by a simulation study and compared them to that of an asymptotic method (Asym) in estimating uncertainty of parameters in nonlinear mixed-effects models (NLMEM) with heteroscedastic error. This simulation was conducted using as an example of the PK model for aflibercept, an anti-angiogenic drug. As expected, we found that the bootstrap methods provided better estimates of uncertainty for parameters in NLMEM with high nonlinearity and having balanced designs compared to the Asym, as implemented in MONOLIX. Overall, the parametric bootstrap performed better than the case bootstrap as the true model and variance distribution were used. However, the case bootstrap is faster and simpler as it makes no assumptions on the model and preserves both between subject and residual variability in one resampling step. The performance of the nonparametric residual bootstrap was found to be limited when applying to NLMEM due to its failure to reflate the variance before resampling in unbalanced designs where the Asym and the parametric bootstrap performed well and better than case bootstrap even with stratification.

Estimating Uncertainty in White Matter Tractography Using Wild Non-local Bootstrap.

Statistics derived from diffusion MRI data, especially those related to tractography, are often highly non-linear and non-Gaussian with unknown complex distributions. In estimating the sampling distributions of these statistics, many existing techniques are limited by their reliance on models that assume normality and that are yet to be verified in complex situations where various noise sources, such as physiologic variation, scanner instability, and imaging noise, might be simultaneously present. In complex conditions as such, a viable solution is the bootstrap, which due to its distribution-independent nature is an appealing tool for the estimation of the variability of almost any statistic, without relying on complicated theoretical calculations, but purely on computer simulation. In this paper, we will examine whether a new bootstrap scheme, called the wild non-local bootstrap (W-NLB), is effective in estimating the uncertainty in tractography data. In contrast to the residual or wild bootstrap, which relies on a predetermined data model, or the repetition bootstrap, which requires repeated signal measurements, W-NLB does not assume a predetermined form of data structure and obviates the need for time-consuming multiple acquisitions. W-NLB hinges on the observation that local imaging information recurs in the image. This self-similarity implies that imaging information coming from spatially distant (non-local) regions can be exploited for more effective estimation of statistics of interest. In silico evaluations indicate that W-NLB produces distribution estimates that are in closer agreement to those generated using Monte Carlo simulations, compared with the conventional residual bootstrap. Evaluations using in vivo data show that W-NLB produces results that are in agreement with our knowledge on the white matter connection architecture.

Estimation and Asymptotic Properties in Periodic GARCH(1, 1) Models

This article studies the probabilistic structure and asymptotic inference of the first-order periodic generalized autoregressive conditional heteroscedasticity (PGARCH(1, 1)) models in which the parameters in volatility process are allowed to switch between different regimes. First, we establish necessary and sufficient conditions for a PGARCH(1, 1) process to have a unique stationary solution (in periodic sense) and for the existence of moments of any order. Second, using the representation of squared PGARCH(1, 1) model as a PARMA(1, 1) model, we then consider Yule-Walker type estimators for the parameters in PGARCH(1, 1) model and derives their consistency and asymptotic normality. The estimator can be surprisingly efficient for quite small numbers of autocorrelations and, in some cases can be more efficient than the least squares estimate (LSE). We use a residual bootstrap to define bootstrap estimators for the Yule-Walker estimates and prove the consistency of this bootstrap method. A set of numerical experiments illustrates the practical relevance of our theoretical results.

Total least squares and bootstrapping with applications in calibration

The solution to the errors-in-variables problem computed through total least squares is highly nonlinear. Because of this, many statistical procedures for constructing confidence intervals and testing hypotheses cannot be applied. One possible solution to this dilemma is bootstrapping. A nonparametric bootstrap technique could fail. Here, the proper nonparametric bootstrap procedure is provided and its correctness is proved. On the other hand, a residual bootstrap is not valid and suitable in this case. The results are illustrated through a simulation study. An application of this approach to calibration data is presented.

BOOTSTRAP FOR RANDOM COEFFICIENT AUTOREGRESSIVE MODELS

In this paper, we deal with autoregressive processes with random coefficients. We propose a least‐squares estimator for the fourth‐order moments of both the innovation and disturbance noises and state its consistency. The main theme of the paper is the development of bootstrap procedures for the autoregressive parameter. We show how to obtain approximative residuals for the process even though the standard method for autoregressive processes does not work in this context since one then would obtain convoluted residuals of the innovation and disturbance noises. These ideas lead to a modification of the classical residual bootstrap for autoregressive processes. The consistency of the bootstrap procedure is established. Further, the estimators proposed in the first part are used to form two wild bootstrap modifications. Finally, the performances of the three bootstrap procedures are explored by a simulation study and compared with each other.

Rates of convergence of the Adaptive LASSO estimators to the Oracle distribution and higher order refinements by the bootstrap

Zou [J. Amer. Statist. Assoc. 101 (2006) 1418-1429] proposed the Adaptive LASSO (ALASSO) method for simultaneous variable selection and estimation of the regression parameters, and established its oracle property. In this paper, we investigate the rate of convergence of the ALASSO estimator to the oracle distribution when the dimension of the regression parameters may grow to infinity with the sample size. It is shown that the rate critically depends on the choices of the penalty parameter and the initial estimator, among other factors, and that confidence intervals (CIs) based on the oracle limit law often have poor coverage accuracy. As an alternative, we consider the residual bootstrap method for the ALASSO estimators that has been recently shown to be consistent; cf. Chatterjee and Lahiri [J. Amer. Statist. Assoc. 106 (2011a) 608-625]. We show that the bootstrap applied to a suitable studentized version of the ALASSO estimator achieves second-order correctness, even when the dimension of the regression parameters is unbounded. Results from a moderately large simulation study show marked improvement in coverage accuracy for the bootstrap CIs over the oracle based CIs.

A decentralized blind source separation algorithm for ambient modal identification in the presence of narrowband disturbances

In this paper, a new decentralized identification method based on the principles of blind source separation (BSS) that can perform modal identification in the presence of narrowband external excitations is presented. This algorithm is based on the principle of sparse BSS involving wavelet transforms to undertake identification for underdetermined static mixtures. Most BSS methods in the context of modal identification assume that the excitation is white and does not contain narrow band excitation frequencies. However, this assumption is not satisfied in many situations when the excitation is a superposition of narrowband harmonic(s) and broadband disturbance, for example, rotating machinery fixed to flexible structures, human-induced vibration in pedestrian bridges, or dynamic interaction with surrounding structures. Under such conditions, traditional BSS methods (such as second-order blind identification) yield sources (modes) without any indication as to whether the identified source(s) is a system or an excitation harmonic. In the proposed method, underdetermined BSS is employed, involving transformation of measurements to the time–frequency domain, resulting in a sparse representation. The statistical characteristics of the sources are estimated using residual bootstrap technique, which are then used to delineate the sources corresponding to external disturbances versus intrinsic modes of the system, in a decentralized fashion. The decentralized architecture is aimed at reducing the cost of instrumentation in both traditional wired and recent wireless applications. The performance of the algorithm is demonstrated using both a numerical study and a bench-scale laboratory experiment. The primary advantage of the proposed method is its ability to identify the modal harmonics of a flexible structure under narrowband excitations, in a decentralized fashion, where traditional BSS algorithms fail to achieve the same goals. Copyright © 2013 John Wiley & Sons, Ltd.

A comparison of bootstrap approaches for estimating uncertainty of parameters in linear mixed‐effects models

A version of the nonparametric bootstrap, which resamples the entire subjects from original data, called the case bootstrap, has been increasingly used for estimating uncertainty of parameters in mixed-effects models. It is usually applied to obtain more robust estimates of the parameters and more realistic confidence intervals (CIs). Alternative bootstrap methods, such as residual bootstrap and parametric bootstrap that resample both random effects and residuals, have been proposed to better take into account the hierarchical structure of multi-level and longitudinal data. However, few studies have been performed to compare these different approaches. In this study, we used simulation to evaluate bootstrap methods proposed for linear mixed-effect models. We also compared the results obtained by maximum likelihood (ML) and restricted maximum likelihood (REML). Our simulation studies evidenced the good performance of the case bootstrap as well as the bootstraps of both random effects and residuals. On the other hand, the bootstrap methods that resample only the residuals and the bootstraps combining case and residuals performed poorly. REML and ML provided similar bootstrap estimates of uncertainty, but there was slightly more bias and poorer coverage rate for variance parameters with ML in the sparse design. We applied the proposed methods to a real dataset from a study investigating the natural evolution of Parkinson's disease and were able to confirm that the methods provide plausible estimates of uncertainty. Given that most real-life datasets tend to exhibit heterogeneity in sampling schedules, the residual bootstraps would be expected to perform better than the case bootstrap.

Asymptotic properties of Lasso+mLS and Lasso+Ridge in sparse high-dimensional linear regression

We study the asymptotic properties of Lasso+mLS and Lasso+ Ridge under the sparse high-dimensional linear regression model: Lasso selecting predictors and then modified Least Squares (mLS) or Ridge estimating their coefficients. First, we propose a valid inference procedure for parameter estimation based on parametric residual bootstrap after Lasso+ mLS and Lasso+Ridge. Second, we derive the asymptotic unbiasedness of Lasso+mLS and Lasso+Ridge. More specifically, we show that their biases decay at an exponential rate and they can achieve the oracle convergence rate of $s/n$ (where $s$ is the number of nonzero regression coefficients and $n$ is the sample size) for mean squared error (MSE). Third, we show that Lasso+mLS and Lasso+Ridge are asymptotically normal. They have an oracle property in the sense that they can select the true predictors with probability converging to $1$ and the estimates of nonzero parameters have the same asymptotic normal distribution that they would have if the zero parameters were known in advance. In fact, our analysis is not limited to adopting Lasso in the selection stage, but is applicable to any other model selection criteria with exponentially decay rates of the probability of selecting wrong models.

Aspects of steady-state operation of the Wendelstein 7-X stellarator

The objective of Wendelstein 7-X is to demonstrate steady-state operation at β -values of up to 5%, at ion temperatures of several keV and plasma densities of up to 2 × 1020 m−3. The second operational phase foresees a fully steady-state high heat flux (HHF) divertor. Preparations are underway to cope with residual bootstrap currents, either by electron cyclotron current drive or by HHF protection elements. The main steady-state heating system is an electron cyclotron resonance heating facility. Various technical improvements of the gyrotrons have been implemented recently. They enable a reliable operation at the 1 MW power level. Some of the technical issues preparing plasma diagnostics for steady-state operation are exemplified. This includes the protection against non-absorbed microwave radiation.

Bootstrap, Jackknife and COLS: Bias and Mean Squared Error in Estimation of Autoregressive Models

AbstractWe compare a number of bias-correction methodologies in terms of mean squared error and remaining bias, including the residual bootstrap, the relatively unexplored Quenouille jackknife, and methods based on analytical approximation of moments. We introduce a new higher-order jackknife estimator for the AR(1) with constant. Simulation results are presented for four different error structures, including GARCH. We include results for a relatively extreme situation where the errors are highly skewed and leptokurtic. It is argued that the bootstrap and analytical-correction (COLS) approaches are to be favoured overall, though the jackknife methods are the least biased. We find that COLS tends to have the lowest mean squared error, though the bootstrap also does well.

Bootstrapping Multilayer Neural Networks for Portfolio Construction

Despite having become firmly established as one of the major cornerstone principles of modern finance, traditional Markowitz mean-variance analysis has, nevertheless, failed to gain widespread acceptance as a practical tool for equity management. The Markowitz optimization enigma essentially centers on the severe estimation risk associated with the input parameters, as well as the resultant financially irrelevant or even false optimal portfolios and asset allocation proposals. We therefore propose a portfolio construction method in the present study which incorporates the adoption of bootstrapping neural network architecture. In specific terms, a residual bootstrapping sample, which is derived from multilayer feedforward neural networks, is incorporated into the estimation of the expected returns and the covariance matrix, which are then, in turn, integrated into the traditional Markowitz optimization procedure. The efficacy of our proposed approach is illustrated by comparing it with traditional Markowitz mean-variance analysis, as well as the James-Stein and minimum-variance estimators, with the empirical results indicating that this novel approach significantly outperforms the benchmark models, in terms of various risk-adjusted performance measures. The evidence provided in this study suggests that this new approach has significant promise with regard to the enhancement of the investment value of Markowitz mean-variance analysis.

Testing for symmetries in multivariate inverse problems

We propose a test for shape constraints which can be expressed by transformations of the coordinates of multivariate regression functions. The method is motivated by the constraint of symmetry with respect to some unknown hyperplane but can easily be generalized to other shape constraints of this type or other semi-parametric settings. In a first step, the unknown parameters are estimated and in a second step, this estimator is used in the L2-type test statistic for the shape constraint. We consider the asymptotic behavior of the estimated parameter and show that it converges with parametric rate if the shape constraint is true. Moreover, we derive the asymptotic distribution of the test statistic under the null hypothesis and furthermore propose a bootstrap test based on the residual bootstrap. In a simulation study, we investigate the finite sample performance of the estimator as well as the bootstrap test.

Efficient Estimation of Conditional Risk Measures in a Semiparametric GARCH Model

This paper proposes efficient estimators of risk measures in a semiparametric GARCH model defined through moment constraints. Moment constraints are often used to identify and estimate the mean and variance parameters and are however discarded when estimating error quantiles. In order to prevent this efficiency loss in quantile estimation we propose a quantile estimator based on inverting an empirical likelihood weighted distribution estimator. It is found that the new quantile estimator is uniformly more efficient than the simple empirical quantile and a quantile estimator based on normalized residuals. At the same time, the efficiency gain in error quantile estimation hinges on the efficiency of estimators of the variance parameters. We show that the same conclusion applies to the estimation of conditional Expected Shortfall. Our comparison also leads to interesting implications of residual bootstrap for dynamic models. We find that these proposed estimators for conditional Value-at-Risk and expected shortfall are asymptotically mixed normal. This asymptotic theory can be used to construct confidence bands for these estimators by taking account of parameter uncertainty. Simulation evidence as well as empirical results are provided.

Properties of the neural network sieve bootstrap

In this paper, a sieve bootstrap scheme, the neural network sieve bootstrap, for nonlinear time series is proposed. The approach, which is nonparametric in its spirit, retains the conceptual simplicity of a classical residual bootstrap, and it has some advantages with respect to the blockwise schemes and kernel bootstrap techniques. The resampling scheme from the residuals of the feedforward neural networks is shown to be asymptotically justified. A Monte Carlo simulation study shows that the procedure performs similar to the autoregressive (AR)-sieve bootstrap for linear processes, while it outperforms the AR-sieve bootstrap, the moving block bootstrap and kernel bootstrap for nonlinear processes, both in terms of bias and variability.

Bootstrapping Lasso Estimators

In this article, we consider bootstrapping the Lasso estimator of the regression parameter in a multiple linear regression model. It is known that the standard bootstrap method fails to be consistent. Here, we propose a modified bootstrap method, and show that it provides valid approximation to the distribution of the Lasso estimator, for all possible values of the unknown regression parameter vector, including the case where some of the components are zero. Further, we establish consistency of the modified bootstrap method for estimating the asymptotic bias and variance of the Lasso estimator. We also show that the residual bootstrap can be used to consistently estimate the distribution and variance of the adaptive Lasso estimator. Using the former result, we formulate a novel data-based method for choosing the optimal penalizing parameter for the Lasso using the modified bootstrap. A numerical study is performed to investigate the finite sample performance of the modified bootstrap. The methodology proposed in the article is illustrated with a real data example.

Probabilistic fiber tracking using the residual bootstrap with constrained spherical deconvolution.

Constrained spherical deconvolution (CSD) is a new technique that, based on high-angular resolution diffusion imaging (HARDI) MR data, estimates the orientation of multiple intravoxel fiber populations within regions of complex white matter architecture, thereby overcoming the limitations of the widely used diffusion tensor imaging (DTI) technique. One of its main applications is fiber tractography. The noisy nature of diffusion-weighted (DW) images, however, affects the estimated orientations and the resulting fiber trajectories will be subject to uncertainty. The impact of noise can be large, especially for HARDI measurements, which employ relatively high b-values. To quantify the effects of noise on fiber trajectories, probabilistic tractography was introduced, which considers multiple possible pathways emanating from one seed point, taking into account the uncertainty of local fiber orientations. In this work, a probabilistic tractography algorithm is presented based on CSD and the residual bootstrap. CSD, which provides accurate and precise estimates of multiple fiber orientations, is used to extract the local fiber orientations. The residual bootstrap is used to estimate fiber tract probability within a clinical time frame, without prior assumptions about the form of uncertainty in the data. By means of Monte Carlo simulations, the performance of the CSD fiber pathway uncertainty estimator is measured in terms of accuracy and precision. In addition, the performance of the proposed method is compared to state-of-the-art DTI residual bootstrap tractography and to an existing probabilistic CSD tractography algorithm using clinical DW data.