Multiple Bootstrap Samples Research Articles

Evaluating the Stability of Embedding-based Word Similarities

Word embeddings are increasingly being used as a tool to study word associations in specific corpora. However, it is unclear whether such embeddings reflect enduring properties of language or if they are sensitive to inconsequential variations in the source documents. We find that nearest-neighbor distances are highly sensitive to small changes in the training corpus for a variety of algorithms. For all methods, including specific documents in the training set can result in substantial variations. We show that these effects are more prominent for smaller training corpora. We recommend that users never rely on single embedding models for distance calculations, but rather average over multiple bootstrap samples, especially for small corpora.

Bootstrapping Time-Course Gene Expression Data for Gene Networks: Application to Gene Relevance Networks.

Identification of gene regulatory networks (GRNs) is a fundamental step to understand the molecular role of each gene and it helps to develop treatment and cure of a disease. To identify GRNs, time-course gene expression data are widely used. However, the identification is hampered by intrinsic attributes of the data such as small sample size, a large number of variables, and complex error structures with high variation. Under this situation, most GRN inference methods utilize point estimators or make numerous assumptions that are often incompatible with the experimental data. Moreover, different inference methods often provide inconsistent results. An alternative to alleviate this problem can be the bootstrap method because it provides more reliable outcomes by integrating results from multiple bootstrap samples without any distributional assumptions. In this study, we propose a bootstrap method for dependent time-course gene expression data and we mainly focus on its application to gene relevance networks. The proposed method is applied to gene networks for zebrafish retina.

Bayesian Latent Class Models for the Multiple Imputation of Categorical Data

Abstract. Latent class analysis has been recently proposed for the multiple imputation (MI) of missing categorical data, using either a standard frequentist approach or a nonparametric Bayesian model called Dirichlet process mixture of multinomial distributions (DPMM). The main advantage of using a latent class model for multiple imputation is that it is very flexible in the sense that it can capture complex relationships in the data given that the number of latent classes is large enough. However, the two existing approaches also have certain disadvantages. The frequentist approach is computationally demanding because it requires estimating many LC models: first models with different number of classes should be estimated to determine the required number of classes and subsequently the selected model is reestimated for multiple bootstrap samples to take into account parameter uncertainty during the imputation stage. Whereas the Bayesian Dirichlet process models perform the model selection and the handling of the parameter uncertainty automatically, the disadvantage of this method is that it tends to use a too small number of clusters during the Gibbs sampling, leading to an underfitting model yielding invalid imputations. In this paper, we propose an alternative approach which combined the strengths of the two existing approaches; that is, we use the Bayesian standard latent class model as an imputation model. We show how model selection can be performed prior to the imputation step using a single run of the Gibbs sampler and, moreover, show how underfitting is prevented by using large values for the hyperparameters of the mixture weights. The results of two simulation studies and one real-data study indicate that with a proper setting of the prior distributions, the Bayesian latent class model yields valid imputations and outperforms competing methods.

A novel bagging approach for variable ranking and selection via a mixed importance measure

ABSTRACTAt present, ensemble learning has exhibited its great power in stabilizing and enhancing the performance of some traditional variable selection methods such as lasso and genetic algorithm. In this paper, a novel bagging ensemble method called BSSW is developed to implement variable ranking and selection in linear regression models. Its main idea is to execute stepwise search algorithm on multiple bootstrap samples. In each trial, a mixed importance measure is assigned to each variable according to the order that it is selected into final model as well as the improvement of model fitting resulted from its inclusion. Based on the importance measure averaged across some bootstrapping trials, all candidate variables are ranked and then decided to be important or not. To extend the scope of application, BSSW is extended to the situation of generalized linear models. Experiments carried out with some simulated and real data indicate that BSSW achieves better performance in most studied cases when compared with several other existing methods.

Subgroup Identification in Clinical Trials by Stochastic SIDEScreen Methods

ABSTRACTSubgroup identification for personalized medicine has become very popular in the last decade. Efficient recursive partitioning procedures adapted from machine learning are natural approaches for performing subgroup identification based on pre-defined biomarkers since they provide subgroups as terminal nodes in the decision tree. However, recursive partitioning is also known as a potentially unstable procedure with results being quite sensitive to normal sampling variability in the data. One common approach, borrowed from ensemble learning, to overcome such instability is application of recursive partitioning to multiple data sets sampled from the observed data followed by averaging the results over the collection of subgroups.This article proposes an alternative approach to subgroup identification in clinical trials that first evaluates the predictive strength of biomarkers based on variable importance and then applies recursive partitioning to the biomarkers with the highest variable importance scores. A deterministic version of this idea was implemented in the Adaptive SIDEScreen method that generates a collection of patient subgroups by retaining multiple candidate splits of each parent group by different biomarkers (Lipkovich and Dmitrienko 2014a, 2014b). Now, we extend the Adaptive SIDEScreen and introduce the Stochastic SIDEScreen method. The key idea is to introduce randomness in the subgroup generation process, borrowing from bagging methods, to produce a broader collection of subgroups. Specifically, the SIDES method, where the most promising biomarkers are selected for each parent group from a set of candidate biomarkers, is applied to multiple bootstrap samples of the data. This new approach leads to a more reliable biomarker selection process, which is especially important for smaller, early phase studies when biomarker selection is typically carried out. The method is illustrated using clinical trial examples.

Analyzing large datasets with bootstrap penalization.

Data with a large p (number of covariates) and/or a large n (sample size) are now commonly encountered. For many problems, regularization especially penalization is adopted for estimation and variable selection. The straightforward application of penalization to large datasets demands a "big computer" with high computational power. To improve computational feasibility, we develop bootstrap penalization, which dissects a big penalized estimation into a set of small ones, which can be executed in a highly parallel manner and each only demands a "small computer". The proposed approach takes different strategies for data with different characteristics. For data with a large p but a small to moderate n, covariates are first clustered into relatively homogeneous blocks. The proposed approach consists of two sequential steps. In each step and for each bootstrap sample, we select blocks of covariates and run penalization. The results from multiple bootstrap samples are pooled to generate the final estimate. For data with a large n but a small to moderate p, we bootstrap a small number of subjects, apply penalized estimation, and then conduct a weighted average over multiple bootstrap samples. For data with a large p and a large n, the natural marriage of the previous two methods is applied. Numerical studies, including simulations and data analysis, show that the proposed approach has computational and numerical advantages over the straightforward application of penalization. An R package has been developed to implement the proposed methods.

A Bootstrap Metropolis–Hastings Algorithm for Bayesian Analysis of Big Data

Markov chain Monte Carlo (MCMC) methods have proven to be a very powerful tool for analyzing data of complex structures. However, their computer-intensive nature, which typically require a large number of iterations and a complete scan of the full dataset for each iteration, precludes their use for big data analysis. In this article, we propose the so-called bootstrap Metropolis–Hastings (BMH) algorithm that provides a general framework for how to tame powerful MCMC methods to be used for big data analysis, that is, to replace the full data log-likelihood by a Monte Carlo average of the log-likelihoods that are calculated in parallel from multiple bootstrap samples. The BMH algorithm possesses an embarrassingly parallel structure and avoids repeated scans of the full dataset in iterations, and is thus feasible for big data problems. Compared to the popular divide-and-combine method, BMH can be generally more efficient as it can asymptotically integrate the whole data information into a single simulation run. The BMH algorithm is very flexible. Like the Metropolis–Hastings algorithm, it can serve as a basic building block for developing advanced MCMC algorithms that are feasible for big data problems. This is illustrated in the article by the tempering BMH algorithm, which can be viewed as a combination of parallel tempering and the BMH algorithm. BMH can also be used for model selection and optimization by combining with reversible jump MCMC and simulated annealing, respectively. Supplementary materials for this article are available online.

Statistical Challenges in Risk Prediction.

Risk prediction is an important area of clinical application of statistical methods. Angela Wood, assisted by Ian White and Simon Thompson, organized a three-day international workshop on Statistical Challenges in Risk Prediction in Cambridge, UK, on November 19–21, 2012, with 21 invited academic statisticians (Fig. 1). The workshop covered statistical challenges faced when developing, assessing, and validating risk models for generic clinical use and in particular for cardiovascular risk prediction. The papers in this special issue (which went through the usual peer review process) represent some of the presentations and discussions from the four workshop sessions. The first workshop session was on development of risk prediction models and included presentations and discussion on model stability issues, using repeated measures of risk factors, and how to choose between age and study duration as the timescale. The session is represented here by the paper of Sauerbrei et al., “On stability issues in deriving multivariable regression models”. A selected model is “stable” if it is unchanged by moderate changes to the data. An unstable model may yield stable predictions, but it may be sensitive to influential data points. Stability is assessed by re-applying the model selection procedure to multiple bootstrap samples drawn from the observed data. Graphical displays are proposed to explore the issue and highlight possible influential data points. The second workshop session was on assessment of risk prediction models. This is represented here by three papers based on presentations and discussion from the workshop. “Graphical assessment of incremental value of novel markers in prediction models: From statistical to decision analytical perspectives” by Steyerberg et al. is a timely and important review of available methods, from graphs of predictions through measures of explained variation to methods based on reclassification. In addition, it proposes the net reclassification risk graph as a new graphical presentation. The other two papers are more technical and focus on new measures of explained variation and discrimination. “Explained variation for recurrent event data” by Alotaibi et al. extends measures of explained variation used for standard survival data to the setting of recurrent events, illustrated by data on episodes of infant diarrhea. “Covariate-adjusted measures of discrimination for survival data” by White and Rapsomaniki argues that covariate adjustment is needed to make discrimination measures comparable across studies, and proposes appropriate methods. The third workshop session was on current challenges in cardiovascular risk prediction and focused on individual participant data meta-analysis, risk prediction using case-cohort and nested case–control studies, multiple imputation and prediction, model validation, and optimizing sequential screening. The session is represented here by “The estimation and use of predictions for the assessment of model performance using large samples with multiply imputed data” by Wood et al., which explores the difficulties of risk prediction in a cardiovascular or other setting when predictors are incomplete. “Ideal” and “pragmatic” model performances are distinguished, and methods are proposed to estimate both. The final workshop session was on evaluating the clinical value of risk models and included presentations on measures to quantify public health impact, including decision curve analysis, and on the economics of risk prediction. No paper from this session appears in this special issue, although Steyerberg et al. (described above) discuss decision curve analysis. Discussion during the workshop concluded that more methods development is needed to advance the evaluation of risk models in the clinical context in which the models will be used, and hence to evaluate the patient benefit and economic benefit of using the risk models in clinical practice. We hope you will enjoy reading these papers and contributing to the development of this important area of biostatistical methodology. We thank the Biometrical Journal and its editors and reviewers for allowing us to publish the papers together in the journal. Angela Wood, Ian White Guest Editors

Bootstrap model selection had similar performance for selecting authentic and noise variables compared to backward variable elimination: a simulation study

Objective Researchers have proposed using bootstrap resampling in conjunction with automated variable selection methods to identify predictors of an outcome and to develop parsimonious regression models. Using this method, multiple bootstrap samples are drawn from the original data set. Traditional backward variable elimination is used in each bootstrap sample, and the proportion of bootstrap samples in which each candidate variable is identified as an independent predictor of the outcome is determined. The performance of this method for identifying predictor variables has not been examined. Study Design and Setting Monte Carlo simulation methods were used to determine the ability of bootstrap model selection methods to correctly identify predictors of an outcome when those variables that are selected for inclusion in at least 50% of the bootstrap samples are included in the final regression model. We compared the performance of the bootstrap model selection method to that of conventional backward variable elimination. Results Bootstrap model selection tended to result in an approximately equal proportion of selected models being equal to the true regression model compared with the use of conventional backward variable elimination. Conclusion Bootstrap model selection performed comparatively to backward variable elimination for identifying the true predictors of a binary outcome.

Using the bootstrap to improve estimation and confidence intervals for regression coefficients selected using backwards variable elimination

Applied researchers frequently use automated model selection methods, such as backwards variable elimination, to develop parsimonious regression models. Statisticians have criticized the use of these methods for several reasons, amongst them are the facts that the estimated regression coefficients are biased and that the derived confidence intervals do not have the advertised coverage rates. We developed a method to improve estimation of regression coefficients and confidence intervals which employs backwards variable elimination in multiple bootstrap samples. In a given bootstrap sample, predictor variables that are not selected for inclusion in the final regression model have their regression coefficient set to zero. Regression coefficients are averaged across the bootstrap samples, and non-parametric percentile bootstrap confidence intervals are then constructed for each regression coefficient. We conducted a series of Monte Carlo simulations to examine the performance of this method for estimating regression coefficients and constructing confidence intervals for variables selected using backwards variable elimination. We demonstrated that this method results in confidence intervals with superior coverage compared with those developed from conventional backwards variable elimination. We illustrate the utility of our method by applying it to a large sample of subjects hospitalized with a heart attack.

Cross-validated bagged learning

Many applications aim to learn a high dimensional parameter of a data generating distribution based on a sample of independent and identically distributed observations. For example, the goal might be to estimate the conditional mean of an outcome given a list of input variables. In this prediction context, bootstrap aggregating (bagging) has been introduced as a method to reduce the variance of a given estimator at little cost to bias. Bagging involves applying an estimator to multiple bootstrap samples and averaging the result across bootstrap samples. In order to address the curse of dimensionality, a common practice has been to apply bagging to estimators which themselves use cross-validation, thereby using cross-validation within a bootstrap sample to select fine-tuning parameters trading off bias and variance of the bootstrap sample-specific candidate estimators. In this article we point out that in order to achieve the correct bias variance trade-off for the parameter of interest, one should apply the cross-validation selector externally to candidate bagged estimators indexed by these fine-tuning parameters. We use three simulations to compare the new cross-validated bagging method with bagging of cross-validated estimators and bagging of non-cross-validated estimators.