Bootstrap Error Estimation Research Articles

Equate: Observed-Score Linking and Equating in R.

Linking and equating are statistical procedures used to convert scores from one measurement scale to another. These procedures are most often used in testing programs that involve multiple test forms, where adjustments are made for form difficulty differences when creating a measurement scale that is common across forms. Linking and equating methods are traditionally distinguished by the type of scores they are applied to, whether observed scores or scores from an item response theory model. Methods are also distinguished by the study design under which measurements are taken. The R package equate (Albano, 2014) is free, open-source software for conducting observed-score linking and equating under single-group, equivalentgroups, and nonequivalent-groups designs with one or more anchor tests. The equate package estimates identity, mean, linear, and equipercentile equating functions (Kolen & Brennan, 2014); general linear functions (Albano, 2015); circle-arc functions (Livingston & Kim, 2009); and composites of these. Equating methods include nominal-weights (Babcock, Albano, & Raymond, 2012), Tucker, Levine observed-score, Levine truescore, Braun/Holland, frequency estimation, and chained equating. Plotting and summary methods are also provided, along with methods for multivariate loglinear presmoothing and for (parametric) bootstrap error estimation. The equate package is available for Linux, Mac, and Windows within the statistical environment R (R Core Team, 2015). Source and binary versions of the package and user manual are distributed free via the comprehensive R archival network, online at https://cran.r-project.org/. The latest development version of the package can be obtained online at https://github.com/

Unified description of efficiency correction and error estimation for moments of conserved quantities in heavy-ion collisions

We provide a unified description of efficiency correction and error estimation for moments of conserved quantifies in heavy-ion collisions. Moments and cumulants are expressed in terms of the factorial moments, which can be easily corrected for the efficiency effect. By deriving the covariance between factorial moments, one can obtain the general error formula for the efficiency corrected moments based on the error propagation derived from the Delta theorem. The skellam distribution based Monto Carlo simulation is used to test the Delta theorem and Bootstrap error estimation methods. The statistical errors calculated from the two methods can well reflect the statistical fluctuations of the efficiency corrected moments.

Unbiased bootstrap error estimation for linear discriminant analysis.

Convex bootstrap error estimation is a popular tool for classifier error estimation in gene expression studies. A basic question is how to determine the weight for the convex combination between the basic bootstrap estimator and the resubstitution estimator such that the resulting estimator is unbiased at finite sample sizes. The well-known 0.632 bootstrap error estimator uses asymptotic arguments to propose a fixed 0.632 weight, whereas the more recent 0.632+ bootstrap error estimator attempts to set the weight adaptively. In this paper, we study the finite sample problem in the case of linear discriminant analysis under Gaussian populations. We derive exact expressions for the weight that guarantee unbiasedness of the convex bootstrap error estimator in the univariate and multivariate cases, without making asymptotic simplifications. Using exact computation in the univariate case and an accurate approximation in the multivariate case, we obtain the required weight and show that it can deviate significantly from the constant 0.632 weight, depending on the sample size and Bayes error for the problem. The methodology is illustrated by application on data from a well-known cancer classification study.Electronic supplementary materialThe online version of this article (doi:10.1186/s13637-014-0015-0) contains supplementary material, which is available to authorized users.

μHEM for identification of differentially expressed miRNAs using hypercuboid equivalence partition matrix

BackgroundThe miRNAs, a class of short approximately 22‐nucleotide non‐coding RNAs, often act post‐transcriptionally to inhibit mRNA expression. In effect, they control gene expression by targeting mRNA. They also help in carrying out normal functioning of a cell as they play an important role in various cellular processes. However, dysregulation of miRNAs is found to be a major cause of a disease. It has been demonstrated that miRNA expression is altered in many human cancers, suggesting that they may play an important role as disease biomarkers. Multiple reports have also noted the utility of miRNAs for the diagnosis of cancer. Among the large number of miRNAs present in a microarray data, a modest number might be sufficient to classify human cancers. Hence, the identification of differentially expressed miRNAs is an important problem particularly for the data sets with large number of miRNAs and small number of samples.ResultsIn this regard, a new miRNA selection algorithm, called μHEM, is presented based on rough hypercuboid approach. It selects a set of miRNAs from a microarray data by maximizing both relevance and significance of the selected miRNAs. The degree of dependency of sample categories on miRNAs is defined, based on the concept of hypercuboid equivalence partition matrix, to measure both relevance and significance of miRNAs. The effectiveness of the new approach is demonstrated on six publicly available miRNA expression data sets using support vector machine. The.632+ bootstrap error estimate is used to minimize the variability and biasedness of the derived results.ConclusionsAn important finding is that the μHEM algorithm achieves lowest B.632+ error rate of support vector machine with a reduced set of differentially expressed miRNAs on four expression data sets compare to some existing machine learning and statistical methods, while for other two data sets, the error rate of the μHEM algorithm is comparable with the existing techniques. The results on several microarray data sets demonstrate that the proposed method can bring a remarkable improvement on miRNA selection problem. The method is a potentially useful tool for exploration of miRNA expression data and identification of differentially expressed miRNAs worth further investigation.

Rough sets for in silico identification of differentially expressed miRNAs

The microRNAs, also known as miRNAs, are the class of small noncoding RNAs. They repress the expression of a gene posttranscriptionally. In effect, they regulate expression of a gene or protein. It has been observed that they play an important role in various cellular processes and thus help in carrying out normal functioning of a cell. However, dysregulation of miRNAs is found to be a major cause of a disease. Various studies have also shown the role of miRNAs in cancer and the utility of miRNAs for the diagnosis of cancer and other diseases. Unlike with mRNAs, a modest number of miRNAs might be sufficient to classify human cancers. However, the absence of a robust method to identify differentially expressed miRNAs makes this an open problem. In this regard, this paper presents a novel approach for in silico identification of differentially expressed miRNAs from microarray expression data sets. It integrates judiciously the theory of rough sets and merit of the so-called B.632+ bootstrap error estimate. While rough sets select relevant and significant miRNAs from expression data, the B.632+ error rate minimizes the variability and bias of the derived results. The effectiveness of the proposed approach, along with a comparison with other related approaches, is demonstrated on several miRNA microarray expression data sets, using the support vector machine.

A Benchmark Test Set for Alchemical Free Energy Transformations and Its Use to Quantify Error in Common Free Energy Methods.

There is a significant need for improved tools to validate thermophysical quantities computed via molecular simulation. In this paper we present the initial version of a benchmark set of testing methods for calculating free energies of molecular transformation in solution. This set is based on molecular changes common to many molecular design problems, such as insertion and deletion of atomic sites and changing atomic partial charges. We use this benchmark set to compare the statistical efficiency, reliability, and quality of uncertainty estimates for a number of published free energy methods, including thermodynamic integration, free energy perturbation, the Bennett acceptance ratio (BAR) and its multistate equivalent MBAR. We identify MBAR as the consistently best performing method, though other methods are frequently comparable in reliability and accuracy in many cases. We demonstrate that assumptions of Gaussian distributed errors in free energies are usually valid for most methods studied. We demonstrate that bootstrap error estimation is a robust and useful technique for estimating statistical variance for all free energy methods studied. This benchmark set is provided in a number of different file formats with the hope of becoming a useful and general tool for method comparisons.

Testing different methods for estimating uncertainties on Rietveld refined parameters using SrRietveld

Abstract The advent of fast computing allows more computationally expensive, though possibly more accurate, approaches for estimating uncertainties on Rietveld refined parameters. Here we compare three such methods, using two different refinement programs, FullProf and GSAS. This is facilitated by the use of a new Rietveld refinement package, SrRietveld, that provides Python wrappers for FullProf and GSAS. The refined values on the parameters from two different refinement engines match each other very well. The uncertainty estimates determined using the different methods are also consistent, though FullProf and GSAS estimate the parameter uncertainties slightly differently from each other, which is discussed. More importantly, we find that the refined results are very sensitive to the statistical weights used in the least squares equation. Different weights, for example from uncertain or incorrectly propagated random errors on the data, lead to significantly different refined values. This means that uncertainty estimates on refined parameters should be increased when the random errors on the data are not well known, as for example in many cases where area detectors are used, and care should be taken to propagate errors correctly in any data preprocessing such as normalization by a smoothed spectrum. The computationally expensive bootstrap error estimation methods are facilitated by the use of SrRietveld, but are not required in most cases though may be useful in some cases, for example if the random errors on the data are not well known.

Statistical measures for validating plant genotype similarity assessments following multivariate analysis of metabolome fingerprint data

Metabolome fingerprinting offers opportunities for ‘first pass’ evaluation of compositional similarity between plant genotypes. Compositional “substantial equivalence” testing is a popular concept in the literature in relation to food safety; however reported studies do not provide a systematic and standard approach to quantify similarity in a high dimensional data context. We have undertaken a large scale screen of Arabidopsis genotypes for evidence that individual genetic modifications effect plant phenotype at the level of the metabolome. From this study we propose pragmatic alternative measures that could in the future be used to assess substantial equivalence in GM foods under realistic data paucity constraints and without prior feature selection. Evaluation of classifier accuracy in supervised data mining approaches by bootstrap error estimation provided a robust tool for model validation. Receiver operating characteristics (such as AUC) provide an alternative measure of predictive ability by displaying the relationship between sensitivity and specificity. Additional specific measures based on scatter matrices and sample margins have also been investigated. We illustrate the application of such metrics on a large metabolic profiling data set derived from analysis of 27 genetically distinct Arabidopsis thaliana mutants. We show that agreement exists between model margins, eigenvalue, accuracies and AUC characteristics produced by three different classifiers (Random Forest, Support Vector Machine and Linear Discriminant Analysis). Comparisons between mutants with no observable phenotypic differences to the parent ecotype provided a baseline for model significance metrics; whilst comparison of mutants with increasingly distinct phenotypic alterations generated predictable changes in these measures of similarity.

Reverse Engineering Galactose Regulation in Yeast through Model Selection

We examine the application of statistical model selection methods to reverse-engineering the control of galactose utilization in yeast from DNA microarray experiment data. In these experiments, relationships among gene expression values are revealed through modifications of galactose sugar level and genetic perturbations through knockouts. For each gene variable, we select predictors using a variety of methods, taking into account the variance in each measurement. These methods include maximization of log-likelihood with Cp, AIC, and BIC penalties, bootstrap and cross-validation error estimation, and coefficient shrinkage via the Lasso.

Bolstered error estimation

We propose a general method for error estimation that displays low variance and generally low bias as well. This method is based on “bolstering” the original empirical distribution of the data. It has a direct geometric interpretation and can be easily applied to any classification rule and any number of classes. This method can be used to improve the performance of any error-counting estimation method, such as resubstitution and all cross-validation estimators, particularly in small-sample settings. We point out some similarities shared by our method with a previously proposed technique, known as smoothed error estimation. In some important cases, such as a linear classification rule with a Gaussian bolstering kernel, the integrals in the bolstered error estimate can be computed exactly. In the general case, the bolstered error estimate may be computed by Monte-Carlo sampling; however, our experiments show that a very small number of Monte-Carlo samples is needed. This results in a fast error estimator, which is in contrast to other resampling techniques, such as the bootstrap. We provide an extensive simulation study comparing the proposed method with resubstitution, cross-validation, and bootstrap error estimation, for three popular classification rules (linear discriminant analysis, k-nearest-neighbor, and decision trees), using several sample sizes, from small to moderate. The results indicate the proposed method vastly improves on resubstitution and cross-validation, especially for small samples, in terms of bias and variance. In that respect, it is competitive with, and in many occasions superior to, bootstrap error estimation, while being tens to hundreds of times faster. We provide a companion web site, which contains: (1) the complete set of tables and plots regarding the simulation study, and (2) C source code used to implement the bolstered error estimators proposed in this paper, as part of a larger library for classification and error estimation, with full documentation and examples. The companion web site can be accessed at the URL http://ee.tamu.edu/~edward/bolster.

Selection bias in gene extraction on the basis of microarray gene-expression data

In the context of cancer diagnosis and treatment, we consider the problem of constructing an accurate prediction rule on the basis of a relatively small number of tumor tissue samples of known type containing the expression data on very many (possibly thousands) genes. Recently, results have been presented in the literature suggesting that it is possible to construct a prediction rule from only a few genes such that it has a negligible prediction error rate. However, in these results the test error or the leave-one-out cross-validated error is calculated without allowance for the selection bias. There is no allowance because the rule is either tested on tissue samples that were used in the first instance to select the genes being used in the rule or because the cross-validation of the rule is not external to the selection process; that is, gene selection is not performed in training the rule at each stage of the cross-validation process. We describe how in practice the selection bias can be assessed and corrected for by either performing a cross-validation or applying the bootstrap external to the selection process. We recommend using 10-fold rather than leave-one-out cross-validation, and concerning the bootstrap, we suggest using the so-called .632+ bootstrap error estimate designed to handle overfitted prediction rules. Using two published data sets, we demonstrate that when correction is made for the selection bias, the cross-validated error is no longer zero for a subset of only a few genes.

Speaker Normalization and Model Selection of Combined Neural Networks

This paper introduces bootstrap error estimation for automatic tuning of parameters in combined networks, applied as front-end preprocessors for a speech recognition system based on hidden Markov models. The method is evaluated on a large-vocabulary (10 000 words) continuous speech recognition task. Bootstrap estimates of minimum mean squared error allow selection of speaker normalization models improving recognition performance. The procedure allows a flexible strategy for dealing with inter-speaker variability without requiring an additional validation set. Recognition results are compared for linear, generalized radial basis functions and multi-layer perceptron network architectures.

Bootstrapping shear-wave splitting errors

Abstract We have developed a bootstrap method to estimate errors associated with inverting SKS waveforms for shear-wave splitting parameters. Although presented for shear-wave splitting inversions, this method is suitable for any waveform inversion procedure. The bootstrap error estimation method consists of multiple inversions of simulated data that imitate the original data with differing noise sequences. The results of the bootstrap inversions are used to directly calculate variances and covariances for all model parameters. We employ a bootstrap error estimation technique to nonlinear inversion for shear-wave splitting parameters. Since seismic data have correlated errors, the bootstrap method was modified for stationary bandlimited time series. This modified bootstrap method was applied to shear-wave splitting measurements from over 60 pairs of horizontal seismograms. The method is stable under a large range of noise conditions. By using this bootstrap method, we can distinguish among data with no apparent splitting, data with splitting, and noisy data.

Estimation of Error Rates by Means of Simulated Bootstrap Distributions

AbstractThe bootstrap error estimation method is investigated in comparison with the known π‐method and with a combined error estimation suggested by us using simulated and normally distributed “populations” in 15 and 30 characters, respectively. For small sample sizes (below the double to threefold number of characters per class) the estimates resulting from the bootstrap method are on the average too small and can no longer be accepted. Significantly better results (with an essentially lower calculation expenditure) are obtained for the π‐method and the combined estimation. The variability is essentially the same for all the three methods.This applies both in the case of rather badly separated and in the case of very well separated populations.A bootstrap estimation modified by us also gives unsatisfactory results.