Resampling Plans Research Articles

Analyzing nested experimental designs-A user-friendly resampling method to determine experimental significance.

While hierarchical experimental designs are near-ubiquitous in neuroscience and biomedical research, researchers often do not take the structure of their datasets into account while performing statistical hypothesis tests. Resampling-based methods are a flexible strategy for performing these analyses but are difficult due to the lack of open-source software to automate test construction and execution. To address this, we present Hierarch, a Python package to perform hypothesis tests and compute confidence intervals on hierarchical experimental designs. Using a combination of permutation resampling and bootstrap aggregation, Hierarch can be used to perform hypothesis tests that maintain nominal Type I error rates and generate confidence intervals that maintain the nominal coverage probability without making distributional assumptions about the dataset of interest. Hierarch makes use of the Numba JIT compiler to reduce p-value computation times to under one second for typical datasets in biomedical research. Hierarch also enables researchers to construct user-defined resampling plans that take advantage of Hierarch's Numba-accelerated functions.

A causal bootstrap

The bootstrap, introduced by The Jackknife, the Bootstrap and Other Resampling Plans ((1982), SIAM), has become a very popular method for estimating variances and constructing confidence intervals. A key insight is that one can approximate the properties of estimators by using the empirical distribution function of the sample as an approximation for the true distribution function. This approach views the uncertainty in the estimator as coming exclusively from sampling uncertainty. We argue that for causal estimands the uncertainty arises entirely, or partially, from a different source, corresponding to the stochastic nature of the treatment received. We develop a bootstrap procedure for inference regarding the average treatment effect that accounts for this uncertainty, and compare its properties to that of the classical bootstrap. We consider completely randomized and observational designs as well as designs with imperfect compliance.

Optimization of Evidence Analysis Cost Using Arbitrary Re-Sampling Techniques for Sample Influx into Forensic Science Laboratory

This study analyzes the sample influx (samples per case file) into forensic science laboratory (FSL) and the corresponding analysis costs and uses arbitrary re-sampling plans to establish the minimum cost function. The demand for forensic analysis increased for all disciplines, especially biology/DNA between 2014 and 2015. While the average distribution of case files was about 42.5%, 40.6% and 17% for the three disciplines, the distribution of samples was rather different being 12%, 82.5% and 5.5% for samples requiring forensic biology, chemistry and toxicology analysis, respectively. Results show that most of the analysis workload was on forensic chemistry analysis. The cost of analysis for case files and the corresponding sample influx varied in the ratio of 35:6:1 and 28:12:1 for forensic chemistry, biology/DNA and toxicology for year 2014 for 2015, respectively. In the two consecutive years, the cost for forensic chemistry analysis was comparatively very high, necessitating re-sampling. The time series of sample influx in all disciplines are strongly stochastic, with higher magnitude for chemistry, biology/DNA and toxicology, in this order. The PDFs of sample influx data are highly skewed to the right, especially forensic toxicology and biology/DNA with peaks at 1 and 3 samples per case file. The arbitrary re-sampling plans were best suited to forensic chemistry case files (where re-sampling conditions apply). The locus of arbitrary number of samples to take from the submitted forensic samples was used to establish the minimum and scientifically acceptable samples by applying minimization function developed in this paper. The cost minimization function was also developed based on the average cost per sample and choice of re-sampling plans depending on the range of sample influx, from which the savings were determined and maximized. Thus, the study gives a forensic scientist a business model and scientific decision making tool on minimum number of samples to analyze focusing on savings on analysis cost.

Cramér-Von Mises Statistic for Repeated Measures

The Cramer-von Mises criterion is employed to compare whether the marginal distribution functions of a k-dimensional random variable are equal or not. The well-known Donsker invariance principle and the Karhunen-Loeve expansion is used in order to derive its asymptotic distribution. Two different resampling plans (one based on permutations and the other one based on the general bootstrap algorithm, gBA) are also considered to approximate its distribution. The practical behaviour of the proposed test is studied from a Monte Carlo simulation study. The statistical power of the test based on the Cramer-von Mises criterion is competitive when the underlying distributions are different in location and is clearly better than the Friedman one when the sole difference among the involved distributions is the spread or the shape. Both resampling plans lead to similar results although the gBA avoids the usual required interchangeability assumption. Finally, the method is applied on the study of the evolution inequality incomes distribution between some European countries along the years 2000 and 2011.

Area under the ROC curve comparison in the presence of missing data

The area under the receiver operating characteristic (ROC) curve (AUC) is broadly accepted and often used as a diagnostic accuracy index. Moreover, the equality among the predictive capacity of two or more diagnostic systems is frequently checked from the comparison of their respective AUCs. In paired designs, this comparison is usually performed by using only the subjects who have collected all the necessary information, in the so-called available-case analysis. On the other hand, the presence of missing data is a frequent problem, especially in retrospective and observational studies. The loss of statistical power and the misuse of the available information (with the resulting ethical implications) are the main consequences. In this paper a non-parametric method is developed to exploit all available information. In order to approximate the distribution for the proposed statistic, the asymptotic distribution is computed and two different resampling plans are studied. In addition, the methodology is applied to a real-world medical problem. Finally, some technical issues are also reported in the Appendix.

General nonparametric ROC curve comparison

Although the equality among two or more ROC (receiver operating characteristic) curves is usually contrasted from the respective AUCs (area under the ROC curve), two different ROC curves can share the same AUC and, in order to compare the ROC curves equality, most general criteria must be considered. In this paper, the authors deal with the general ROC curve comparison problem on paired design. They extend the tests for the classical cumulative distribution functions (CDF) comparison to the ROC curves context. To approximate the statistic distribution, two different resampling plans are considered; the usual one based on permutations and a new bootstrap procedure which does not require the exchangeability assumption. As usual, from Monte Carlo simulations, the performance of the proposed methodology is studied for two traditional tests; one based on the Kolmogorov–Smirnov criteria and the other one on the L2-measure. The observed results suggest that the proposed bootstrap provides a good statistic distribution approximation for medium sample size. Therefore the studied methodology allows us to compare the equality of ROC curves by defining a criteria according to the needs of the problem.

A Multicenter Study of [-2]Pro-Prostate Specific Antigen Combined With Prostate Specific Antigen and Free Prostate Specific Antigen for Prostate Cancer Detection in the 2.0 to 10.0 ng/ml Prostate Specific Antigen Range

Prostate specific antigen and free prostate specific antigen have limited specificity to detect clinically significant, curable prostate cancer, leading to unnecessary biopsy, and detection and treatment of some indolent tumors. Specificity to detect clinically significant prostate cancer may be improved by [-2]pro-prostate specific antigen. We evaluated [-2]pro-prostate specific antigen, free prostate specific antigen and prostate specific antigen using the formula, ([-2]pro-prostate specific antigen/free prostate specific antigen × prostate specific antigen(1/2)) to enhance specificity to detect overall and high grade prostate cancer. We enrolled 892 men with no history of prostate cancer, normal rectal examination, prostate specific antigen 2 to 10 ng/ml and 6-core or greater prostate biopsy in a prospective multi-institutional trial. We examined the relationship of serum prostate specific antigen, free-to-total prostate specific antigen and the prostate health index with biopsy results. Primary end points were specificity and AUC using the prostate health index to detect overall and Gleason 7 or greater prostate cancer on biopsy compared with those of free-to-total prostate specific antigen. In the 2 to 10 ng/ml prostate specific antigen range at 80% to 95% sensitivity the specificity and AUC (0.703) of the prostate health index exceeded those of prostate specific antigen and free-to-total prostate specific antigen. An increasing prostate health index was associated with a 4.7-fold increased risk of prostate cancer and a 1.61-fold increased risk of Gleason score greater than or equal to 4 + 3 = 7 disease on biopsy. The AUC of the index exceeded that of free-to-total prostate specific antigen (0.724 vs 0.670) to discriminate prostate cancer with Gleason 4 or greater + 3 from lower grade disease or negative biopsy. Prostate health index results were not associated with age and prostate volume. The prostate health index may be useful in prostate cancer screening to decrease unnecessary biopsy in men 50 years old or older with prostate specific antigen 2 to 10 ng/ml and negative digital rectal examination with minimal loss in sensitivity.

Use of Combined Shewhart‐CUSUM Control Charts for Ground Water Monitoring Applications

Statistical properties of combined Shewhart-CUSUM control charts are examined in terms of the site-wide false positive rate, false negative rate, and average run length (i.e., the average number of sampling events that it takes to detect a significant exceedance). The effects of multiple comparisons (i.e., multiple monitoring wells and constituents), verification resampling, background sample size, and background updating are incorporated into the analysis. The effects of (1) verification resampling plans including no resampling, pass one resample, pass one of two resamples, and pass two of two resamples; (2) background sample sizes (n = 8,12,16,20, and 24); and (3) multiple comparisons (k = 1 to 500 future comparisons) were examined. In addition, various alternative control limit factors (h = SCL= 4.5, 5.0,5.5, and 6.0) were considered, where the control limit is x + hs. Results of simulations revealed that it is possible to balance site-wide false positive and false negative rates at low levels using combined Shewhart-CUSUM control charts. The results revealed that it is important to select the appropriate multiplier, verification resampling plan, number of background samples, and number of intervening samples prior to updating background.

The precision of data obtained in large‐scale science assessments: An investigation of bootstrapping and half‐sample replication methods

Powerful computational procedures are now available to better determine the accuracy of statistical estimates derived from data that have unknown distributions or do not meet parametric requirements. These techniques are generally called resampling plans and include the recently developed bootstrap. Science educators often face the problem of nonnormal distributions especially when conducting large-scale assessments or evaluating national curriculum projects that require complex sampling plans. Resampling techniques permit the researcher to make inferences without the strong distributional assumptions needed for more traditional parametric approaches. In this study, the bootstrap and a simplified version of a half-sample replication are used to examine the precision of science test scores obtained in a large-scale evaluation of Scope, Sequence, and Coordination, a national science curriculum project. The resampling plans are described in some detail and the results are compared with those obtained from parametric methods. © 1998 John Wiley & Sons, Inc. J Res Sci Teach 35: 697–704, 1998.

A bootstrap for point processes

A nonparametric bootstrap for stationary point processes with weak dependence is proposed. Two variations of the bootstrap are described. The methods are block resampling plans which are similar in spirit to an algorithm which has been successfully applied to short-memory time series. The validity of the bootstrap for estimating first and second order intensity measures of certain Poisson cluster processes is investigated by simulation. The first order intensity bootstrap is proven to be asymptotically correct. The method is applied to some meteorological data.

Ground Water Monitoring Statistics Update: Part II: Nonparametric Prediction Limits

AbstractNonparametric prediction limits can be useful statistical tools for ground water monitoring at facilities regulated under RCRA Subtitle C. Subtitle D. and similar regulations. New, exact tables arc presented for both “1 of m” plans (m chances to gel one observation inbounds at each of r monitoring wells to avoid a statistically significant increase) and “California” plans (first or all of the next m‐1 observations inbounds at each well). The tables provide per‐constituent significance levels (false positive rates) as a function of the background sample size n. m. r, the prediction limit (the largest or the next to largest, background observation), and the confirmatory resampling plan selected.When used in a monitoring program, future observations from several wells are compared with a prediction limit obtained from a common background sample. The table significance levels therefore depend critically on having IID (independent and identically distributed) observations. In particular, the false positive rate computations are not valid, and the procedures should not be used, with constituents whose measurements exhibit inherent spatial or systematic temporal variability.Recent U.S. EPA guidance explicitly encourages controlling facility‐wide false positive rates over both constituents and wells. Nonparametric prediction limits, particularly with California resampling plans, will have greater difficulty in meeting the new. lower per‐constituent false positive rate goals than previous ones, especially if many constituents are involved. Nonetheless, nonparametric prediction limits remain superior to other commonly used procedures for dealing with data with high proportions of nondctects.

Adjusting for Multiplicity of Statistical Tests in the Analysis of Carcinogenicity Studies

AbstractBecause the evaluation of rodent carcinogenicity studies involves performing a statistical analysis at each tumor site encountered it is important to understand the extent to which this multiplicity affects the false positive rate. It is equally important to apply methods of accounting for this multiplicity in the analysis.In this paper we discuss one such method which involves calculating the overall significance level associated with P1, the most extreme isolated trend P‐value observed among the tumor sites encountered. The method constructs the distribution of trend scores simultaneously for each tumor site using a multiresponse randomization procedure. As such, it recognizes the discrete nature of the data and incorporates inherent dependencis that may exist between the tumor sites.For small studies it is possible to perform a complete rerandomization and compute an exact adjusted trend P‐value. However, for moderate or large studies the need exists for approximations based on efficient resampling plans. We report one such approximation proposed by Dr. John Tukey which involves correcting the exact Bonferroni upper bound. Also, we show that the independence assumption used in methods proposed by Mantel (1980) and Mantel et al. (1982) seems to be a reasonable approximation for the study discussed in the present report. This result needs to be supported further using additional studies.

Bootstrapping the Kaplan—Meier Estimator

Abstract Randomly censored data consist of iid pairs of observations (Xi, δi), i = 1, …, n; if δ i = 0, Xi denotes a censored observation, and if δ i = 1, Xi denotes an exact “survival” time, which is the variable of interest. For estimating the distribution F of the survival times, the product-limit estimator proposed by Kaplan and Meier (1958) has been studied extensively and it has been shown to enjoy a number of optimality properties (Wellner 1982). See Gill (1980, chaps. 1–4; 1983) for a modern treatment and for references. With censored data, bootstrapping can be carried out using two different resampling plans introduced by Efron (1981) and Reid (1981), respectively. With Efron's plan one takes a random sample with replacement from (X 1), …, (Xn , δ n ), whereas with Reid's plan one takes a random sample from the Kaplan—Meier estimator. The purpose of this article is to study the asymptotic behavior of the bootstrapped Kaplan—Meier estimator with both resampling plans. The approach adopted uses the...

Book Review of the Jackknife, the Bootstrap and Other Resampling Plans by Bradley Efron

Click to increase image sizeClick to decrease image sizeKey Words and Phrases: JackknifeBootstrapDistribution FreeResampling Plans

The Jackknife, the Bootstrap and Other Resampling Plans.

The Jackknife, the Bootstrap and Other Resampling Plans.