Use Of Randomization Tests Research Articles

Regulatory Guidance on Randomization and the Use of Randomization Tests in Clinical Trials: A Systematic Review

Randomization is a critical aspect of the clinical trial. Regulatory guidance plays a role in clinical research. Randomization tests provide valid methodology for statistical inference. In the present paper, we intended to survey: (a) information contained in regulatory guidance on randomization; and (b) the use of randomization tests in trials supporting marketing applications. This systematic review used the Cortellis Regulatory Intelligence database (IDRAC). For (a), of n = 156 guidance documents, nine provided recommendations on randomization methods. For (b), n = 48 trials (52 submissions) employed randomization tests. 40 (83.3%) were phase III trials, 31 (64.6%) employed dynamic allocation. Randomization test was the primary in 5 (10.4%), and sensitivity analysis in 34 (70.8%) trials. Furthermore, randomization tests were performed when treatment allocation was not according to plan (4 (8.3%)), or violations of model assumptions (5 (10.4%)) were observed. Overall, there are sparse recommendations on randomization in current regulatory guidance, with little mention of recently developed methods which might outperform current practice in certain contexts. Greater adoption of such methods in pharmaceutical industry trials may lead to a better shared sense between regulators and trial sponsors on their appropriate use. Randomization tests are employed with greater frequency, reflecting increased appreciation of the validity and value of randomization tests.

Responses of β‐glucosidase, permanganate oxidizable carbon, and fluorescein diacetate hydrolysis to conservation practices

AbstractResponsiveness of biochemical soil quality indicators (SQIs) (β‐glucosidase activity [BG], fluorescein diacetate hydrolysis [FDA], and permanganate oxidizable C [POXC]) to management and their interactions with site and experimental factors were evaluated using a meta‐database to quantify effect sizes (ESs). Eighty percent of responses were positive, with ESs ranking grassland > organic fertilizer > conservation tillage = cover crop ≥ residue return exceeding conventional controls. The ESs were, on average, larger for BG (0.34) and FDA (0.35) than POXC (0.29), and responses to site and experimental covariates varied. This work found that POXC ES was greater in soils that are drier and finer‐textured, increased with study length, varied under high and low biomass crops, and increased by use of analytical methods that recovered a larger proportion of SOC. These findings collectively suggest that POXC quantifies accumulation of protected C. Both BG and FDA were more responsive in wetter and medium‐ or coarse‐textured soils, suggesting these enzyme‐based indicators are closely associated with biologically active C. The ESs of FDA were the most spatially and temporally dynamic, exhibiting significant (P < .05) declines with depth and sampling time. Use of randomization tests and general linear models to quantify SQI response to management effectively identified significant covariates needed to understand differences among SQI responses to conservation management. This work demonstrated the importance of considering site and experimental covariates and a model for how to interpret SQI responsiveness using meta‐analytical summary.

A Randomized Case Series Approach to Testing Efficacy of Interventions for Minimally Verbal Autistic Children.

BackgroundRandomized Controlled Trials (RCTs) are the gold standard for assessing whether an intervention is effective; however, they require large sample sizes in order to detect small effects. For rare or complex populations, we advocate a case series approach as a more realistic and useful first step for intervention evaluation. We consider the importance of randomization to such designs, and advocate for the use of Randomization Tests and Between Case Effect Sizes to provide a robust and statistically powerful evaluation of outcomes. In this tutorial, we describe the method, procedures, and analysis code necessary to conduct robust single case series, using an empirical example with minimally verbal autistic children.MethodWe applied a pre-registered (https://osf.io/9gvbs) randomized baseline design with between-case effect size to a case series (n = 19), to test the efficacy of a novel, parent-mediated, app-based speech production intervention (BabbleBooster) for minimally verbal autistic children. Parent-rated probe scores were used to densely sample performance accuracy over time.ResultsParents were able to reliably code their children’s speech productions using BabbleBooster. A non-significant Randomization Test and small Between-Case Effect Size (d = 0.267), suggested there was no evidence that BabbleBooster improved speech production in minimally verbal autistic children, relative to baseline scores, during this brief period of intervention.ConclusionThe current analyses exemplify a more robust approach to examining treatment effects in rare or complex populations, where RCT may be difficult or premature to implement. To facilitate adoption of this method by researchers and practitioners, we provide analysis code that can be adapted using open source R packages. Future studies could use this case series design to evaluate interventions aiming to improve speech and language outcomes for minimally verbal autistic children, and other heterogeneous and hard to reach populations.

Randomization tests for changing criterion designs

Randomization tests for alternating treatments designs, multiple baseline designs, and withdrawal/reversal designs are well-established. Recent classifications, however, also mention the “changing criterion design” as a fourth important type of single-case experimental design. In this paper, we examine the potential of randomization tests for changing criterion designs. We focus on the rationale of the randomization test, the random assignment procedure, the choice of the test statistic, and the calculation of randomization test p-values. Two examples using empirical data and an R computer program to perform the calculations are provided. We discuss the problems associated with conceptualizing the changing criterion design as a variant of the multiple baseline design, the potential of the range-bound changing criterion design, experimental control as an all-or-none phenomenon, the necessity of random assignment for the statistical-conclusion validity of the randomization test, and the use of randomization tests in nonrandomized designs.

Randomized single-case AB phase designs: Prospects and pitfalls.

Single-case experimental designs (SCEDs) are increasingly used in fields such as clinical psychology and educational psychology for the evaluation of treatments and interventions in individual participants. The AB phase design, also known as the interrupted time series design, is one of the most basic SCEDs used in practice. Randomization can be included in this design by randomly determining the start point of the intervention. In this article, we first introduce this randomized AB phase design and review its advantages and disadvantages. Second, we present some data-analytical possibilities and pitfalls related to this design and show how the use of randomization tests can mitigate or remedy some of these pitfalls. Third, we demonstrate that the Type I error of randomization tests in randomized AB phase designs is under control in the presence of unexpected linear trends in the data. Fourth, we report the results of a simulation study investigating the effect of unexpected linear trends on the power of the randomization test in randomized AB phase designs. The implications of these results for the analysis of randomized AB phase designs are discussed. We conclude that randomized AB phase designs are experimentally valid, but that the power of these designs is sufficient only for large treatment effects and large sample sizes. For small treatment effects and small sample sizes, researchers should turn to more complex phase designs, such as randomized ABAB phase designs or randomized multiple-baseline designs.

On the use of randomization tests following adaptive designs

ABSTRACTRandomization tests (sometimes referred to as “re-randomization” tests) are used in clinical trials, either as an assumption-free confirmation of parametric analyses, or as an independent analysis based on the principle of randomization-based inference. In the context of adaptive randomization, either restricted or response-adaptive procedures, it is unclear how accurate such Monte Carlo approximations are, or how many Monte Carlo sequences to generate. In this paper, we describe several randomization procedures for which there is a known exact or asymptotic distribution of the randomization test. For a special class of procedures, called , and binary responses, the exact test statistic has a simple closed form. For the limited subset of existing procedures with known exact and asymptotic distributions, we can use these as a benchmark for the accuracy of Monte Carlo randomization techniques. We conclude that Monte Carlo tests are very accurate, and require minimal computation time. For simple tests with binary response in the class of procedures, the exact distribution provides the best test, but Monte Carlo approximations can be used when the exact distribution is difficult to compute.

Brain reactivity to emotional, neutral and cigarette‐related stimuli in smokers

Addiction has been described as the pathological usurpation of the neural mechanisms normally involved in emotional processing. Event-related potentials (ERPs) can provide a non-invasive index of neural responses associated with the processing of emotionally relevant stimuli and serve as a tool for examining temporal and spatial commonalities between the processing of intrinsically motivating stimuli and drug cues. Before beginning a smoking cessation program, 116 smokers participated in a laboratory session in which dense-array ERPs (129 sensors) were recorded during the presentation of pictures with emotional (pleasant and unpleasant), neutral and cigarette-related content. ERP differences among categories were analyzed with use of randomization tests on time regions of interest identified by temporal principal component analysis. Both emotional and cigarette-related pictures prompted significantly more positivity than did neutral pictures over central, parietal, and frontal sites in the 452-508 ms time window. During the 212-316 ms time window, both pleasant and cigarette-related pictures prompted less positivity than neutral images did. Cigarette-related pictures enhanced the amplitude of the P1 component (136-144 ms) above the levels measured in the emotional and neutral conditions. These results support the hypothesis that for smokers, cigarette-related cues are motivationally relevant stimuli that capture attentional resources early during visual processing and engage brain circuits normally involved in the processing of intrinsically emotional stimuli.

The Use of Randomization Tests in Single-Subject Research

Some researchers have argued for the use of or have attempted to make use of randomization tests in single-subject research. To address this tide of interest, the authors of this article describe randomization tests, discuss the theoretical rationale for applying them to single-subject research, and provide an overview of the methodological research into the Type I error rates and statistical power of randomization tests in single-subject designs. It is found that randomization tests generally are an awkward fit to single-subject designs, and they are found to have low statistical power in single-subject contexts. It is therefore recommended that single-subject researchers make sparing, if any, use of randomization tests to analyze their data.

Heterozygosity and extra‐pair paternity: biased tests result from the use of shared markers

Recent studies of extra-pair paternity have found support for the idea that heterozygous males have an advantage in siring offspring. Most studies use DNA microsatellite loci to determine paternity and then use the same loci to estimate individual heterozygosity. However, because the likelihood of detecting extra-pair offspring depends on the combinations of parental alleles, it is possible that biases arise from particular allele combinations. This might produce false support for the influence of heterozygosity on mating behaviour. We used a simulation model to assess how large this bias might be. We found two sources of bias. First, we found a bias in the null hypothesis of a simple statistical test commonly used to test several predictions of the heterozygosity hypothesis. The use of randomization tests could eliminate this bias. Second, we found that using the same loci for both paternity and heterozygosity can cause an increase in results supporting the heterozygosity hypothesis when no effect of heterozygosity actually exists. This bias is reduced through the use of more markers with higher levels of polymorphism and heterozygosity, but can be eliminated entirely by using a separate set of markers to determine paternity and assess heterozygosity. The two sources of bias reduce evidence favouring the heterozygosity hypothesis, but do not negate all of the studies that support it. We suggest that further studies of heterozygosity and extra-pair paternity are important and likely to be informative, but our recommendations should be incorporated by researchers to improve the reliability of their conclusions.

Testing multiple-group variance equality with randomization procedures

The operating characteristics of different statistical tests for evaluating multiple-group variance equality were examined using a simulation study. Normal-theory tests including Bartlett's test, Levene's test and Hartley's test were compared to randomization tests in this study. The randomization tests maintained nominal Type I error rates over a variety of conditions. Levene's test was conservative under all conditions examined. Both Bartlett's test and Hartley's test were anti-conservative (rejects more frequently than nominally specified) when the distribution was lighter-tailed than a normal distribution (double exponential) and were conservative when the distribution was heavier-tailed than a normal distribution (uniform). Bartlett's test performed exceptionally well when the distribution was normal. Use of randomization test is recommended as a test of variance equality as this test performed at nominal Type I error rates and exhibited better power properties than Levene's test, the other test that controlled Type I error rates (albeit conservatively).

Randomization tests to identify significant effects in experimental designs for robustness testing

The screening designs applied in robustness tests are usually fractional factorial or Plackett–Burman designs. Different methods to identify significant factor effects estimated from experimental designs for robustness testing are described. In this paper, the use of randomization tests as a statistical interpretation method is examined and compared with both graphical (half-normal probability plot) and statistical methods, such as the estimation of error based on a priori considered negligible effects and the algorithm of Dong. It was found that all statistical methods usually gave similar results, i.e. the same effects are found to be significant. However, sometimes randomization tests indicate either less or more significant factor effects compared to the other methods, regardless the design size. Both randomization tests and the algorithm of Dong become unreliable when about 50% of the examined factors are significant. In such situation, it is advisable to perform an experimental design from which enough negligible effects can be estimated. The graphical interpretation method did not always succeed in indicating the correct number of significant effects.

Randomization tests for time effects and heterogeneity in capture probabilities for closed populations

We propose the use of randomization tests to detect time effects and heterogeneity in capture probabilities of individuals in a closed-population mark-recapture study. The results of a simulation study show the randomization procedures can be implemented so that tests are robust to extraneous sources of variation, with good power to detect the desired effects. We apply the tests to two examples of real data, finding very strong evidence of heterogeneous capture probabilities in both data sets and highly significant time effects for one of the sets.

Randomization Tests: Example Using Morphological Differences in Aphis gossypii (Homoptera: Aphididae)

Morphometric data of Aphis gossypii Glover are used as a case study to illustrate the use of randomization tests. The application of randomization tests in morphological evaluation and identification of species is a powerful tool for characterizing populations and species. It offers the advantage of reducing our reliance on the robustness of more classical approaches to overcome problems of small sample size, unequal sample size, and departures from normality. We review randomization test methodology. We address a few errors that have appeared in the literature. One question is how many randomizations. As a generic starting point, the number of randomizations should be 2 orders of magnitude larger than the inverse of the significant P value, but in critical cases an exact figure can be determined. A new methodology is introduced for using randomization tests to determine if the average of several observations is different from a constant. An extension of the method is used when the null hypothesis states that there are differences. This is important where there is reason to suspect that one is dealing with different populations (e.g., morphological measurements were taken from several distinct populations of an insect) and one needs to identify which populations are the same. This test should not be confused with the typical case where it is simply impossible to identify differences between different sets of observation. We present a SAS program to perform 2-tailed tests for differences between means.

RTS, a computer program for the experimental set-up and interpretation of ruggedness tests

A computer program is described for the experimental set-up and interpretation of ruggedness tests. The implemented strategy was based on a number of case studies and contains both recommended designs and minimal designs. The minimal designs reduce the number of experiments, but they cannot be statistically interpreted based on the interaction or dummy factor effects. The use of randomization tests as an alternative statistical interpretation method for the significance of the effects was examined. Some of the minimal designs are expandable to designs with characteristics similar to those of the recommended designs. The program is designed to facilitate the selection of the designs and the interpretation of the results and to prevent or detect problems such as drifting of responses.

Pseudorandom test-length analysis using differential solutions

As the size of VLSI circuits increases, the use of random testing is becoming more common. One of the most important aspects of random testing is the determination of the test pattern length that guarantees a high confidence of fault detection. Generally, random test length is estimated by assuming that the set of test patterns applied is purely random. The assumption is not completely correct in applications where linear feedback shift registers (LFSR's) are employed to generate input vectors. In this paper, we have developed a test (Markov) model which faithfully reflects the pseudorandom behavior of test patterns, and all detectable single stuck-at faults (instead of the worst single stuck-fault only) are considered. The required test length is then determined by solving differential equations to achieve the specified test confidence. Based on the test model, analysis is first dedicated to the two-fault case, results are then extended to the k-fault analysis where k/spl ges/3. The test length thus determined is smaller than that derived based on the random pattern assumption, and test costs can be greatly reduced.

Pitfalls in computing and interpreting randomization testp values: A commentary on Chen and Dunlap

Chen and Dunlap (1993) added to the growing list of papers promoting the use of randomization tests in statistical testing. Their particular contribution was an SAS program that could bring computation of these tests to a wider audience. The present paper points to several problems with the presentation of Chen and Dunlap and provides solutions to these problems. It is concluded that randomization tests deserve more attention, but that they are best computed by programs written in a low-level programming language or, if using SAS on a mainframe, by using the MULTTEST procedure.

The spectre of 'spurious' correlations.

Ecologists often 'standardize' data through the use of ratios and indices. Such measures are employed generally to remove a 'size effect' induced by some relatively uniteresting variable. The implications of using the resultant data in correlation and regression analyses are poorly recognized. We show that ratios and indices often provide surprising and 'spurious' results due to their unusual properties. As a solution, we advocate the use of randomization tests to evaluate hypotheses confounded by 'spurious' correlations. In addition, we emphasize that identifying the appropriate null correlation is of utmost importance when statistically evaluating ratios, although this issue is frequently ignored.

STUDIES ON BLOOD ALCOHOL IN THE WORKERS OF A ZAMBIAN COPPER MINE

The results of alcohol studies in workers of a Zambian copper mine showed that 30% of accident cases had measurable blood alcohol levels. A selected group of employees subject to random breath tests before starting work showed that one third had measurable blood alcohol and that this was over 17.6 mmol/l. in 9% of cases. These two groups were not comparable. The results were considered significant enough to justify further study. Two thirds of a group of workers referred to hospital for alcohol tests on suspicion of being under the influence of alcohol had alcohol levels exceeding 35.2 mmol/l. It is concluded that there is a range of significant blood alcohol levels where employees are at increased risk of accidents but have a limited chance of detection by their supervisor and that this circumstance justifies the use of random testing particularly of employees responsible for the safety of others.

Obstacles in Using Randomization Tests in Single-Case Experimentation

Edgington has just proposed the use of randomization tests for single-case experimental designs. The specific tests presented require rapidly alternating treatments over time for a given subject. Although randomization tests may provide viable statistical options, several features of single-case research ordinarily would preclude their use. Characteristics of the interventions and responses that are encompassed by the substantive areas of single-case research are likely to preclude shifting rapidly from one treatment to another on a randomized schedule. Also, the possibility of multiple-treatment interference may be especially acute with randomization tests. Finally, the use of randomization tests may compete with the logic of single-case experimental designs. The present paper discusses both applied and methodological restrictions on the use of randomization tests.

Overcoming Obstacles to Single-Subject Experimentation

Kazdin (this issue) discussed two types of problems supposedly associated with the use of randomization tests for single-subject experiments: the random introduction of treatments and the repeated alternation of treatments. Both types of problems represent obstacles to single-subject experimentation, generally, not just to the determination of significance by the randomization test procedure. Ways to reduce the adverse effects associated with the random introduction of treatments and the repeated alternation of treatments are presented.