Multi-stage Adaptive Design Research Articles

A case for conducting business-to-business experiments with multi-arm multi-stage adaptive designs

Many businesses conduct experiments to scientifically test, measure, and optimize decisions in areas like sales, marketing, and operations efficiency. While randomized controlled trials (RCTs) or A/B tests are the dominant method for conducting business experiments especially for business-to-consumer marketing, adaptive designs have yet to make extensive inroads outside of the pharmaceutical and medical industries. In this study, we aim to raise awareness of the applicability and advantages of multi-arm multi-stage adaptive designs outside of clinical settings and we use simulations to demonstrate the value of these designs to modern business experiments, with a focus on business-to-business experiments such as testing alternative sales techniques. Our simulation results show that, compared to RCT, multi-arm multi-stage adaptive designs (MAMS) can reduce the sample size requirements and expected time to experiment completion whilst maintaining a similar level of statistical power. We also demonstrate that these benefits can translate into actual cost savings in conjunction with shorter time to market, resulting in higher overall efficiency over the traditional RCTs. MAMS serves as a strong alternative methodology in experiments where not all customers can be contacted at once such as business-to-business campaigns and general live channel programs which typically take weeks to months to complete.

An adaptive multi-stage phase I dose-finding design incorporating continuous efficacy and toxicity data from multiple treatment cycles

ABSTRACT Phase I designs traditionally use the dose-limiting toxicity (DLT), a binary endpoint from the first treatment cycle, to identify the maximum-tolerated dose (MTD) assuming a monotonically increasing relationship between dose and efficacy. In this article, we establish a general framework for a multi-stage adaptive design where we jointly model a continuous efficacy outcome and continuous/quasi-continuous toxicity endpoints from multiple treatment cycles. The normalized Total Toxicity Profile (nTTP) is used as an illustration for quasi-continuous toxicity endpoints, and we replace DLT with nTTP to take into account multiple grades and types of toxicities. In addition, the proposed design accommodates non-monotone dose-efficacy relationships, and longitudinal toxicity data in effort to capture the adverse events from multiple cycles. Stage 1 of our design uses toxicity data to perform dose-escalation and identify a set of initially allowable (safe) doses; stage 2 of our design incorporates an efficacy outcome to update the set of allowable doses for each new cohort and randomizes the new cohort of patients to the allowable doses with emphasis towards those with higher predicted efficacy. Stage 3 uses all data from all treated patients at the end of the trial to make final recommendations. Simulations showed that the design had a high probability of making the correct dose selection and good overdose control across various dose-efficacy and dose-toxicity scenarios. In addition, the proposed design allows for early termination when all doses are too toxic. To our best knowledge, the proposed dual-endpoint dose-finding design is the first such study to incorporate multiple cycles of toxicities and a continuous efficacy outcome.

A proof-of-concept-to-confirmatory multiple adaptation design in the development of an anti-viral treatment.

In the clinical development of some new infectious disease drugs, early clinical pharmacology trials may predict with high confidence that the efficacious doses are well below the range of the safety margin. In this case, a dose-ranging study may be unnecessary after a proof-of-concept (PoC) study testing the highest dose. A multi-stage adaptive design spanning both PoC and confirmatory stages is proposed in this context. The design incorporates two interim analyses allowing strategies for stopping, continuing, or expanding the study. A conditional power threshold for a binary endpoint is proposed to assess futility. Additional components of early efficacy and sample size adjustment are also included to enhance the design's flexibility and robustness. Design operating characteristics are evaluated by numerical calculation. We show that the proposed streamlined trial design has the same statistical rigor as a conventional phase 3 clinical trial with adequate power and a properly controlled type 1 error rate. Additional adaptive design options are also investigated and discussed.

Implementing Optimal Designs for Dose\u2013Response Studies Through Adaptive Randomization for a Small Population Group

In dose–response studies with censored time-to-event outcomes, D-optimal designs depend on the true model and the amount of censored data. In practice, such designs can be implemented adaptively, by performing dose assignments according to updated knowledge of the dose–response curve at interim analysis. It is also essential that treatment allocation involves randomization—to mitigate various experimental biases and enable valid statistical inference at the end of the trial. In this work, we perform a comparison of several adaptive randomization procedures that can be used for implementing D-optimal designs for dose–response studies with time-to-event outcomes with small to moderate sample sizes. We consider single-stage, two-stage, and multi-stage adaptive designs. We also explore robustness of the designs to experimental (chronological and selection) biases. Simulation studies provide evidence that both the choice of an allocation design and a randomization procedure to implement the target allocation impact the quality of dose–response estimation, especially for small samples. For best performance, a multi-stage adaptive design with small cohort sizes should be implemented using a randomization procedure that closely attains the targeted D-optimal design at each stage. The results of the current work should help clinical investigators select an appropriate randomization procedure for their dose–response study.

Twenty-five years of confirmatory adaptive designs: opportunities and pitfalls.

‘Multistage testing with adaptive designs’ was the title of an article by Peter Bauer that appeared 1989 in the German journal Biometrie und Informatik in Medizin und Biologie. The journal does not exist anymore but the methodology found widespread interest in the scientific community over the past 25 years. The use of such multistage adaptive designs raised many controversial discussions from the beginning on, especially after the publication by Bauer and Köhne 1994 in Biometrics: Broad enthusiasm about potential applications of such designs faced critical positions regarding their statistical efficiency. Despite, or possibly because of, this controversy, the methodology and its areas of applications grew steadily over the years, with significant contributions from statisticians working in academia, industry and agencies around the world. In the meantime, such type of adaptive designs have become the subject of two major regulatory guidance documents in the US and Europe and the field is still evolving. Developments are particularly noteworthy in the most important applications of adaptive designs, including sample size reassessment, treatment selection procedures, and population enrichment designs. In this article, we summarize the developments over the past 25 years from different perspectives. We provide a historical overview of the early days, review the key methodological concepts and summarize regulatory and industry perspectives on such designs. Then, we illustrate the application of adaptive designs with three case studies, including unblinded sample size reassessment, adaptive treatment selection, and adaptive endpoint selection. We also discuss the availability of software for evaluating and performing such designs. We conclude with a critical review of how expectations from the beginning were fulfilled, and – if not – discuss potential reasons why this did not happen. © 2015 The Authors. Statistics in Medicine Published by John Wiley & Sons Ltd.

Multistage adaptive biomarker-directed targeted design for randomized clinical trials

BackgroundPrecision medicine is changing how patients are treated and how therapies are developed. In recent years, biomarker-directed targeted designs have been developed for pharmaceutical development aimed at patient subpopulation with a specific disease etiology. To integrate multistage testing into the targeted designs enhances flexibility of targeted trials by enabling sequential monitoring and stochastic curtailment. Methods and resultsWe studied a multistage adaptive design for targeted trials with either normally distributed endpoint or binary endpoint. The design is based on the fact that distribution of the sequence of test statistics from multistage testing is asymptotically well approximated by a Brownian motion in targeted trials with normally distributed continuous endpoint or binary endpoint. This study has demonstrated that the targeted multistage design improves study efficiency, information accumulation and conditional power as compared with its untargeted counterpart. Furthermore, our study has indicated that biomarker performance plays a crucial role in efficiency and effectiveness of the multistage adaptive design. The sensitivity and specificity of a biomarker used for patient enrichment influence level of heterogeneity of the targeted study population, and subsequently impact overall trial efficiency and statistical power as well as information accruement and conditional/predictive power for stochastic curtailment. When performance of a biomarker is imperfect, conditional/predictive power at an earlier stage may be over-estimated, resulting invalid early stopping decision. Thus, great care is needed to ensure that biomarker performance is considered in statistical planning of the multistage targeted trials. ConclusionsIn summary, the multistage adaptive design provides targeted trials with flexibility in multistage testing and early stopping while retaining the rigor of the study design.

POTENTIAL IMPACT OF CONTEXT EFFECTS ON THE SCORING AND EQUATING OF THE MULTISTAGE GRE® REVISED GENERAL TEST

ABSTRACTBoth theoretical and practical considerations have led the revision of the Graduate Record Examinations® (GRE®) revised General Test, here called the rGRE, to adopt a multistage adaptive design that will be continuously or nearly continuously administered and that can provide immediate score reporting. These circumstances sharply constrain the available options for equating or linking together the thousands of unique test forms that will need to be administered each year. The only practical method is to pre‐equate each form through item response theory (IRT) methods. Unfortunately, doing so means that item position and context effects are both vitally important and potentially difficult to control. This study examines item position effects in a GRE context and attempts to assess the extent to which they may affect the rGRE. It was found that position effects are in fact present in GRE data and that they can be particularly troublesome under multistage testing. Pretest strategies are able to mitigate effects to some degree, but complete control will require careful attention be paid to the time limits and associated test configuration details in the design of the rGRE.

Determining optimal sample sizes for multi-stage randomized clinical trials using value of information methods

Traditional sample size calculations for randomized clinical trials depend on somewhat arbitrarily chosen factors, such as Type I and II errors. An effectiveness trial (otherwise known as a pragmatic trial or management trial) is essentially an effort to inform decision-making, i.e., should treatment be adopted over standard? Taking a societal perspective and using Bayesian decision theory, Willan and Pinto (Stat. Med. 2005; 24:1791-1806 and Stat. Med. 2006; 25:720) show how to determine the sample size that maximizes the expected net gain, i.e., the difference between the cost of doing the trial and the value of the information gained from the results. These methods are extended to include multi-stage adaptive designs, with a solution given for a two-stage design. The methods are applied to two examples. As demonstrated by the two examples, substantial increases in the expected net gain (ENG) can be realized by using multi-stage adaptive designs based on expected value of information methods. In addition, the expected sample size and total cost may be reduced. Exact solutions have been provided for the two-stage design. Solutions for higher-order designs may prove to be prohibitively complex and approximate solutions may be required. The use of multi-stage adaptive designs for randomized clinical trials based on expected value of sample information methods leads to substantial gains in the ENG and reductions in the expected sample size and total cost.

Benefit–Risk Evaluation of Multi-Stage Adaptive Designs

Under a group sequential design, the conditional power at an interim analysis represents the probability of success in the future given the past data. Because an adequate overall power is not a guarantee of an adequate conditional power, many authors in the literature have proposed multi-stage adaptive designs allowing sample size adjustment. In this paper, we evaluate the consequences of various such adaptive designs. This is achieved through the use of not only a conventional efficiency criterion of expected sample size, but also an effectiveness criterion from a benefit–risk analysis which takes into consideration some constraints associated with the drug development process. We show that with a proper choice of an adaptive design the conditional power can be maintained at a desirable level, which improves the efficiency, and more importantly, effectiveness of a clinical trial.

Multistage adaptive designs for clinical trials

Multistage adaptive designs for clinical trials

Flexible Interim Analyses in Clinical Trials Using Multistage Adaptive Test Designs

Data-dependent interim analyses are a useful tool in confirmatory Phase III or IV clinical research. In particular, redesigning the sample size in an interim analysis based on the results observed to date considerably improves the power of the trial since the best available information at the time is used for the sample size adjustment. In recent years, several methods were proposed that enable a flexible design through the use of adaptive interim analyses while maintaining the type I error rate. In this article, these methods are briefly reviewed. We recommend a strategy that copes well with the demands of practice. Examples illustrate the use of multistage adaptive designs that make it possible to calculate confidence intervals and bias adjusted point estimates.

A combined acoustic and trawl survey for efficiently estimating fish abundance

When fish are patchily distributed, unstratified trawl surveys give highly variable abundance estimates. If the patches are small and mobile, it is impractical to use pre-stratified or multi-stage adaptive designs to reduce variability. Based on recent trawl surveys for mackerel icefish Champsocephalus gunnari, we have modified the survey design and analysis to reduce estimation variance at minimal cost, using concomitant qualitative acoustic data. We have also developed new confidence interval procedures that should be accurate even for small samples. In computer simulations, the new method substantially outperformed a standard trawl survey.