A case for conducting business-to-business experiments with multi-arm multi-stage adaptive designs

Abstract

Many businesses conduct experiments to scientifically test, measure, and optimize decisions in areas like sales, marketing, and operations efficiency. While randomized controlled trials (RCTs) or A/B tests are the dominant method for conducting business experiments especially for business-to-consumer marketing, adaptive designs have yet to make extensive inroads outside of the pharmaceutical and medical industries. In this study, we aim to raise awareness of the applicability and advantages of multi-arm multi-stage adaptive designs outside of clinical settings and we use simulations to demonstrate the value of these designs to modern business experiments, with a focus on business-to-business experiments such as testing alternative sales techniques. Our simulation results show that, compared to RCT, multi-arm multi-stage adaptive designs (MAMS) can reduce the sample size requirements and expected time to experiment completion whilst maintaining a similar level of statistical power. We also demonstrate that these benefits can translate into actual cost savings in conjunction with shorter time to market, resulting in higher overall efficiency over the traditional RCTs. MAMS serves as a strong alternative methodology in experiments where not all customers can be contacted at once such as business-to-business campaigns and general live channel programs which typically take weeks to months to complete.