Abstract
Motivated by customer loyalty plans and scholarship programs, we study tie-breaker designs which are hybrids of randomized controlled trials (RCTs) and regression discontinuity designs (RDDs). We quantify the statistical efficiency of a tie-breaker design in which a proportion $\Delta $ of observed subjects are in the RCT. In a two line regression, statistical efficiency increases monotonically with $\Delta $, so efficiency is maximized by an RCT. We point to additional advantages of tie-breakers versus RDD: for a nonparametric regression the boundary bias is much less severe and for quadratic regression, the variance is greatly reduced. For a two line model we can quantify the short term value of the treatment allocation and this comparison favors smaller $\Delta $ with the RDD being best. We solve for the optimal tradeoff between these exploration and exploitation goals. The usual tie-breaker design applies an RCT on the middle $\Delta $ subjects as ranked by the assignment variable. We quantify the efficiency of other designs such as experimenting only in the second decile from the top. We also show that in some general parametric models a Monte Carlo evaluation can be replaced by matrix algebra.
Highlights
Airlines, hotels and other companies may offer incentives such as free upgrades to their most loyal customers
A natural choice in this context is the regression discontinuity design (RDD) but that has the disadvantage of only estimating a causal impact right at the threshold point separating treated from untreated study subjects
We work primarily with a model in which there are two linear regressions, one for treatment and one for control. This model was used by Goldberger (1972) and Jacob et al (2012), who both find randomized controlled trials (RCTs) more efficient than RDDs, and we think it is the simplest one in which the tradeoff we study is interesting
Summary
Hotels and other companies may offer incentives such as free upgrades to their most loyal customers. We ordinarily expect that our outcome variable will show the greatest gains if we give the treatment to the highest ranked subjects and a tie-breaker design will reduce those gains. If p(x) satisfies p(−x) = 1 − p(x), as with a symmetric CDF, in a two line model both the information gained and the value from the experimental subjects in any sliding scale can be attained by tie-breaker design using only levels. Reducing the data set that way would greatly increase the variance of both the RDD and tie-breaker designs. In this example, the efficiency ratio between the two approaches is almost unchanged.
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a call
