A Comparison of Methods for Treatment Assignment with an Application to Playlist Generation

Abstract

This study presents a systematic comparison of methods for individual treatment assignment. We group the various methods proposed in the literature into three general classes of algorithms (or metalearners): learning models to predict outcomes (the O-learner), learning models to predict causal effects (the E-learner), and learning models to predict optimal treatment assignments (the A-learner). We discuss how the metalearners differ in their level of generality and their objective function, which has critical implications for modeling and decision making. Notably, we demonstrate that optimizing for the prediction of outcomes or causal effects is not the same as optimizing for treatment assignments, suggesting that, in general, the A-learner should lead to better treatment assignments than the other metalearners. We then compare the metalearners in the context of choosing, for each user, the best algorithm for playlist generation in order to optimize engagement. This is the first comparison of the three different metalearners on a real-world application at scale (based on more than half a billion treatment assignments). In addition to supporting our analytical findings, the results show how large A/B tests can provide substantial value for learning treatment-assignment policies, rather than simply choosing the variant that performs best on average.