Dynamic allocation optimization in A/B tests using classification-based preprocessing

Abstract

An <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">A/B-Test</small> evaluates the impact of a new technology by running it in a real production environment and testing its performance on a set of items. Recent development efforts around <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">A/B-Tests</small> revolve around dynamic allocation. They allow for quicker determination of the best variation (A or B), thus saving money for the user. However, dynamic allocation by traditional methods requires certain assumptions, which are not always valid in reality. This is often due to the fact that the populations being tested are not homogeneous. This article reports on a new reinforcement learning methodology which has been deployed by the commercial <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">A/B-Test</small> platform AB Tasty. We provide a new method that not only builds homogeneous groups of users, but also allows the best variation for these groups to be found in a short period of time. This article provides numerical results on AB Tasty's data, in addition to public datasets, tha demonstrate an improvement over traditional methods.