When web personalization misleads bucket testing

Abstract

Online service platforms like search engines, news websites, information portal sites and others offer highly personalized content to the users based on their interest and taste. At the same time, these online sites, with large audiences, frequently use bucket testing to evaluate the impact of a new feature or service on a small subset of its users before releasing it to the entire user population. In general, web personalization leads to an improved user engagement for the sites, but it can also interfere and adversely impact the online bucket testing experiments. In this work, we show empirically through real experiments conducted on Yahoo pages that how personalization can mislead to erroneous interpretation of the bucket testing results. We also present a novel algorithmic framework that addresses this challenge and draws a more accurate inference from the bucket testing results by factoring in the personalization experience of the users. The effectiveness of our algorithm is demonstrated through experiments conducted on Yahoo pages.