A/B Testing at Vungle

Abstract

Two recently graduated MBA students are tasked with developing an ad-serving learning algorithm for a mobile ad-serving company. The case illustrates the way in which hypotheses can be tested in an A/B format or horse race in order to establish customer preferences and superior profitability. The case was written for a course elective covering hypothesis testing. Excerpt UVA-QA-0821 Rev. Mar. 7, 2017 A/B Testing at Vungle Andrew Kritzer and Hammond Guerin stared at the screen and then at each other. It was June 30, 2014—six weeks since they had graduated from the Darden School of Business. The ad-serving algorithm Kritzer and Guerin had spent six months developing for Vungle, a mobile advertising company, seemed to be outperforming the company's current algorithm. But they did not want to start celebrating too soon. Could their algorithm really deliver the type of improvement they had promised Vungle's CEO? Would install rates of advertised apps really increase? Would Vungle see an increase in ad-serving efficiency as a result? Neither Kritzer nor Guerin could afford for the algorithm to disappoint. Now that he had graduated, Kritzer was headed to LinkedIn, having left a legend among MBA students for his appreciation of data science, tech, and media and raising expectations for what Darden students knew and could learn about data science, analytics, and the ever-growing world of big data. His work on the Vungle project during his second year had received a lot of attention, and he was looking forward to having the results support the effort. Guerin's data science capabilities were also legendary among his MBA peers. He won every school forecasting competition, and his data mining algorithms even beat those of the professional consultants who did classroom visits. Late in his second year, Guerin decided to turn down a generous offer from a well-known consulting firm in favor of an offer from Vungle for an annual salary of $ 100,000 and stock options to serve as the head of Vungle's brand new data science team out in the company's San Francisco headquarters. The job was a dream for the computer scientist turned MBA. He and his wife were already house hunting in the Bay Area, looking for the right place to raise their baby daughter. . . .