The Upworthy Research Archive, a time series of 32,487 experiments in U.S. media

J Nathan Matias,Charles Ebersole,Kevin Munger,Marianne Aubin Le Quere

doi:10.1038/s41597-021-00934-7

J Nathan Matias, Charles Ebersole + Show 2 more

Open Access

https://doi.org/10.1038/s41597-021-00934-7

Copy DOI

Abstract

The pursuit of audience attention online has led organizations to conduct thousands of behavioral experiments each year in media, politics, activism, and digital technology. One pioneer of A/B tests was Upworthy.com, a U.S. media publisher that conducted a randomized trial for every article they published. Each experiment tested variations in a headline and image “package,” recording how many randomly-assigned viewers selected each variation. While none of these tests were designed to answer scientific questions, scientists can advance knowledge by meta-analyzing and data-mining the tens of thousands of experiments Upworthy conducted. This archive records the stimuli and outcome for every A/B test fielded by Upworthy between January 24, 2013 and April 30, 2015. In total, the archive includes 32,487 experiments, 150,817 experiment arms, and 538,272,878 participant assignments. The open access dataset is organized to support exploratory and confirmatory research, as well as meta-scientific research on ways that scientists make use of the archive.

Highlights

Background & SummaryUpworthy is a media publisher founded in 2012 to reach large audiences with “stuff that matters.” In pursuit of the largest possible audience, Upworthy conducted thousands of randomized behavioral experiments
The Cornell University IRB has determined that this project is not governed by U.S research ethics regulations, since (a) the dataset does not include any information about individual people, and (b) the studies were conducted several years ago by Upworthy alone, with no involvement from us or our institutions
For a single article on Upworthy, writers and editors created multiple variations of headlines, images, and in some cases descriptive text. These stimuli were bundled into a number of “packages” that constituted the arms of each experiment (Fig. 1)

Summary

Introduction

Background & SummaryUpworthy is a media publisher founded in 2012 to reach large audiences with “stuff that matters.” In pursuit of the largest possible audience, Upworthy conducted thousands of randomized behavioral experiments. As a uniquely-large dataset of causal studies, The Upworthy Research Archive can be used to advance fields including communication, political science, psychology, media studies, computer science, marketing, and business. The Cornell University IRB has determined that this project is not governed by U.S research ethics regulations, since (a) the dataset does not include any information about individual people, and (b) the studies were conducted several years ago by Upworthy alone, with no involvement from us or our institutions.

Results

Conclusion