How Notification Queues Contain Hidden Natural Experiments

Abstract

Randomized experiments, or A/B tests are used to estimate the causal impact of a feature on the behavior of users by creating two parallel universes in which members are simultaneously assigned to treatment and control. However, it is not always feasible or desirable to run an experiment due to engineering costs, or concerns related to the user experience or ethics of randomizing the presence of a feature. Naturally occurring exogenous variation, or 'natural experiments,' allow researchers to recover causal estimates of peer effects from observational data in the absence of experimental manipulation. Natural experiments trade off the engineering costs and some of the ethical concerns associated with randomization with the search costs of finding instrumental variables. To mitigate the search costs associated with discovering natural counterfactuals, we identify a common engineering requirement used to scale massive online systems, in which natural exogenous variation is likely to exist: notification queuing. We identify a natural experiment on the LinkedIn platform based on the order of notification queues to estimate the causal peer effect of a received message on the engagement of a message recipient (based on work anniversary announcements distributed to a member's connections). We show that receiving a message from another member significantly increases a member's engagement, but that some popular observational specifications, such as fixed-effects estimators, overestimate this effect by as much as 2.7x. The study points to the benefits of using notification queues to discover naturally occurring counterfactuals for the estimation of causal effects without experimenter intervention or the engineering costs associated with online randomization to different feature sets. It also implies a potential benefit of involving data scientists in the system design process to maximize the informational benefits of software systems.