Stable Unit Treatment Value Assumption Research Articles

Does college selectivity reduce obesity? A partial identification approach.

We use data from the National Longitudinal Study of Adolescent to Adult Health to investigate whether the quality of tertiary education -measured by college selectivity-causally affects obesity prevalence in the medium run (by age 24-34) and in the longer run (about 10years later). We use partial identification methods, which allow us, while relying on weak assumptions, to overcome the potential endogeneity of college selectivity as well as the potential violation of the stable unit treatment value assumption due to students interacting with each other, and to obtain informative identification regions for the average treatment effect of college selectivity on obesity. We find that attending a more selective college causally reduces obesity, both in the medium and in the longer run. We provide evidence that the mechanisms through which the impact of college selectivity on obesity operates include an increase in income, a reduction in physical inactivity and in the consumption of fast food and sweetened drinks.

Learning Individual Treatment Effects under Heterogeneous Interference in Networks

Estimating individual treatment effects in networked observational data is a crucial and increasingly recognized problem. One major challenge of this problem is violating the Stable Unit Treatment Value Assumption (SUTVA), which posits that a unit’s outcome is independent of others’ treatment assignments. However, in network data, a unit’s outcome is influenced not only by its treatment (i.e., direct effect) but also by the treatments of others (i.e., spillover effect) since the presence of interference. Moreover, the interference from other units is always heterogeneous (e.g., friends with similar interests have a different influence than those with different interests). In this paper, we focus on the problem of estimating individual treatment effects (including direct effect and spillover effect) under heterogeneous interference in networks. To address this problem, we propose a novel Dual Weighting Regression (DWR) algorithm by simultaneously learning attention weights to capture the heterogeneous interference from neighbors and sample weights to eliminate the complex confounding bias in networks. We formulate the learning process as a bi-level optimization problem. Theoretically, we give a generalization error bound for the expected estimation error of the individual treatment effects. Extensive experiments on four benchmark datasets demonstrate that the proposed DWR algorithm outperforms the state-of-the-art methods in estimating individual treatment effects under heterogeneous network interference.

Reducing Interference Bias in Online Marketplace Experiments Using Cluster Randomization: Evidence from a Pricing Meta-experiment on Airbnb

Online marketplace designers frequently run randomized experiments to measure the impact of proposed product changes. However, given that marketplaces are inherently connected, total average treatment effect (TATE) estimates obtained through individual-level randomized experiments may be biased because of violations of the stable unit treatment value assumption, a phenomenon we refer to as “interference bias.” Cluster randomization (i.e., the practice of randomizing treatment assignment at the level of “clusters” of similar individuals) is an established experiment design technique for countering interference bias in social networks, but it is unclear ex ante if it will be effective in marketplace settings. In this paper, we use a meta-experiment or “experiment over experiments” conducted on Airbnb to both provide empirical evidence of interference bias in online marketplace settings and assess the viability of cluster randomization as a tool for reducing interference bias in marketplace TATE estimates. Results from our meta-experiment indicate that at least 20% of the TATE estimate produced by an individual-level randomized evaluation of the platform fee increase we study is attributable to interference bias and eliminated through the use of cluster randomization. We also find suggestive, nonstatistically significant evidence that interference bias in seller-side experiments is more severe in demand-constrained geographies and that the efficacy of cluster randomization at reducing interference bias increases with cluster quality. This paper was accepted by Chris Forman, information systems. Supplemental Material: The online appendix and data files are available at https://doi.org/10.1287/mnsc.2020.01157 .

Experimental Design in Marketplaces

Classical Randomized Controlled Trials (RCTs), or A/B tests, are designed to draw causal inferences about a population of units, for example, individuals, plots of land or visits to a website. A key assumption underlying a standard RCT is the absence of interactions between units, or the stable unit treatment value assumption (Ann. Statist. 6 (1978) 34–58). Modern experimentation, however, is often conducted in settings characterized by complex interactions between units. Such interactions can invalidate the standard estimators and make classical experimental designs ineffective. Although the presence of interference forces us to make untestable assumptions on the nature of the interactions even under randomization, sophisticated experimental designs can ameliorate the dependence on such assumptions. In this manuscript, we review the recent and rapidly growing literature on novel experimental designs for these settings. One key feature common to many of these designs is the presence of multiple layers of randomization within the same experiment. We discuss a novel experimental design, called Multiple Randomization Designs or MRDs, that provides a general framework for such experiments. Through these complex designs, we can study questions about causal effects in the presence of interference that cannot be answered by classical RCTs.

Identifying and estimating effects of sustained interventions under parallel trends assumptions.

Many research questions in public health and medicine concern sustained interventions in populations defined by substantive priorities. Existing methods to answer such questions typically require a measured covariate set sufficient to control confounding, which can be questionable in observational studies. Differences-in-differences rely instead on the parallel trends assumption, allowing for some types of time-invariant unmeasured confounding. However, most existing difference-in-differences implementations are limited to point treatments in restricted subpopulations. We derive identification results for population effects of sustained treatments under parallel trends assumptions. In particular, in settings where all individuals begin follow-up with exposure status consistent with the treatment plan of interest but may deviate at later times, a version of Robins' g-formula identifies the intervention-specific mean under stable unit treatment value assumption, positivity, and parallel trends. We develop consistent asymptotically normal estimators based on inverse-probability weighting, outcome regression, and a double robust estimator based on targeted maximum likelihood. Simulation studies confirm theoretical results and support the use of the proposed estimators at realistic sample sizes. As an example, the methods are used to estimate the effect of a hypothetical federal stay-at-home order on all-cause mortality during the COVID-19 pandemic in spring 2020 in the UnitedStates.

Invited Perspective: The Potential of Potential Outcomes in Air Pollution Epidemiology.

Invited Perspective: The Potential of Potential Outcomes in Air Pollution Epidemiology.

Network A/B Testing: Nonparametric Statistical Significance Test Based on Cluster-Level Permutation

A/B testing is widely used for comparing two versions of a product and evaluating new proposed product features. It is of great importance for decision-making and has been applied as a golden standard in the IT industry. It is essentially a form of two-sample statistical hypothesis testing. Average treatment effect (ATE) and the corresponding p-value can be obtained under certain assumptions. One key assumption in traditional A/B testing is the stable-unit-treatment-value assumption (SUTVA): there is no interference among different units. It means that the observation on one unit is unaffected by the particular assignment of treatments to the other units. Nonetheless, interference is very common in social network settings where people communicate and spread information to their neighbors. Therefore, the SUTVA assumption is violated. Analysis ignoring this network effect will lead to biased estimation of ATE. Most existing works focus mainly on the design of experiment and data analysis in order to produce estimators with good performance in regards to bias and variance. Little attention has been paid to the calculation of p-value. We work on the calculation of p-value for the ATE estimator in network A/B tests. After a brief review of existing research methods on design of experiment based on graph cluster randomization and different ATE estimation methods, we propose a permutation method for calculating p-value based on permutation test at the cluster level. The effectiveness of the method against that based on individual-level permutation is validated in a simulation study mimicking realistic settings.

The causal effect of a timeout at stopping an opposing run in the NBA

In the summer of 2017, the National Basketball Association reduced the number of total timeouts, along with other rule changes, to regulate the flow of the game. With these rule changes it becomes increasingly important for coaches to effectively manage their timeouts. Understanding the utility of a timeout under various game scenarios, for example, during an opposing team’s run, is of the utmost importance. There are two schools of thought when the opposition is on a run: (1) call a timeout and allow your team to rest and regroup, or (2) save a timeout and hope your team can make corrections during play. This paper investigates the credence of these tenets using the Rubin causal model framework to quantify the causal effect of a timeout in the presence of an opposing team’s run. Too often overlooked, we carefully consider the stable unit-treatment-value assumption (SUTVA) in this context and use the SUTVA to motivate our definition of units. To measure the effect of a timeout, we introduce a novel, interpretable outcome based on the score difference to describe broad changes in the scoring dynamics. This outcome is well suited for situations where the quantity of interest fluctuates frequently, a commonality in many sports analytics applications. We conclude from our analysis that, while comebacks frequently occur after a run, it is slightly disadvantageous to call a timeout during a run by the opposing team and further demonstrate that the magnitude of this effect varies by franchise.

Generalized propensity score approach to causal inference with spatial interference.

Many spatial phenomena exhibit interference, where exposures at one location may affect the response at other locations. Because interference violates the stable unit treatment value assumption, standard methods for causal inference do not apply. We propose a new causal framework to recover direct and spill-over effects in the presence of spatial interference, taking into account that exposures at nearby locations are more influential than exposures at locations further apart. Under the no unmeasured confounding assumption, we show that a generalized propensity score is sufficient to remove all measured confounding. To reduce dimensionality issues, we propose a Bayesian spline-based regression model accounting for a sufficient set of variables for the generalized propensity score. A simulation study demonstrates the accuracy and coverage properties. We apply the method to estimate the causal effect of wildland fires on air pollution in the Western United States over2005-2018.

Industry-initiated catch limit management: The case of purse seine fishery in Japan’s EEZ waters of the North Pacific Ocean

The need for effective fishery management is growing worldwide due to the overexploited status of many fishery resources. While co-management between the government and fishers is considered a viable option, the performance of such a management is not well understood. This study describes the historical context, the current situation, and the price changes associated with the mackerel (chub mackerel Scomber japonicus, blue mackerel Scomber australasicus) purse seine fishery management in Japan’s EEZ waters of the North Pacific Ocean. This fishery is unusual in that an industry organization plays a major role in managing the total allowable catch (TAC) set by the government. Notably, the organization voluntarily introduced vessel-level monthly catch limits (CL) in 2007. This co-management system has been effective in adhering to the TAC. Moreover, we examine the price changes associated with the CL management by using difference-in-differences design where the prices in ports where vessels under the CL management land and where they do not are compared before and after 2007. Results indicate that the introduction of this management is associated with an increase in their mackerel price by 21.335 yen/kg (s.e. = 10.397), which virtually eliminated the price gaps with mackerel in other regions in Japan. Controlling for income, export price, and time trends do not change our result quantitatively. However, further study is needed to clarify the causal impact of the CL management, as the common trend assumption is likely to hold but the stable unit treatment value assumption is likely not in our setting.

Strategic complementarities, geographical agglomeration, and firm investment

We estimate the effect of strategic complementarities (measured by the geographical agglomeration of firms) on firm investment. Individual firm investment responds significantly to industry investment in agglomerated industries, whereas the response is null in dispersed industries. Industry-region sales do not explain the differential effect between these two types of industries. These results have been overlooked in the literature, provide a justification for regional and industry policies, and challenge the stable unit treatment value assumption (SUTVA) required in potential outcome analysis.

Solving the Reflection Problem in Social Interaction Models with Endogeneity

We identify peer and contextual effects in social interactions models with endogenous covariates (e.g., self-selected treatments). Unobserved individual characteristics are correlated with these endogenous covariates and affect peer outcomes in the reduced form. Our method uses instruments for endogenous covariates, but does not require additional instruments for simultaneity in outcomes, which are often hard to find in linear-in-means models. Nor does it require any exclusion restriction that some covariates have no contextual effects. The method can be applied to relax the Stable Unit Treatment Value Assumption (SUTVA) in the program evaluation literature, allowing individual treatments to influence the outcomes of other group members through both peer and contextual effects. We apply our method to estimate social effects in Grade 3 math scores of elementary school students in the State of Tennessee. Using lagged class sizes, teacher qualifications and students' self-reported motivation scores as instruments for Grade 2 scores, we find significant evidence for positive peer effects and path dependence on G2 scores.

Reassessing Quasi-experiments: Policy Evaluation, Induction, and SUTVA

This paper defends the use of quasi-experiments for causal estimation in economics against the widespread objection that quasi-experimental estimates lack external validity. The defence is that quasi-experimental replication of estimates can yield defeasible evidence for external validity. The paper then develops a different objection. The stable unit treatment value assumption (SUTVA), on which quasi-experiments rely, is argued to be implausible due to the influence of social interaction effects on economic outcomes. A more plausible stable marginal unit treatment value assumption (SMUTVA) is proposed, but it is demonstrated to severely limit the usefulness of quasi-experiments for economic policy evaluation.

Causal inference in collaboration networks using propensity score methods

Using panel data of school-class networks of 11–13-year-old students, this study investigates effects of schoolwork collaboration-networks on grades and school-related well-being. It suggests propensity score weighting-regression as a method of causal inference for data collected in social contexts, and in studies analyzing node-attributes as outcomes of interest. It will argued that this alternative approach is useful when stochastic actor-based models (SAOMs) show convergence problems in sparse networks. Three methods of causal analysis dealing with the problems of endogeneity bias and interference between observations will be discussed in this study: first, SAOMs for the co-evolution of networks and behavior/attitudes will be estimated, but this results in a systematic loss of data. Second, propensity score matching compares treated cases with untreated nearest neighbors. However, the stable-unit-treatment-value assumption (SUTVA) requires that the analysis controls for network embeddedness in the final analysis. This is possible by using propensity score weighting-regression, which is a flexible approach to capture treatment diffusion via multiplex networks.

Reducing Interference Bias in Online Marketplace Pricing Experiments

Online marketplace designers frequently run A/B tests to measure the impact of proposed product changes. However, given that marketplaces are inherently connected, total average treatment effect estimates obtained through Bernoulli randomized experiments are often biased due to violations of the stable unit treatment value assumption. This can be particularly problematic for experiments that impact sellers' strategic choices, affect buyers' preferences over items in their consideration set, or change buyers' consideration sets altogether. In this work, we measure and reduce bias due to interference in online marketplace experiments by using observational data to creating clusters of similar listings, and then using those clusters to conduct cluster-randomized field experiments. We provide a lower bound on the magnitude of bias due to interference by conducting a meta-experiment that randomizes over two experiment designs: one Bernoulli randomized, one cluster randomized. In both meta-experiment arms, treatment sellers are subject to a different platform fee policy than control sellers, resulting in different prices for buyers. By conducting a joint analysis of the two meta-experiment arms, we find a large and statistically significant difference between the total average treatment effect estimates obtained with the two designs, and estimate that 32.60% of the Bernoulli-randomized treatment effect estimate is due to interference bias. We also find weak evidence that the magnitude and/or direction of interference bias depends on extent to which a marketplace is supply- or demand-constrained, and analyze a second meta-experiment to highlight the difficulty of detecting interference bias when treatment interventions require intention-to-treat analysis.

Adjusting for Peer-Influence in Propensity Scoring When Estimating Treatment Effects

Analyses of treatments, experiments, policies, and observational data, are confounded when people's treatment outcomes and/or participation decisions are influenced by those of their friends and acquaintances. This invalidates standard matching techniques as estimation tools. For instance, the vaccination decisions of a person's peers affect the person's choice to vaccinate and the probability that the person is exposed to a disease (violating the usual Stable Unit Treatment Value Assumption). We account for these interferences by explicitly modeling peer interaction in treatment participation decisions, and balancing matchings to overcome correlation in outcomes. We incorporate these interaction effects into one of the most common techniques used to evaluate treatment effects: propensity score matching, and provide asymptotic results. We illustrate that peer-influenced propensity score matching gives more accurate results than standard propensity score matching in the estimation of the effectiveness of vaccinations and estimation of the impact of exercise participation on depression.

Does China’s Municipal Solid Waste Source Separation Program Work? Evidence from the Spatial-Two-Stage-Least Squares Models

This paper evaluates the impact of the second municipal solid waste (MSW) source separation program on municipal solid waste generation (MSWG) in China. Without considering the spatial interactions between cities, the second MSW source separation program has a nonsignificant adverse effect on the per capita municipal solid waste generation (PMSWG). Relaxing the stable-unit-treatment-value assumption (SUTVA), which holds in most of the previous estimation literature on treatment effects, involving the spatial spillover effect among cities, as well as correcting the endogenous local policy has a significantly negative but not robust impact on the PMSWG. The estimation results of the generalized nesting spatial regression models (GNS) imply that the spatial interaction characteristics among Chinese prefecture-level cities may, if neglected, lead to underestimation of the reduction effects of the second MSW source separation policy on the absolute amount of PMSWG. More importantly, our study indicates that although not all the spatial econometric models support the significant reduction effect of source separation on the absolute amount of PMSWG, the source separation program significantly reduces the relative amount of PMSWG, and this result is robust in all spatial models.

Identification of the Average Treatment Effect When SUTVA Is Violated

The stable unit treatment value assumption (SUTVA) ensures that only two potential outcomes exist and that one of them is observed for each individual. After providing new insights on SUTVA validity, we derive sharp bounds on the average treatment effect (ATE) of a binary treatment on a binary outcome as a function of the share of units, a, for which SUTVA is potentially violated. Then we show how to compute the maximum value of a such that the sign of the ATE is still identified. After decomposing SUTVA into two separate assumptions, we provide weaker conditions that might help sharpening our bounds. Furthermore, we show how some of our results can be extended to continuous outcomes. Finally, we estimate our bounds in two well known experiments, the U.S. Job Corps training program and the Colombian PACES vouchers for private schooling.

Considering Network Effects in the Design and Analysis of Field Experiments on State Legislatures

Recent work on legislative politics has documented complex patterns of interaction and collaboration through the lens of network analysis. In a largely separate vein of research, the field experiment—with many applications in state legislatures—has emerged as an important approach in establishing causal identification in the study of legislative politics. The stable unit treatment value assumption (SUTVA)—the assumption that a unit's outcome is unaffected by other units' treatment statuses— is required in conventional approaches to causal inference with experiments. When SUTVA is violated via networked social interaction, treatment effects spread to control units through the network structure. We review recently developed methods that can be used to account for interference in the analysis of data from field experiments on state legislatures. The methods we review require the researcher to specify a spillover model, according to which legislators influence each other, and specify the network through which spillover occurs. We discuss these and other specification steps in detail. We find mixed evidence for spillover effects in data from two previously published field experiments. Our replication analyses illustrate how researchers can use recently developed methods to test for interference effects, and support the case for considering interference effects in experiments on state legislatures.

A Framework for Separating Individual-Level Treatment Effects From Spillover Effects

This article suggests a causal framework for separating individual-level treatment effects and spillover effects such as general equilibrium, interference, or interaction effects related to treatment distribution. We relax the stable unit treatment value assumption assuming away treatment-dependent interaction between study participants and permit spillover effects within aggregates, for example, regions. Based on our framework, we systematically categorize the individual-level and spillover effects considered in the previous literature and clarify the assumptions required for identification under different designs, for instance, based on randomization or selection on observables. Furthermore, we propose a novel difference-in-differences approach and apply it to a policy intervention extending unemployment benefit durations in selected regions of Austria that arguably affected ineligibles in treated regions through general equilibrium effects in local labor markets.