Counterfactual Outcomes Research Articles

Sports stadiums and local economic activity: Evidence from sales tax collections

ABSTRACT Though most studies find that sports stadiums are not strong drivers of economic activity in metropolitan areas, localized development effects may be sufficient to justify public subsidies for a host municipality if circumstances are favorable. This analysis examines the economic ramifications of an intra-metropolitan area relocation of Atlanta’s professional baseball team from a traditional standalone downtown stadium to a new stadium-anchored mixed-use development in suburban Cobb County. Using the synthetic control method, the study employs metro-Atlanta counties to construct a counterfactual outcome for estimating changes in sales tax revenue after the ballpark opened. The findings indicate a net increase in taxable sales in the county; however, the magnitude of the effect is small and not statistically significant. Though net new spending is evident, approximately one-third of the project’s sales appear to derive from crowding out other local economic activity. In total, added tax collections fall well short of covering the public subsidies provided by Cobb. The stadium’s limited economic impact, despite its favorable location and ancillary mixed-use development, further supports past findings that sports venues are poor investments as economic development projects.

Sufficient dimension reduction for average causal effect estimation

A large number of covariates can have a negative impact on the quality of causal effect estimation since confounding adjustment becomes unreliable when the number of covariates is large relative to the number of samples. Propensity score is a common way to deal with a large covariate set, but the accuracy of propensity score estimation (normally done by logistic regression) is also challenged by the large number of covariates. In this paper, we prove that a large covariate set can be reduced to a lower dimensional representation which captures the complete information for adjustment in causal effect estimation. The theoretical result enables effective data-driven algorithms for causal effect estimation. Supported by the result, we develop an algorithm that employs a supervised kernel dimension reduction method to learn a lower dimensional representation from the original covariate space, and then utilises nearest neighbour matching in the reduced covariate space to impute the counterfactual outcomes to avoid the large sized covariate set problem. The proposed algorithm is evaluated on two semisynthetic and three real-world datasets and the results show the effectiveness of the proposed algorithm.

Without Commitment to an Ontology, There Could Be No Causal Inference.

Without Commitment to an Ontology, There Could Be No Causal Inference.

Parental feeding and childhood genetic risk for obesity: exploring hypothetical interventions with causal inference methods

BackgroundParental-feeding behaviors are common intervention targets for childhood obesity, but often only deliver small changes. Childhood BMI is partly driven by genetic effects, and the extent to which parental-feeding interventions can mediate child genetic liability is not known. Here we aim to examine how potential interventions on parental-feeding behaviors can mitigate some of the association between child genetic liability and BMI in early adolescence, using causal inference methods.MethodsData from the Avon Longitudinal Study of Parents and Children were used to estimate an interventional disparity measure for a child polygenic score for BMI (PGS-BMI) on BMI at 12 years. The approach compares counterfactual outcomes for different hypothetical interventions on parental-feeding styles applied when children are 10–11 years (n = 4248). Results are presented as adjusted total association (Adj-Ta) between genetic liability (PGS-BMI) and BMI at 12 years, versus the interventional disparity measure-direct effect (IDM-DE), which represents the association that would remain, had we intervened on parental-feeding under different scenarios.ResultsFor children in the top quintile of genetic liability, an intervention shifting parental feeding to the levels of children with lowest genetic risk, resulted in a difference of 0.81 kg/m2 in BMI at 12 years (Adj-Ta = 3.27, 95% CI: 3.04, 3.49; versus IDM-DE = 2.46, 95% CI: 2.24, 2.67).ConclusionsFindings suggest that parental-feeding interventions have the potential to buffer some of the genetic liability for childhood obesity. Further, we highlight a novel way to analyze potential interventions for health conditions only using secondary data analyses, by combining methodology from statistical genetics and social epidemiology.

Observational Study of the Effect of the Juvenile Stay-At-Home Order on SARS-CoV-2 Infection Spread in Saline County, Arkansas

We assess the treatment effect of juvenile stay-at-home orders (JSAHO) on reducing the rate of SARS-CoV-2 infection spread in Saline County (”Saline”), Arkansas, by examining the difference between Saline’s and control Arkansas counties’ changes in daily and mean log infection rates of pretreatment (March 28–April 5, 2020) and treatment periods (April 6–May 6, 2020). Synthetic control county is constructed based on the parallel-trends assumption, least-squares fitting on pretreatment and socio-demographic covariates, and elastic-net-based methods, from which the counterfactual outcome is predicted and the treatment effect is estimated using the difference-in-differences, the synthetic control, and the changes-in-changes methodologies. Both the daily and average treatment effects of JSAHO are shown to be significant. Despite its narrow scope and lack of enforcement for compliance, JSAHO reduced the rate of the infection spread in Saline.

Optimal dynamic treatment regime estimation using information extraction from unstructured clinical text.

The wide-scale adoption of electronic health records (EHRs) provides extensive information to support precision medicine and personalized health care. In addition to structured EHRs, we leverage free-text clinical information extraction (IE) techniques to estimate optimal dynamic treatment regimes (DTRs), a sequence of decision rules that dictate how to individualize treatments to patients based on treatment and covariate history. The proposed IE of patient characteristics closely resembles "The clinical Text Analysis and Knowledge Extraction System" and employs named entity recognition, boundary detection, and negation annotation. It also utilizes regular expressions to extract numerical information. Combining the proposed IE with optimal DTR estimation, we extract derived patient characteristics and use tree-based reinforcement learning (T-RL) to estimate multistage optimal DTRs. IE significantly improved the estimation in counterfactual outcome models compared to using structured EHR data alone, which often include incomplete data, data entry errors, and other potentially unobserved risk factors. Moreover, including IE in optimal DTR estimation provides larger study cohorts and a broader pool of candidate tailoring variables. We demonstrate the performance of our proposed method via simulations and an application using clinical records to guide blood pressure control treatments among critically ill patients with severe acute hypertension. This joint estimation approach improves the accuracy of identifying the optimal treatment sequence by 14-24% compared to traditional inference without using IE, based on our simulations over various scenarios. In the blood pressure control application, we successfully extracted significant blood pressure predictors that are unobserved or partially missing from structuredEHR.

Late-action effect: Heightened counterfactual potency and perceived outcome reversibility make actions closer to a definitive outcome seem more causally impactful

We report a consistent effect in the evaluation of actions: later actions – specifically, actions that are closer to a final, positive outcome - are considered as contributing more to that outcome, compared to earlier actions. Twelve experiments (total n = 5658, six pre-registered, with U.S. American and British participants, manipulating action timing both within-subjects and between-subjects) provide evidence in support of a late-action effect. This effect extends to different domains, from sports to business to academics. Further, this effect is disproportionately larger when actions are closer to the end of an event. We identify two mediators. The first is a heightened tendency to imagine negative alternative scenarios when the action is closer to an outcome, that is, higher counterfactual potency for later actions. The second is perceived outcome reversibility – lower for later actions, when there is less time for the provisional outcome to change. The effect is mitigated if people are instructed to imagine counterfactual outcomes (consistent with the notion of a counterfactual potency mediation), and if the action does not produce a provisional outcome change. We discuss theoretical implications for the psychological understanding of timing, causality attributions, and counterfactual thinking, as well as future research directions for goal-gradient (the tendency to give more weight to the last steps towards goal completion).

Do social protection cash transfers reduce poverty in Rwanda? Evidence from an econometric analysis of Vision Umurenge Program Direct Support

AbstractCountries across the world continue to invest in social protection programs with the goal of ending poverty by 2030. It is important to examine these programs to know whether they are serving the purpose of reducing poverty. In Rwanda, the impact of the Vision Umurenge Program (VUP) regarding poverty‐related outcomes has not been consistently revealed. This leads to uncertainty about the program's effectiveness in achieving the stated goals. This paper intends to fill this gap by estimating the impact of the VUP Direct Support on poverty outcomes. It uses the Propensity Score Matching method to compare the effects of the VUP with a counterfactual outcome. The findings show that the program is benefitting households in response to economic risks. However, the scheme is not lifting them above the national poverty lines. The main contribution of the paper is that it uses a novel method using the latest statistics to evaluate the VUP. In addition, it uses the most recent household data. The study provides a significant contribution to evaluation of the effectiveness of anti‐poverty interventions in Rwanda, as well as the Sub‐Saharan region, where the journey towards achieving the goal of reducing poverty is demanding.

Locational decisions and subjective well-being: an empirical study of Chinese urban migrants

ABSTRACT Motivated by the popularity of ‘Run away from Beijing, Shanghai, and Guangzhou’ saying in China this paper tests the determinants of subjective well-being (SWB) of long-term migrants and the extent to which migrants make optimal locational decisions in terms of SWB. China’s cities are divided into 4 core first-tier cities, 15 new first-tier cities, and non-first-tier cities (94 in the sample) reflecting different levels of economic opportunity and amenities. Migrants can only be observed in their present location. Hence, we use a semi-parametric marginal treatment effect method to compare actual and counterfactual SWB outcomes. Results shows that migrants’ SWB is greater in non-first-tier cities than first-tier cities; the determinants of SWB differ for male and female migrants; and, female migrants with urban hukou are typically better off in the four core first-tier cities as well as other first-tier cities. Overall, there is thus a marked gender differences in migrants’ optimal locational choice. From a productivity and social cohesion perspective, these results make it imperative that public policy in China addresses the gender and spatial sorting imbalances that arises from differences in SWB determinants and optimal location as ever more educated young people join the labour force.

Instrumental variable estimation for a time-varying treatment and a time-to-event outcome via structural nested cumulative failure time models

BackgroundIn many applications of instrumental variable (IV) methods, the treatments of interest are intrinsically time-varying and outcomes of interest are failure time outcomes. A common example is Mendelian randomization (MR), which uses genetic variants as proposed IVs. In this article, we present a novel application of g-estimation of structural nested cumulative failure models (SNCFTMs), which can accommodate multiple measures of a time-varying treatment when modelling a failure time outcome in an IV analysis.MethodsA SNCFTM models the ratio of two conditional mean counterfactual outcomes at time k under two treatment strategies which differ only at an earlier time m. These models can be extended to accommodate inverse probability of censoring weights, and can be applied to case-control data. We also describe how the g-estimates of the SNCFTM parameters can be used to calculate marginal cumulative risks under nondynamic treatment strategies. We examine the performance of this method using simulated data, and present an application of these models by conducting an MR study of alcohol intake and endometrial cancer using longitudinal observational data from the Nurses’ Health Study.ResultsOur simulations found that estimates from SNCFTMs which used an IV approach were similar to those obtained from SNCFTMs which adjusted for confounders, and similar to those obtained from the g-formula approach when the outcome was rare. In our data application, the cumulative risk of endometrial cancer from age 45 to age 72 under the “never drink” strategy (4.0%) was similar to that under the “always ½ drink per day” strategy (4.3%).ConclusionsSNCFTMs can be used to conduct MR and other IV analyses with time-varying treatments and failure time outcomes.

Multi-agent modeling of crowd dynamics under mass shooting cases

Terrorist attacks lead to huge social cost, and, especially, mass shooting cases result in massive deaths and injuries worldwide. Hence, it is necessary to explore the accurate and dynamic processes of mass shooting cases. Taking the Cinema Shooting case in 2012 as the real target case, the agent-based model is applied to explore the dynamics of human behaviors under common risk of shooting. Parameter traverse and repeated simulations can be used to obtain robust outcomes. According to real target case, the optimal solution with the highest matchiness can be obtained, which supports the validity and robustness of our agent-based model. Under this optimal solution, the same numbers of deaths and injuries can be achieved, and the dynamic process can be also back-calculated. Besides, we can infer counterfactual outcomes, which strengthens explanatory capability of our model. Hence, the multiple outcomes under different situations (settings or strategies) can be obtained, and optimal perception range R* can be solved as well. Combining agent-based model and counterfactual simulations, emergency responses, public governance, and facility designs will be improved.

Telescope Matching for Reducing Model Dependence in the Estimation of the Effects of Time-Varying Treatments: An Application to Negative Advertising

Abstract Time-varying treatments are prevalent in the social sciences. For example, a political campaign might decide to air attack ads against an opponent, but this decision to go negative will impact polling and, thus, future campaign strategy. If an analyst naively applies methods for point exposures to estimate the effect of earlier treatments, this would lead to post-treatment bias. Several existing methods can adjust for this type of time-varying confounding, but they typically rely on strong modelling assumptions. In this paper, we propose a novel two-step matching procedure for estimating the effect of two-period treatments. This method, telescope matching, reduces model dependence without inducing post-treatment bias by using matching with replacement to impute missing counterfactual outcomes. It then employs flexible regression models to correct for bias induced by imperfect matches. We derive the asymptotic properties of the telescope matching estimator and provide a consistent estimator for its variance. We illustrate telescope matching by investigating the effect of negative campaigning in US Senate and gubernatorial elections. Using the method, we uncover a positive effect on turnout of negative ads early in a campaign and a negative effect of early negativity on vote shares.

The Role of the Natural Course in Causal Analysis.

The average causal effect compares counterfactual outcomes if everyone had been exposed versus if everyone had been unexposed, which can be an unrealistic contrast. Alternatively, we can target effects that compare counterfactual outcomes against the factual outcomes observed in the sample (i.e., we can compare against the natural course). Here, we demonstrate how the natural course can be estimated and used in causal analyses for model validation and effect estimation. Our example is an analysis assessing the impact of taking aspirin on pregnancy, 26 weeks after randomization, in the Effects of Aspirin in Gestation and Reproduction trial (United States, 2006-2012). To validate our models, we estimated the natural course using g-computation and then compared that against the observed incidence of pregnancy. We observed good agreement between the observed and model-based natural courses. We then estimated an effect that compared the natural course against the scenario in which participants assigned to aspirin always complied. If participants had always complied, there would have been 5.0 (95% confidence interval: 2.2, 7.8) more pregnancies per 100 women than was observed. It is good practice to estimate the natural course for model validation when using parametric models, but whether one should estimate a natural course contrast depends on the underlying research questions.

Excessive entry and investment in deregulated markets: Evidence from the electricity sector

The staggered and partial restructuring of state electricity markets beginning in 1996 is leveraged to identify the effect of entry deregulation on investment in generation capacity during the U.S. Gas Boom. Using synthetic control to construct the counterfactual outcome, this paper finds restructuring led to a 17 percent increase in capacity beyond the counterfactual in the seven years following restructuring. The overinvestment of the electricity industry is consistent with market coordination failure, contagion and low real interest rates, leading to an environmental welfare gain of $27.6 billion but overall net welfare loss of $18.85 billion.

Ranking intersecting distribution functions

SummarySecond‐degree dominance has become a widely accepted criterion for ordering distribution functions according to social welfare. However, it provides only a partial ordering, and it may fail to rank distributions that intersect. To rank intersecting distribution functions, we propose a general approach based on rank‐dependent theory. This approach avoids making arbitrary restrictions or parametric assumptions about social welfare functions and allows researchers to identify the weakest set of assumptions needed to rank distributions according to social welfare. Our approach is based on two complementary sequences of nested dominance criteria. The first (second) sequence extends second‐degree stochastic dominance by placing more emphasis on differences that occur in the lower (upper) part of the distribution. The sequences characterize two separate systems of nested subfamilies of rank‐dependent social welfare functions. This allows us to identify the least restrictive rank‐dependent social preferences that give an unambiguous ranking of a given set of distribution functions. We also provide an axiomatization of the sequences of dominance criteria and the corresponding subfamilies of social welfare functions. We show the usefulness of our approach using two empirical applications; the first assesses the welfare implications of changes in household income distributions over the business cycle, while the second performs a social welfare comparison of the actual and counterfactual outcome distributions from a policy experiment.

Restricted sub-tree learning to estimate an optimal dynamic treatment regime using observational data.

Dynamic treatment regimes (DTRs), consisting of a sequence of tailored treatment decision rules that span multiple stages of care, present a unique opportunity in our drive toward personalized medicine. Given that estimation of optimal DTRs is often exploratory and communication with clinicians is vital, robust and flexible methods that yield interpretable results are needed. Tree-based methods utilizing a purity measure defined on the full set of covariates have enjoyed much success in meeting this goal. Often, however, it is necessary for clinical, practical, or ethical reasons to restrict certain covariates that should be used when making treatment decisions. Herein we present restricted sub-tree learning (ReST-L), a flexible and robust, sub-tree-based method to estimate an optimal multi-stage multi-treatment DTR that enables restrictions to the set of prespecified candidate tailoring variables. ReST-L employs a purity measure derived from an augmented inverse probability weighted estimator for the counterfactual mean outcome, using observational data to build multi-stage decision trees that are restricted in sub-tree spaces defined by the corresponding prescriptive covariates. We show that ReST-L is able to correctly estimate the optimal DTR searching over a large number of variables with relatively small sample sizes and improves upon competing estimation methods. We demonstrate the utility of ReST-L to estimate a two-stage fluid resuscitation strategy for patients admitted to an intensive care unit with acute emergent sepsis.

RNN-Based Counterfactual Prediction, With an Application to Homestead Policy and Public Schooling

Abstract This paper proposes a method for estimating the effect of a policy intervention on an outcome over time. We train recurrent neural networks (RNNs) on the history of control unit outcomes to learn a useful representation for predicting future outcomes. The learned representation of control units is then applied to the treated units for predicting counterfactual outcomes. RNNs are specifically structured to exploit temporal dependencies in panel data and are able to learn negative and non-linear interactions between control unit outcomes. We apply the method to the problem of estimating the long-run impact of US homestead policy on public school spending.

A counterfactual simulation model of causation by omission

When do people say that an event that did not happen was a cause? We extend the counterfactual simulation model (CSM) of causal judgment (Gerstenberg, Goodman, Lagnado, & Tenenbaum, 2021) and test it in a series of three experiments that look at people's causal judgments about omissions in dynamic physical interactions. The problem of omissive causation highlights a series of questions that need to be answered in order to give an adequate causal explanation of why something happened: what are the relevant variables, what are their possible values, how are putative causal relationships evaluated, and how is the causal responsibility for an outcome attributed to multiple causes? The CSM predicts that people make causal judgments about omissions in physical interactions by using their intuitive understanding of physics to mentally simulate what would have happened in relevant counterfactual situations. Prior work has argued that normative expectations affect judgments of omissive causation. Here we suggest a concrete mechanism of how this happens: expectations affect what counterfactuals people consider, and the more certain people are that the counterfactual outcome would have been different from what actually happened, the more causal they judge the omission to be. Our experiments show that both the structure of the physical situation as well as expectations about what will happen affect people's judgments.

Measuring for Success: Evaluating Leadership Training Programs for Sustainable Impact.

Background:In an era of global health security challenges such as the COVID-19 pandemic, there is greater need for strong leadership. Over the past decades, significant investments have been made in global health leadership development programs by governments and philanthropic organizations to address this need. Evaluating the societal impact of these programs remains challenging, despite consensus on the importance of public health leadership.Objective:This article identifies the gaps and highlights the critical role of monitoring and evaluation approaches in assessing the impact of global health leadership programs. Importantly, we also propose the theory of change (TOC) as a common framework and identify a set of tools and indicators that leadership programs can adapt and use.Methods:We carried out an informal review of major global health leadership programs, including a literature review on leadership program evaluation approaches. Current practices in assessing the short- to long-term outcomes of leadership training programs were explored and synthesized. We also examined use of program theory frameworks, such as theory of change to guide the evaluation strategy. We find the TOC approach can be enhanced by integrating evaluation-specific frameworks and establishing broad stakeholder buy-in. We highlight measurement challenges, proposed outcome indicators and evaluation methodologies, and outline the future direction for such efforts.Findings:Most evaluation of current leadership programs is focused on short-term individual-level outcomes, while reports on long-term societal impact were limited. Reciprocal impacts on and benefits for the “host” organizations were not included in evaluation metrics. Most programs had program logic or result chains, but with no well-articulated program theories.Conclusion:Key stakeholders involved in leadership training programs benefit from the evidence of rigorous program evaluations to inform decisions that address barriers in fostering global health leadership and improving population health outcomes. Insight into reciprocal change in host organizations is important. Evaluation of global health leadership training must go beyond the individual trainee and encompass organizational and community-level impacts. Documentation of long-lasting organizational and societal impacts is essential for donors to appreciate the return on their investment.Key TakeawaysEvaluation plays an important role in understanding how leadership development takes place and how it contributes to improving public health outcomes.Making the case for investments in leadership development programs requires robust evidence from monitoring and evaluation strategies that link investments beyond the individual-level to longer-term societal impacts.The first critical step towards a strategy for success is for leadership programs to clearly build, articulate, share, and use their program theories or theories of change.Theories of change help identify the pathways (and potential tensions) through which leadership development programs effect change at the individual, organizational and community levels.Evaluation methods that examine outcomes of leadership programs should be multi-method, multi-level, and where possible include counterfactual outcomes.Allocation of funds to evaluate on-going and long-lasting societal impact of leadership programs should be a routine practice.

Does city lockdown prevent the spread of COVID-19? New evidence from the synthetic control method

BackgroundAt 10 a.m. on January 23, 2020 Wuhan, China imposed a 76-day travel lockdown on its 11 million residents in order to stop the spread of COVID-19. This lockdown represented the largest quarantine in the history of public health and provides us with an opportunity to critically examine the relationship between a city lockdown on human mobility and controlling the spread of a viral epidemic, in this case COVID-19. This study aims to assess the causal impact of the Wuhan lockdown on population movement and the increase of newly confirmed COVID-19 cases.MethodsBased on the daily panel data from 279 Chinese cities, our research is the first to apply the synthetic control approach to empirically analyze the causal relationship between the Wuhan lockdown of its population mobility and the progression of newly confirmed COVID-19 cases. By using a weighted average of available control cities to reproduce the counterfactual outcome trajectory that the treated city would have experienced in the absence of the lockdown, the synthetic control approach overcomes the sample selection bias and policy endogeneity problems that can arise from previous empirical methods in selecting control units.ResultsIn our example, the lockdown of Wuhan reduced mobility inflow by approximately 60 % and outflow by about 50 %. A significant reduction of new cases was observed within four days of the lockdown. The increase in new cases declined by around 50% during this period. However, the suppression effect became less discernible after this initial period of time. A 2.25-fold surge was found for the increase in new cases on the fifth day following the lockdown, after which it died down rapidly.ConclusionsOur study provided urgently needed and reliable causal evidence that city lockdown can be an effective short-term tool in containing and delaying the spread of a viral epidemic. Further, the city lockdown strategy can buy time during which countries can mobilize an effective response in order to better prepare. Therefore, in spite of initial widespread skepticism, lockdowns are likely to be added to the response toolkit used for any future pandemic outbreak.