Estimation Of Average Treatment Effects Research Articles

Multiple robustness estimation in causal inference

Estimation of average treatment effect is crucial in causal inference for evaluation of treatments or interventions in biostatistics, epidemiology, econometrics, sociology. However, existing estimators require either a propensity score model, an outcome vector model, or both is correctly specified, which is difficult to verify in practice. In this paper, we allow multiple models for both the propensity score models and the outcome models, and then construct a weighting estimator based on observed data by using two-sample empirical likelihood. The resulting estimator is consistent if any one of those multiple models is correctly specified, and thus provides multiple protection on consistency. Moreover, the proposed estimator can attain the semiparametric efficiency bound when one propensity score model and one outcome vector model are correctly specified, without requiring knowledge of which models are correct. Simulations are performed to evaluate the finite sample performance of the proposed estimators. As an application, we analyze the data collected from the AIDS Clinical Trials Group Protocol 175.

Regression adjustment for treatment effect with multicollinearity in high dimensions

Randomized experiment is an important tool for studying the Average Treatment Effect (ATE). This paper considers the regression adjustment estimation of the Sample Average Treatment Effect (SATE) in high-dimensional case, where the multicollinearity problem is often encountered and needs to be properly handled. Many existing regression adjustment methods fail to achieve satisfactory performances. To solve this issue, an Elastic-net adjusted estimator for SATE is proposed under the Rubin causal model of randomized experiments with multicollinearity in high dimensions. The asymptotic properties of the proposed SATE estimator are shown under some regularity conditions, and the asymptotic variance is proved to be not greater than that of the unadjusted estimator. Furthermore, Neyman-type conservative estimators for the asymptotic variance are proposed, which yields tighter confidence intervals than both the unadjusted and the Lasso-based adjusted estimators. Some simulation studies are carried out to show that the Elastic-net adjusted method is better in addressing collinearity problem than the existing methods. The advantages of our proposed method are also shown in analyzing the dataset of HER2 breast cancer patients.

Estimating Average Treatment Effects With Propensity Scores Estimated With Four Machine Learning Procedures: Simulation Results in High Dimensional Settings and With Time to Event Outcomes

Background: The increased availability of claims data allows one to build high dimensional datasets, rich in covariates, for accurately estimating treatment effects in medical and epidemiological cohort studies. This paper shows the full potential of machine learning for the estimation of average treatment effects with propensity score methods in a context rich and high dimensional datasets. Methods: Four different methods are used to estimate average treatment effects in the context of time to event outcomes. The four methods explored in this study are LASSO, Random Forest, Gradient Descent Boosting and Artificial Neural networks. Simulations based on an actual medical claims data set are used to assess the efficiency of these methods. The simulations are performed with over 100, 000 observations and 1,100 explanatory variables. Each method is tested on 500 datasets that are created from the original dataset, allowing us to report the mean and standard deviation of estimated average treatment effects. Results: The results are very promising for all four methods; however, LASSO, Random Forest and Gradient Boosting seem to be performing better than Random Forest. Conclusion: Machine Learning methods can be helpful for observational studies that use the propensity score when a very large number of covariates are available, the total number of observations is large, and the dependent event rare. This is an important result given the availability of big data related to Health Economics and Outcomes Research (HEOR) around the world.

Generalizability of heterogeneous treatment effect estimates across samples

The extent to which survey experiments conducted with nonrepresentative convenience samples are generalizable to target populations depends critically on the degree of treatment effect heterogeneity. Recent inquiries have found a strong correspondence between sample average treatment effects estimated in nationally representative experiments and in replication studies conducted with convenience samples. We consider here two possible explanations: low levels of effect heterogeneity or high levels of effect heterogeneity that are unrelated to selection into the convenience sample. We analyze subgroup conditional average treatment effects using 27 original-replication study pairs (encompassing 101,745 individual survey responses) to assess the extent to which subgroup effect estimates generalize. While there are exceptions, the overwhelming pattern that emerges is one of treatment effect homogeneity, providing a partial explanation for strong correspondence across both unconditional and conditional average treatment effect estimates.

How much do conditional cash transfers increase the utilization of maternal and child health care services? New evidence from Janani Suraksha Yojana in India

Janani Suraksha Yojana (safe motherhood scheme, or JSY) provides cash incentives to marginal pregnant women in India conditional on having mainly institutional delivery. Using the fourth round of district level household survey (DLHS-4), we have estimated its effects on both intended and unintended outcomes. Our estimates of average treatment effect on the treated (ATT) from propensity score matching are remarkably higher than those found in previous prominent studies using the second and third rounds of the survey (DLHS-2 and DLHS-3). When we apply fuzzy regression discontinuity design exploiting the second birth order, our estimates of local average treatment effect (LATE) are much higher than that of ATT. For example, due to JSY, institutional delivery increases by around 16 percentage points according to ATT estimate but about 23 percentage points according to LATE estimate.

Developing a Digital Marketplace for Family Planning: Pilot Randomized Encouragement Trial.

BackgroundFamily planning is an effective tool for preventing death among women who do not want to become pregnant and has been shown to improve newborn health outcomes, advance women’s empowerment, and bring socioeconomic benefits through reductions in fertility and population growth. Yet among the populations that would benefit the most from family planning, uptake remains too low. The emergence of digital health tools has created new opportunities to strengthen health systems and promote behavior change. In this study, women with an unmet need for family planning in Western Kenya were randomized to receive an encouragement to try an automated investigational digital health intervention that promoted the uptake of family planning.ObjectiveThe objectives of the pilot study were to explore the feasibility of a full-scale trial—in particular, the recruitment, encouragement, and follow-up data collection procedures—and to examine the preliminary effect of the intervention on contraception uptake.MethodsThis pilot study tested the procedures for a randomized encouragement trial. We recruited 112 women with an unmet need for family planning from local markets in Western Kenya, conducted an eligibility screening, and randomized half of the women to receive an encouragement to try the investigational intervention. Four months after encouraging the treatment group, we conducted a follow-up survey with enrolled participants via short message service (SMS) text message.ResultsThe encouragement sent via SMS text messages to the treatment group led to differential rates of intervention uptake between the treatment and control groups; however, uptake by the treatment group was lower than anticipated (19/56, 33.9% vs 1/56, 1.8%, in the control group). Study attrition was also substantial. We obtained follow-up data from 44.6% (50/112) of enrolled participants. Among those in the treatment group who tried the intervention, the instrumental variables estimate of the local average treatment effect was an increase in the probability of contraceptive uptake of 41.0 percentage points (95% uncertainty interval −0.03 to 0.85).ConclusionsThis randomized encouragement design and study protocol is feasible but requires modifications to the recruitment, encouragement, and follow-up data collection procedures.Trial RegistrationClinicalTrials.gov NCT03224390; https://clinicaltrials.gov/ct2/show/NCT03224390 (Archived by WebCite at http://www.webcitation.org/70yitdJu8)

Design-Based Estimators for Average Treatment Effects for Multi-Armed RCTs

Design-based methods have recently been developed as a way to analyze randomized controlled trial (RCT) data for designs with a single treatment and control group. This article builds on this framework to develop design-based estimators for evaluations with multiple research groups. Results are provided for a wide range of designs used in education research, including clustered and blocked designs. Because analysis in the multi-armed setting involves pairwise contrasts across the research groups, the key methodological question addressed is: How do the estimators for the two-group design need to be adjusted for multi-armed trials? The critical insight is that in multi-armed trials where the goal is to identify the most effective treatments, the samples for each pairwise contrast are representative of the full set of randomized units, not just of themselves. The implication is that variance terms need to be adjusted slightly under the finite-population framework that can reduce precision, and blocks need to be weighted to reflect the full randomized sample in the block or biases can result. An empirical example using data from a multi-armed education RCT demonstrates the issues.

Regression-assisted inference for the average treatment effect in paired experiments

SummaryIn paired randomized experiments, individuals in a given matched pair may differ on prognostically important covariates despite the best efforts of practitioners. We examine the use of regression adjustment to correct for persistent covariate imbalances after randomization, and present two regression-assisted estimators for the sample average treatment effect in paired experiments. Using the potential outcomes framework, we prove that these estimators are consistent for the sample average treatment effect under mild regularity conditions even if the regression model is improperly specified, and describe how asymptotically conservative confidence intervals can be constructed. We demonstrate that the variances of the regression-assisted estimators are no larger than that of the standard difference-in-means estimator asymptotically, and illustrate the proposed methods by simulation. The analysis does not require a superpopulation model, a constant treatment effect, or the truth of the regression model, and hence provides inference for the sample average treatment effect with the potential to increase power without unrealistic assumptions.

Worth Weighting? How to Think About and Use Weights in Survey Experiments

The popularity of online surveys has increased the prominence of using sampling weights to enhance claims of representativeness. Yet, much uncertainty remains regarding how these weights should be employed in survey experiment analysis: should they be used? If so, which estimators are preferred? We offer practical advice, rooted in the Neyman–Rubin model, for researchers working with survey experimental data. We examine simple, efficient estimators, and give formulas for their biases and variances. We provide simulations that examine these estimators as well as real examples from experiments administered online through YouGov. We find that for examining the existence of population treatment effects using high-quality, broadly representative samples recruited by top online survey firms, sample quantities, which do not rely on weights, are often sufficient. We found that sample average treatment effect (SATE) estimates did not appear to differ substantially from their weighted counterparts, and they avoided the substantial loss of statistical power that accompanies weighting. When precise estimates of population average treatment effects (PATE) are essential, we analytically show poststratifying on survey weights and/or covariates highly correlated with outcomes to be a conservative choice. While we show substantial gains in simulations, we find limited evidence of them in practice.

2SLS versus 2SRI: Appropriate methods for rare outcomes and/or rare exposures.

This study used Monte Carlo simulations to examine the ability of the two-stage least squares (2SLS) estimator and two-stage residual inclusion (2SRI) estimators with varying forms of residuals to estimate the local average and population average treatment effect parameters in models with binary outcome, endogenous binary treatment, and single binary instrument. The rarity of the outcome and the treatment was varied across simulation scenarios. Results showed that 2SLS generated consistent estimates of the local average treatment effects (LATE) and biased estimates of the average treatment effects (ATE) across all scenarios. 2SRI approaches, in general, produced biased estimates of both LATE and ATE under all scenarios. 2SRI using generalized residuals minimized the bias in ATE estimates. Use of 2SLS and 2SRI is illustrated in an empirical application estimating the effects of long-term care insurance on a variety of binary health care utilization outcomes among the near-elderly using the Health and Retirement Study.

An alternative robust estimator of average treatment effect in causal inference.

The problem of estimating the average treatment effects is important when evaluating the effectiveness of medical treatments or social intervention policies. Most of the existing methods for estimating the average treatment effect rely on some parametric assumptions about the propensity score model or the outcome regression model one way or the other. In reality, both models are prone to misspecification, which can have undue influence on the estimated average treatment effect. We propose an alternative robust approach to estimating the average treatment effect based on observational data in the challenging situation when neither a plausible parametric outcome model nor a reliable parametric propensity score model is available. Our estimator can be considered as a robust extension of the popular class of propensity score weighted estimators. This approach has the advantage of being robust, flexible, data adaptive, and it can handle many covariates simultaneously. Adopting a dimension reduction approach, we estimate the propensity score weights semiparametrically by using a non-parametric link function to relate the treatment assignment indicator to a low-dimensional structure of the covariates which are formed typically by several linear combinations of the covariates. We develop a class of consistent estimators for the average treatment effect and study their theoretical properties. We demonstrate the robust performance of the estimators on simulated data and a real data example of investigating the effect of maternal smoking on babies' birth weight.

Heterogeneous Treatment Effects and Optimal Targeting Policy Evaluation

We discuss how to construct optimal targeting policies and document the difference in profits from alternative targeting policies by using estimation approaches that are based on recent advances in causal inference and machine learning. We introduce an approach to evaluate the profit of any targeting policy using only one single randomized sample. This approach is qualitatively equivalent to conducting a field test, but reduces the cost of multiple field tests because all comparisons can be conducted in only one sample. The approach allows us to compare many alternative optimal targeting policies that are constructed based on different estimates of the conditional average treatment effect, i.e. the incremental effect of targeting. We draw a conceptual distinction between methods that predict the conditional average treatment effect indirectly via the conditional expectation function trained on the outcome level, and methods that directly predict the conditional average treatment effect. We propose a new direct estimation method, called treatment effect projection. The empirical application is to a catalog mailing with a high-dimensional set of customer features. We find that the optimal targeting policies based on the direct estimation methods typically outperform the indirect estimation methods, both in the validation sets from the same population from which the training set is drawn and in the data obtained one year after the training set was collected. In particular, the treatment effect projection performs similar to the recently introduced causal forest of Wager and Athey (2017). We also compare targeting policies based on conditional average treatment effects with a sophisticated application of the traditional CRM approach that is based on a prediction of the outcome level. Even though based on a conceptually incorrect metric — outcome levels — the sophisticated application of the traditional CRM approach often yields larger profits than the targeting policies based on the indirect estimation methods.

Flexible Causal Inference for Political Science

Measuring the causal impact of state behavior on outcomes is one of the biggest methodological challenges in the field of political science, for two reasons: behavior is generally endogenous, and the threat of unobserved variables that confound the relationship between behavior and outcomes is pervasive. Matching methods, widely considered to be the state of the art in causal inference in political science, are generally ill-suited to inference in the presence of unobserved confounders. Heckman-style multiple-equation models offer a solution to this problem; however, they rely on functional-form assumptions that can produce substantial bias in estimates of average treatment effects. We describe a category of models, flexible joint likelihood models, that account for both features of the data while avoiding reliance on rigid functional-form assumptions. We then assess these models’ performance in a series of neutral simulations, in which they produce substantial (55% to ${>}$90%) reduction in bias relative to competing models. Finally, we demonstrate their utility in a reanalysis of Simmons’ (2000) classic study of the impact of Article VIII commitment on compliance with the IMF’s currency-restriction regime.

On Heckits, LATE, and Numerical Equivalence

Structural econometric methods are often criticized for being sensitive to functional form assumptions. We study parametric estimators of the local average treatment effect (LATE) derived from a widely used class of latent threshold crossing models and show they yield LATE estimates algebraically equivalent to the instrumental variables (IV) estimator. Our leading example is Heckman's (1979) two‐step (“Heckit”) control function estimator which, with two‐sided non‐compliance, can be used to compute estimates of a variety of causal parameters. Equivalence with IV is established for a semiparametric family of control function estimators and shown to hold at interior solutions for a class of maximum likelihood estimators. Our results suggest differences between structural and IV estimates often stem from disagreements about the target parameter rather than from functional form assumptions per se. In cases where equivalence fails, reporting structural estimates of LATE alongside IV provides a simple means of assessing the credibility of structural extrapolation exercises.

The Intergenerational Transmission of Welfare Dependency

There is a strong intergenerational correlation in welfare participation, but this does not imply that parental welfare receipt induces child receipt. While there are a few quasi-experimental studies that provide estimates of the causal effect of parental welfare participation for children from marginal welfare participants, we know very little about intergenerational spillovers of welfare participation onto the children of average welfare participants. By combining rich administrative data from Norway with weak mean-monotonicity assumptions, we estimate nonparametric bounds around the average causal effect of parental welfare participation on children’s welfare participation in the general population, as well as the average causal effect for children growing up in welfare-dependent families. We find that these average causal effects are considerably lower than the intergenerational correlation in welfare participation, and substantially below available local average treatment effect estimates in the literatu . We further find important differences between intergenerational spillovers of disability insurance and intergenerational spillovers of financial assistance, a traditional means-tested welfare program.

The impact of response-guided baseline phase extensions on treatment effect estimates

BackgroundWhen developmental disabilities researchers use multiple-baseline designs they are encouraged to delay the start of an intervention until the baseline stabilizes or until preceding cases have responded to intervention. Using ongoing visual analyses to guide the timing of the start of the intervention can help to resolve potential ambiguities in the graphical display; however, these forms of response-guided experimentation have been criticized as a potential source of bias in treatment effect estimation and inference. Aims and methodsMonte Carlo simulations were used to examine the bias and precision of average treatment effect estimates obtained from multilevel models of four-case multiple-baseline studies with series lengths that varied from 19 to 49 observations per case. We varied the size of the average treatment effect, the factors used to guide intervention decisions (baseline stability, response to intervention, both, or neither), and whether the ongoing analysis was masked or not. ResultsNone of the methods of responding to the data led to appreciable bias in the treatment effect estimates. Furthermore, as timing-of-intervention decisions became responsive to more factors, baselines became longer and treatment effect estimates became more precise. ConclusionsAlthough the study was conducted under limited conditions, the response-guided practices did not lead to substantial bias. By extending baseline phases they reduced estimation error and thus improved the treatment effect estimates obtained from multilevel models.

Causal inference with measurement error in outcomes: Bias analysis and estimation methods.

Inverse probability weighting estimation has been popularly used to consistently estimate the average treatment effect. Its validity, however, is challenged by the presence of error-prone variables. In this paper, we explore the inverse probability weighting estimation with mismeasured outcome variables. We study the impact of measurement error for both continuous and discrete outcome variables and reveal interesting consequences of the naive analysis which ignores measurement error. When a continuous outcome variable is mismeasured under an additive measurement error model, the naive analysis may still yield a consistent estimator; when the outcome is binary, we derive the asymptotic bias in a closed-form. Furthermore, we develop consistent estimation procedures for practical scenarios where either validation data or replicates are available. With validation data, we propose an efficient method for estimation of average treatment effect; the efficiency gain is substantial relative to usual methods of using validation data. To provide protection against model misspecification, we further propose a doubly robust estimator which is consistent even when either the treatment model or the outcome model is misspecified. Simulation studies are reported to assess the performance of the proposed methods. An application to a smoking cessation dataset is presented.

Identification and Estimation of Treatment Effects in the Presence of (Correlated) Neighborhood Interactions: Model and Stata Implementation via Ntreatreg

In this article, I present a counterfactual model identifying average treatment effects by conditional mean independence when considering peer- or neighborhood-correlated effects, and I provide a new command, ntreatreg, that implements such models in practical applications. The model and its accompanying command provide an estimation of average treatment effects when the stable unit treatment-value assumption is relaxed under specific conditions. I present two instructional applications: the first is a simulation exercise that shows both model implementation and ntreatreg correctness; the second is an application to real data, aimed at measuring the effect of housing location on crime in the presence of social interactions. In the second application, results are compared with a no-interaction setting.

Estimation of Population Average Treatment Effects in the FIRST Trial: Application of a Propensity Score-Based Stratification Approach.

To estimate and compare sample average treatment effects (SATE) and population average treatment effects (PATE) of a resident duty hour policy change on patient and resident outcomes using data from the Flexibility in Duty Hour Requirements for Surgical Trainees Trial ("FIRST Trial"). Secondary data from the National Surgical Quality Improvement Program and the FIRST Trial (2014-2015). The FIRST Trial was a cluster-randomized pragmatic noninferiority trial designed to evaluate the effects of a resident work hour policy change to permit greater flexibility in scheduling on patient and resident outcomes. We estimated hierarchical logistic regression models to estimate the SATE of a policy change on outcomes within an intent-to-treat framework. Propensity score-based poststratification was used to estimate PATE. This study was a secondary analysis of previously collected data. Although SATE estimates suggested noninferiority of outcomes under flexible duty hour policy versus standard policy, the noninferiority of a policy change was inconclusively noninferior based on PATE estimates due to imprecision. Propensity score-based poststratification can be valuable tools to address trial generalizability but may yield imprecise estimates of PATE when sparse strata exist.

The Effect of Sale Methods on the Interest Rate of Municipal Bonds: A Heterogeneous Endogenous Treatment Estimation

Conventional wisdom argues that competitive bidding, compared to negotiated sale, can reduce the interest cost of municipal bonds in the primary market. Studies correcting for self‐selection bias have mixed findings. However, these studies are subject to some potential methodological issues. This study circumvents these pit‐falls and provides an estimate of the heterogeneous average treatment effect (ATE). After correcting for the endogeneity bias of sale method and its interaction effect, on average competitive sale still significantly lowers the interest rate, but the interest cost advantage decreases as issuer experience increases.