Continuous difference-in-differences with double/debiased machine learning

Commentary: Can a Quasi-experimental Design Be a Better Idea than an Experimental One?

Summary Randomized experiments have been the gold standard for drawing causal inference. The conventional model-based approach has been one of the most popular methods of analysing treatment effects from randomized experiments, which is often carried out through inference for certain model parameters. In this paper, we provide a systematic investigation of model-based analyses for treatment effects under the randomization-based inference framework. This framework does not impose any distributional assumptions on the outcomes, covariates and their dependence, and utilizes only randomization as the reasoned basis. We first derive the asymptotic theory for $ Z $-estimation in completely randomized experiments, and propose sandwich-type conservative covariance estimation. We then apply the developed theory to analyse both average and individual treatment effects in randomized experiments. For the average treatment effect, we consider model-based, model-imputed and model-assisted estimation strategies, where the first two strategies can be sensitive to model misspecification or require specific methods for parameter estimation. The model-assisted approach is robust to arbitrary model misspecification and always provides consistent average treatment effect estimation. We propose optimal ways to conduct model-assisted estimation using generally nonlinear least squares for parameter estimation. For the individual treatment effects, we propose directly modelling the relationship between individual effects and covariates, and discuss the model’s identifiability, inference and interpretation allowing for model misspecification.

If we recognize heterogeneity of treatment effect can we lessen waste?

In the last chapter we introduced the Fisher Exact P-value (FEP) approach for assessing sharp null hypotheses. As we saw, a sharp null hypothesis allowed us to fill in the values for all missing potential outcomes in the experiment. This was the basis for deriving the randomization distributions of various statistics, that is, the distributions induced by the random assignment of the treatments given fixed potential outcomes under that sharp null hypothesis. During the same period in which Fisher was developing this method, Neyman (1923, 1990) was focused on methods for the estimation of, and inference for, average treatment effects, also using the distribution induced by randomization, sometimes in combination with repeated sampling of the units in the experiment from a larger population of units. At a general level, he was interested in the long-run operating characteristics of statistical procedures under both repeated sampling from the population and randomized assignment of treatments to the units in the sample. Specifically, he attempted to find point estimators that were unbiased, and also interval estimators that had the specified nominal coverage in large samples. As noted before, his focus on average effects was different from the focus of Fisher; the average effect across a population may be equal to zero even when some, or even all, unit-level treatment effects differ from zero.

Randomized trials, observational studies, and the illusive search for the source of truth

Recently, there has been a surge in methodological development for the difference-in-differences approach to evaluate causal effects. Standard methods in the literature rely on the parallel trends assumption to identify the average treatment effect on the treated. However, the parallel trends assumption may be violated in the presence of unmeasured confounding, and the average treatment effect on the treated may not be useful in learning a treatment assignment policy for the entire population. In this article, we propose a general instrumented difference-in-differences approach for learning the optimal treatment policy. Specifically, we establish identification results using a binary instrumental variable when the parallel trends assumption fails to hold. Additionally, we construct a Wald estimator, novel inverse probability weighting estimators, and a class of semiparametric efficient and multiply robust estimators, with theoretical guarantees on consistency and asymptotic normality, even when relying on flexible machine learning algorithms for nuisance parameters estimation. Furthermore, we extend the instrumented difference-in-differences to the panel data setting. We evaluate our methods in extensive simulations and in an analysis of the Australian Longitudinal Survey.

This article shows that the observed sample adoption rate does not consistently estimate the population adoption rate even if the sample is random. It is proved that instead the sample adoption rate is a consistent estimate of the population joint exposure and adoption rate, which does not inform about adoption per se. Likewise, it is shown that a model of adoption with observed adoption outcome as a dependent variable and where exposure to the technology is not observed and controlled for cannot yield consistent estimates of the determinants of adoption. The article uses the counterfactual outcomes framework to show that the true population adoption rate corresponds to what is defined in the modern policy evaluation literature as the average treatment effect (ATE), which measures the effect or impact of a “treatment” on a person randomly selected in the population. In the adoption context, a “treatment” corresponds to exposure to the technology. The article uses the ATE estimation framework to derive consistent nonparametric and parametric estimators of population adoption rates and their determinants and applies the results to consistently estimate the population adoption rates and determinants of the NERICA (New Rice for Africa) rice varieties in Côte d'Ivoire. The ATE methodological approach developed in the article has significant policy implications with respect to judging the intrinsic merit of a new technology in terms of its potential demand by the target population independently of issues related to its accessibility and in terms of the decision to invest or not in its wide‐scale dissemination.

It is essential to meet climate goals outlined in the Paris Agreement that the Oil and Gas industry reduces carbon emissions along with achieving production targets. The described body of work herein provided a machine learning (ML) framework to predict upcoming shutdown events which often resulted in event driven flaring with increased carbon emission. Such flaring events are common and many times are not analyzed for root cause until unless they start to become bigger problems or occurred numerous times. One of the primary reasons for event driven flaring is poor process and equipment reliability. Therefore, ML framework is focused towards developing a tool to improve system reliability. Some of the challenges predicting these events are changing process, equipment and operating conditions and highly interactive processes which often results in generating false alarms. To address this issue, this paper proposes an adaptive ML framework where principle component analysis (PCA) is used for anomaly detection. PCA model is updated every hour so that it can update the model with changing operating conditions. PCA model utilizes sensor data from upstream and downstream of the process under consideration along with available equipment data to predict shutdown event. In the current work, three level of alarms are generated using threshold levels corresponding to 95%, 99% and 99.9% of confidence limits for D-statistics on the training data. These alarms correspond to 1) low risk (only reports and no urgency to act), 2) moderate risk (event review is must) and 3) high risk (urgency to act) respectively. The ML model was also able to identify top N sensors which contribute towards shutdown event. The solution was applied to an offshore compressor train and it was found that without using adaptive framework, only 70% of shutdown events can be predicted with lot of false alarms. Using adaptive framework all of the shutdown events were predicted. Although, false alarms were predicted, however the frequency at which false alarms were generated was found low and acceptable for practical application. Two case studies were presented which revealed that adaptive ML framework was found effective to capture events several hours in advance. Root-cause analysis was automated using contribution charts and top 4 key sensors were identified which were contributing towards various shutdown events and were resulting in 1 to 10 hours of flaring. After root-cause analysis for each event, a mitigation solution was presented to avoid similar repeated shutdowns. Several process and operational setting changes were identified to avoid future shutdowns caused by compressor train. Overall, annually 46MMSCF reduction in flare volume and 30-thousand-barrel reduction in lost oil production volume was identified.

Deepwater oil and gas facilities typically encounter on an average up to 5% annual production losses due to unplanned downtime, conservatively estimated at billions of dollars impact for the industry. The existing toolkit and systems in place are not always adequate to identify and predict abnormal events that could lead towards unplanned facility shutdown. The interaction amongst process sub-systems and disturbances that propagate across these sub-systems with changing operating conditions are hard to predict without a fit-for-purpose model (or a digital twin). The focus of current work is on deepwater facility having several oil export pipeline pumps in parallel and several gas compressors in series. The alarm database showed records of several unplanned shutdown events around these critical equipements that resulted in undesirable outcomes such as production deferment, complete facility shutdown, loss of sales volumes and increased operational costs. In this work, an intelligent prognostic solution is proposed using machine learning (ML) framework for automatic prediction of impending facility downtime, and identification of key causative process variables. A systematic workflow was developed to identify, cleanse and process real time data for both model training and prediction. Several ML methods were evaluated; anomaly detection based on Principal Component Analysis (PCA) and Autoencoder (AE) algorithms were found performing better for the type of data available for the deepwater facility. The ML framework also supported analysis of underlying downtime causes to propose suitable mitigation steps. Knowledge based on physical understanding of the process was used to select each sub-system boundary and sensor list on which ML model was trained. These models were then cross-validated to test the accuracy of trained models. Finally, the alarm database was used to confirm the accuracy of the machine leaning models and identify root-causes for unplanned shutdowns. If the operating condition changes over time, the anomaly detection based ML models were setup to adapt to changing conditions by automatic model updates, resulting in significant reduction in false alarms. The adaptive ML models, when applied to one of the sub-system (with 30 different sensor data), predicted 24 unplanned events in 6 months of period, while when applied to another sub-system (with 40 sensor data), predicted only 6 unplanned downtime events. Several predictions were found as early as 30 mins to 2 hours, providing adequate early warning to take proactive actions. Case studies shown in the paper present diagnostic charts and identified early indicators were found in agreement with pre-alarms generated by existing alarm system, thus validating the ML solution. Current toolkit available to identify anomalous process behavior is limited to exception based surveillance with fixed min-max limits on each sensor data. Therefore, proposed adaptive ML solution has shown potential to revolutionize the topside process surveillance. This paper also describes how the ML framework can be scaled for a sustainable solution that provides prediction every minute, keeps the model evergreen utilizing cloud-based model deployment platform to train, predict and trigger automatic model updates and also span multiple process systems and facilities. Finally, we present directions for future work, where the current model can keep predicting various events and over time when sufficient events are collected, more advanced machine learning methods based on supervised ML can be developed and deployed.

Common causal estimands include the average treatment effect, the average treatment effect of the treated, and the average treatment effect on the controls. Using augmented inverse probability weighting methods, parametric models are judiciously leveraged to yield doubly robust estimators, that is, estimators that are consistent when at least one the parametric models is correctly specified. Three sources of uncertainty are associated when we evaluate these estimators and their variances, that is, when we estimate the treatment and outcome regression models as well as the desired treatment effect. In this article, we propose methods to calculate the variance of the normalized, doubly robust average treatment effect of the treated and average treatment effect on the controls estimators and investigate their finite sample properties. We consider both the asymptotic sandwich variance estimation, the standard bootstrap as well as two wild bootstrap methods. For the asymptotic approximations, we incorporate the aforementioned uncertainties via estimating equations. Moreover, unlike the standard bootstrap procedures, the proposed wild bootstrap methods use perturbations of the influence functions of the estimators through independently distributed random variables. We conduct an extensive simulation study where we vary the heterogeneity of the treatment effect as well as the proportion of participants assigned to the active treatment group. We illustrate the methods using an observational study of critical ill patients on the use of right heart catherization.

Propensity score analyses (PSA) of continuous treatments often operationalize the treatment as a multi-indicator composite, and its composite reliability is unreported. Latent variables or factor scores accounting for this unreliability are seldom used as alternatives to composites. This study examines the effects of the unreliability of indicators of a latent treatment in PSA using the generalized propensity score (GPS). A Monte Carlo simulation study was conducted varying composite reliability, continuous treatment representation, variability of factor loadings, sample size, and number of treatment indicators to assess whether Average Treatment Effect (ATE) estimates differed in their relative bias, Root Mean Squared Error, and coverage rates. Results indicate that low composite reliability leads to underestimation of the ATE of latent continuous treatments, while the number of treatment indicators and variability of factor loadings show little effect on ATE estimates, after controlling for overall composite reliability. The results also show that, in correctly specified GPS models, the effects of low composite reliability can be somewhat ameliorated by using factor scores that were estimated including covariates. An illustrative example is provided using survey data to estimate the effect of teacher adoption of a workbook related to a virtual learning environment in the classroom.

A Deep Learning Framework for Sickle Cell Disease Microfluidic Biomarker Assays

Stepped wedge design is a popular research design that enables a rigorous evaluation of candidate interventions by using a staggered cluster randomization strategy. While analytical methods were developed for designing stepped wedge trials, the prior focus has been solely on testing for the average treatment effect. With a growing interest on formal evaluation of the heterogeneity of treatment effects across patient subpopulations, trial planning efforts need appropriate methods to accurately identify sample sizes or design configurations that can generate evidence for both the average treatment effect and variations in subgroup treatment effects. To fill in that important gap, this article derives novel variance formulas for confirmatory analyses of treatment effect heterogeneity, that are applicable to both cross-sectional and closed-cohort stepped wedge designs. We additionally point out that the same framework can be used for more efficient average treatment effect analyses via covariate adjustment, and allows the use of familiar power formulas for average treatment effect analyses to proceed. Our results further sheds light on optimal design allocations of clusters to maximize the weighted precision for assessing both the average and heterogeneous treatment effects. We apply the new methods to the Lumbar Imaging with Reporting of Epidemiology Trial, and carry out a simulation study to validate our new methods.

The problem of missingness in observational data is ubiquitous. When the confounders are missing at random, multiple imputation is commonly used; however, the method requires congeniality conditions for valid inferences, which may not be satisfied when estimating average causal treatment effects. Alternatively, fractional imputation, proposed by Kim 2011, has been implemented to handling missing values in regression context. In this article, we develop fractional imputation methods for estimating the average treatment effects with confounders missing at random. We show that the fractional imputation estimator of the average treatment effect is asymptotically normal, which permits a consistent variance estimate. Via simulation study, we compare fractional imputation’s accuracy and precision with that of multiple imputation.

This work assesses the causal impact of the EU trade preferences granted to the Southern Mediterranean Countries (SMCs) in agriculture and fishery products over the period 2004–2014. It overcomes some of the weaknesses of previous assessments and presents several methodological improvements. Firstly, it relies on a continuous treatment – i.e. preferential margins – to capture the ‘average treatment effect’ of trade preferences, rather than on a binary treatment based on dummy variables. Secondly, it uses highly disaggregated data at sectoral level in order to evaluate properly the preferential treatment. Thirdly, it applies a non-parametric matching technique for continuous treatment – specifically, a generalized propensity score matching. The results show, on the one hand, that the impact of the EU preferences is positive and significant on SMCs trade and is better evaluated using impact evaluation techniques. On the other hand, they demonstrate that the relationship between preferences and trade flows is asymmetric and warn against the risk of providing too much of a good thing. These results raise important issues for policy-making. First, they demonstrate that raising the level of preferences is not the solution to foster the SMCs trade towards EU. Second, that the policy-makers should put more emphasis on complementary factors other than trade barriers.