Precision Of Treatment Effect Estimates Research Articles

An Adaptive Targeted Field Experiment: Job Search Assistance for Refugees in Jordan

Abstract We introduce an adaptive targeted treatment assignment methodology for field experiments. Our Tempered Thompson Algorithm balances the goals of maximizing the precision of treatment effect estimates and maximizing the welfare of experimental participants. A hierarchical Bayesian model allows us to adaptively target treatments. We implement our methodology in Jordan, testing policies to help Syrian refugees and local jobseekers to find work. The immediate employment impacts of a small cash grant, information and psychological support are small, but targeting raises employment by 1 percentage-point (20%). After 4 months, cash has a sizable effect on employment and earnings of Syrians.

Covariate adjustment in randomized clinical trials with missing covariate and outcome data.

When analyzing data from randomized clinical trials, covariate adjustment can be used to account for chance imbalance in baseline covariates and to increase precision of the treatment effect estimate. A practical barrier to covariate adjustment is the presence of missing data. In this article, in the light of recent theoretical advancement, we first review several covariate adjustment methods with incomplete covariate data. We investigate the implications of the missing data mechanism on estimating the average treatment effect in randomized clinical trials with continuous or binary outcomes. In parallel, we consider settings where the outcome data are fully observed or are missing at random; in the latter setting, we propose a full weighting approach that combines inverse probability weighting for adjusting missing outcomes and overlap weighting for covariate adjustment. We highlight the importance of including the interaction terms between the missingness indicators and covariates as predictors in the models. We conduct comprehensive simulation studies to examine the finite-sample performance of the proposed methods and compare with a range of common alternatives. We find that conducting the proposed adjustment methods generally improves the precision of treatment effect estimates regardless of the imputation methods when the adjusted covariate is associated with the outcome. We apply the methods to the Childhood Adenotonsillectomy Trial to assess the effect of adenotonsillectomy on neurocognitive functioning scores.

Inference in regression discontinuity designs with high-dimensional covariates

Summary We study regression discontinuity designs in which many predetermined covariates, possibly much more than the number of observations, can be used to increase the precision of treatment effect estimates. We consider a two-step estimator which first selects a small number of ‘important’ covariates through a localised lasso-type procedure, and then, in a second step, estimates the treatment effect by including the selected covariates linearly into the usual local linear estimator. We provide an in-depth analysis of the algorithm’s theoretical properties, showing that, under an approximate sparsity condition, the resulting estimator is asymptotically normal, with asymptotic bias and variance that are conceptually similar to those obtained in low-dimensional settings. Bandwidth selection and inference can be carried out using standard methods. We also provide simulations and an empirical application.

An Approximation Algorithm for Blocking of an Experimental Design

AbstractBlocked randomized designs are used to improve the precision of treatment effect estimates compared to a completely randomized design. A block is a set of units that are relatively homogeneous and consequently would tend to produce relatively similar outcomes if the treatment had no effect. The problem of finding the optimal blocking of the units into equal sized blocks of any given size larger than two is known to be a difficult problem—there is no polynomial time method guaranteed to find the optimal blocking. All available methods to solve the problem are heuristic methods. We propose methods that run in polynomial time and guarantee a blocking that is provably close to the optimal blocking. In all our simulation studies, the proposed methods perform better, create better homogeneous blocks, compared with the existing methods. Our blocking method aims to minimize the maximum of all pairwise differences of units in the same block. We show that bounding this maximum difference ensures that the error in the average treatment effect estimate is similarly bounded for all treatment assignments. In contrast, if the blocking bounds the average or sum of these differences, the error in the average treatment effect estimate can still be large in several treatment assignments.

241 Utilizing real-world evidence to increase efficiency of randomized controlled trials with application to repurposed therapeutics for COVID-19

OBJECTIVES/GOALS: We aim to extend a novel statistical method called the Semi-Supervised Mixture Multisource Exchangeability Model (SS-MIX-MEM) and to implement the SS-MIX-MEM approach to supplement ALPS-COVID data with N3C data to achieve analyses with greater precision and actionable conclusions. METHODS/STUDY POPULATION: We will apply the SS-MIX-MEM to supplement the Angiotensin receptor blocker-based Lung Protective Strategy for COVID-19 (ALPS-COVID) RCTs with the National COVID Cohort Collaborative (N3C) database. ALPS-COVID includes both an inpatient and outpatient trial, which investigate losartan as a treatment for COVID-19. The outpatient trial sought to randomize 580 individuals but only enrolled 117, whereas the inpatient trial met its enrollment target and randomized 205 individuals. The N3C database has 3,237,344 COVID-19 cases alongside demographics, lab values, and more. RESULTS/ANTICIPATED RESULTS: In simulation studies, the proposed SS-MIX-MEM approach effectively leveraged a subgroup of supplemental real world data for RCT analyses, improving trial efficiency by increasing precision of treatment effect estimates, decreasing necessary sample size, and introducing minimal bias. In an influenza trial real world data application, the SS-MIX-MEM approach was able to effectively provide insight into treatment effect heterogeneity found in an RCT analogous to incorporating around 80 individuals into a subgroup analysis. We anticipate that leveraging external real world data in a re-analysis of the ALPS-COVID RCTs could provide new insights into losartan, a readily available, potentially beneficial therapeutic for COVID-19. DISCUSSION/SIGNIFICANCE: The high blood pressure drug, losartan, is readily available, has an established safety profile, and might be effective as a treatment for COVID-19. Given that we have very few effective treatment options and are still in the midst of a global pandemic, patients with COVID-19 would greatly benefit from a repurposed, readily available treatment.

Degree irregularity and rank probability bias in network meta-analysis.

Network meta-analysis (NMA) is a statistical technique for the comparison of treatment options. Outcomes of Bayesian NMA include estimates of treatment effects, and the probabilities that each treatment is ranked best, second best and so on. How exactly network topology affects the accuracy and precision of these outcomes is not fully understood. Here we carry out a simulation study and find that disparity in the number of trials involving different treatments leads to a systematic bias in estimated rank probabilities. This bias is associated with an increased variation in the precision of treatment effect estimates. Using ideas from the theory of complex networks, we define a measure of "degree irregularity" to quantify asymmetry in the number of studies involving each treatment. Our simulations indicate that more regular networks have more precise treatment effect estimates and smaller bias of rank probabilities. Conversely, these topological effects are not observed for the accuracy of treatment effect estimates. This reinforces the importance of taking into account multiple measures, rather than making decisions based on a single metric. We also find that degree regularity is a better indicator for the accuracy and precision of parameter estimates in NMA than both the total number of studies in a network and the disparity in the number of trials per comparison. These results have implications for planning future trials. We demonstrate that choosing trials which reduce the network's irregularity can improve the precision and accuracy of parameter estimates from NMA.

An Adaptive Targeted Field Experiment: Job Search Assistance for Refugees in Jordan

We introduce a novel adaptive targeted treatment assignment methodology for field experiments. Our Tempered Thompson Algorithm balances the goals of maximizing the precision of treatment effect estimates and maximizing the welfare of experimental participants. A hierarchical Bayesian model allows us to adaptively target treatments. We implement our methodology in Jordan, testing policies to help Syrian refugees and local jobseekers to find work. The immediate employment impacts of a small cash grant, information and psychological support are close to zero, but targeting raises employment by 1 percentage-point (20%). After four months, cash has a sizable effect on employment and earnings of Syrians.

The Effect of Workforce Mobility on Intervention Effectiveness Estimates.

Little is known about how mobile populations of workers may influence the ability to implement, measure, and evaluate health and safety interventions delivered at worksites. A simulation study is used to objectively measure both precision and relative bias of six different analytic methods as a function of the amount of mobility observed in the workforce. Those six methods are then used to reanalyze a previously conducted cluster-randomized control trial involving a highly mobile workforce in the construction industry. As workforce mobility increases, relative bias in treatment effects derived from standard models to analyze cluster-randomized trials also increases. Controlling for amount of time exposed to the intervention can greatly reduce this bias. Analyzing only subsets of workers who exhibit the least amount of mobility can result in decreased precision of treatment effect estimates. We demonstrate a 59% increase in the treatment effect size from the reanalysis of the previously conducted trial. When evaluating organizational interventions implemented at specific worksites by measuring perceptions and outcomes of workers present at those sites, researchers should consider the effects that the mobility of the workforce may have on the estimated treatment effects. The choice of analytic method can greatly affect both precision and accuracy of estimates.

The Quality of Reporting of Measures of Precision in Animal Experiments in Implant Dentistry: A Methodological Study.

Information on precision of treatment-effect estimates is pivotal for understanding research findings. In animal experiments, which provide important information for supporting clinical trials in implant dentistry, inaccurate information may lead to biased clinical trials. The aim of this methodological study was to determine whether sample size calculation, standard errors, and confidence intervals for treatment-effect estimates are reported accurately in publications describing animal experiments in implant dentistry. MEDLINE (via PubMed), Scopus, and SciELO databases were searched to identify reports involving animal experiments with dental implants published from September 2010 to March 2015. Data from publications were extracted into a standardized form with nine items related to precision of treatment estimates and experiment characteristics. Data selection and extraction were performed independently and in duplicate, with disagreements resolved by discussion-based consensus. The chi-square and Fisher exact tests were used to assess differences in reporting according to study sponsorship type and impact factor of the journal of publication. The sample comprised reports of 161 animal experiments. Sample size calculation was reported in five (2%) publications. P values and confidence intervals were reported in 152 (94%) and 13 (8%) of these publications, respectively. Standard errors were reported in 19 (12%) publications. Confidence intervals were better reported in publications describing industry-supported animal experiments (P = .03) and with a higher impact factor (P = .02). Information on precision of estimates is rarely reported in publications describing animal experiments in implant dentistry. This lack of information makes it difficult to evaluate whether the translation of animal research findings to clinical trials is adequate.

Methods for network meta-analysis of continuous outcomes using individual patient data: a case study in acupuncture for chronic pain.

BackgroundNetwork meta-analysis methods, which are an extension of the standard pair-wise synthesis framework, allow for the simultaneous comparison of multiple interventions and consideration of the entire body of evidence in a single statistical model. There are well-established advantages to using individual patient data to perform network meta-analysis and methods for network meta-analysis of individual patient data have already been developed for dichotomous and time-to-event data. This paper describes appropriate methods for the network meta-analysis of individual patient data on continuous outcomes.MethodsThis paper introduces and describes network meta-analysis of individual patient data models for continuous outcomes using the analysis of covariance framework. Comparisons are made between this approach and change score and final score only approaches, which are frequently used and have been proposed in the methodological literature. A motivating example on the effectiveness of acupuncture for chronic pain is used to demonstrate the methods. Individual patient data on 28 randomised controlled trials were synthesised. Consistency of endpoints across the evidence base was obtained through standardisation and mapping exercises.ResultsIndividual patient data availability avoided the use of non-baseline-adjusted models, allowing instead for analysis of covariance models to be applied and thus improving the precision of treatment effect estimates while adjusting for baseline imbalance.ConclusionsThe network meta-analysis of individual patient data using the analysis of covariance approach is advocated to be the most appropriate modelling approach for network meta-analysis of continuous outcomes, particularly in the presence of baseline imbalance. Further methods developments are required to address the challenge of analysing aggregate level data in the presence of baseline imbalance.Electronic supplementary materialThe online version of this article (doi:10.1186/s12874-016-0224-1) contains supplementary material, which is available to authorized users.

The role of cluster size and intra-cluster correlations when adjusting for covariates in the analysis of cluster randomised trials

Reports of clinical trials often include adjusted analyses, which incorporate covariate data into the analysis model. Adjusting for covariates can increase the precision of treatment effect estimates and increase the power of statistical tests, without the need to increase sample size. In individually randomised trials, the main reason to adjust for a particular covariate is that it is expected to be strongly associated with the primary outcome. The larger the association between covariate and outcome, the greater the increase in power achieved from an adjusted analysis. A valid analysis of a cluster randomised trial (CRT) must take into account the clustered structure of the data, for example by using a mixed effects model. The selection of covariates for an adjusted analysis of a CRT is more complicated because covariates exist at both the cluster level and individual level. Further, adjustment for an individual level covariate can affect the residual variance of the outcome at both the cluster and individual levels. Using results from simulations, and some analytic investigation, I show how the size of clusters and the intra-cluster correlations of the covariate and outcome affect power and precision in adjusted analyses of CRTs using linear mixed effects models. I also consider logistic mixed effects models and show how adjusting for individual level or cluster level covariates affect what treatment effect is being estimated.

Information Theory and Propensity Score Matching

This paper develops a generalized maximum entropy (GME) approach to propensity score matching (PSM). A GME discrete choice model is used to develop propensity scores and estimate treatment effects in a set of Monte Carlo simulations. The GME PSM is compared to a more traditional logit PSM. Sample sizes and common support regions are varied across simulations to reflect common problems in the program evaluation literature. The GME PSM exhibits bias levels that are comparable to the logit PSM, but it provides substantial improvements in the precision of treatment effect estimates (including lower standard deviation of the treatment effect estimate and lower RMSE).

Changing cluster composition in cluster randomised controlled trials: design and analysis considerations.

BackgroundThere are many methodological challenges in the conduct and analysis of cluster randomised controlled trials, but one that has received little attention is that of post-randomisation changes to cluster composition. To illustrate this, we focus on the issue of cluster merging, considering the impact on the design, analysis and interpretation of trial outcomes.MethodsWe explored the effects of merging clusters on study power using standard methods of power calculation. We assessed the potential impacts on study findings of both homogeneous cluster merges (involving clusters randomised to the same arm of a trial) and heterogeneous merges (involving clusters randomised to different arms of a trial) by simulation. To determine the impact on bias and precision of treatment effect estimates, we applied standard methods of analysis to different populations under analysis.ResultsCluster merging produced a systematic reduction in study power. This effect depended on the number of merges and was most pronounced when variability in cluster size was at its greatest. Simulations demonstrate that the impact on analysis was minimal when cluster merges were homogeneous, with impact on study power being balanced by a change in observed intracluster correlation coefficient (ICC). We found a decrease in study power when cluster merges were heterogeneous, and the estimate of treatment effect was attenuated.ConclusionsExamples of cluster merges found in previously published reports of cluster randomised trials were typically homogeneous rather than heterogeneous. Simulations demonstrated that trial findings in such cases would be unbiased. However, simulations also showed that any heterogeneous cluster merges would introduce bias that would be hard to quantify, as well as having negative impacts on the precision of estimates obtained. Further methodological development is warranted to better determine how to analyse such trials appropriately. Interim recommendations include avoidance of cluster merges where possible, discontinuation of clusters following heterogeneous merges, allowance for potential loss of clusters and additional variability in cluster size in the original sample size calculation, and use of appropriate ICC estimates that reflect cluster size.

Covariance Adjustments for the Analysis of Randomized Field Experiments

It has become common practice to analyze randomized experiments using linear regression with covariates. Improved precision of treatment effect estimates is the usual motivation. In a series of important articles, David Freedman showed that this approach can be badly flawed. Recent work by Winston Lin offers partial remedies, but important problems remain. In this article, we address those problems through a reformulation of the Neyman causal model. We provide a practical estimator and valid standard errors for the average treatment effect. Proper generalizations to well-defined populations can follow. In most applications, the use of covariates to improve precision is not worth the trouble.

Ethnic Sensitive or Molecular Sensitive Beyond All Regions Being Equal in Multiregional Clinical Trials

For decades, clinical trials have been the primary mechanism for medical products to enter the marketplace. Over more than a decade, globalization of medical product development via a multiregional clinical trial (MRCT) approach has generated greater enthusiasm because of tangible benefits in terms of cost and time for drug development. There are, however, many challenges including and not limited to design issues, statistical analysis methods, interpretation of extreme region performance, and in-process quality assurance issues. This article presents a number of examples to exemplify regional variability expected versus precision of treatment effect estimates that are generally impacted by the type of primary efficacy endpoint evaluated. We explore region-driven intrinsic and extrinsic ethnic factors for potential explanation of regional heterogeneity caused by differences in medical practice and / or disease etiology. Bayesian credible interval may be considered as a viable approach to assess the robustness of region-specific treatment effect. Ethnic-sensitive or molecular-sensitive region-driven designs may be explored to prospectively address the potential regional heterogeneity versus the potential predictiveness of causal genetic variants or molecular target biomarkers on treatment effect.