Biased Estimates Of Treatment Effects Research Articles

Conditional bias adjusted estimator of treatment effect in 2-in-1 adaptive design

ABSTRACT For implementation of adaptive design, the adjustment of bias in treatment effect estimation becomes an increasingly important topic in recent years. While adaptive design literature traditionally focuses on the control of type I error rate and the adjustment of overall unconditional bias, the research on adjusting conditional bias has been limited. This paper proposes a conditional bias adjustment estimator of treatment effect under the context of 2-in-1 adaptive design and aims to provide a comprehensive investigation on their statistical properties including bias, mean squared error and coverage probability of confidence intervals. It demonstrated that conditional bias adjusted estimators greatly reduce the conditional bias and have similarly negligible unconditional bias compared with mean and median (unconditional) unbiased estimators. In addition, the test statistics is constructed based on the conditional bias adjustment estimators and compared with the naive unadjusted test.

Propensity Score Analysis With Unreliable Covariates: A Comparison of Five Reliability-Adjustment Methods

Propensity score analysis (PSA) is a crucial tool for researchers in mitigating selection bias arising from multiple covariates in quasi-experimental studies. Nevertheless, the impact of low-reliability covariates on PSA necessitates careful consideration. This study employs Monte Carlo simulation to assess five methods to adjust propensity scores for unreliability of covariates. The findings reveal that the latent variable model incorporating inclusive factor scores (PSIF) results in the smallest relative bias of treatment effect estimates. Notably, only PSIF consistently provides unbiased treatment effect estimates across all conditions. Furthermore, the study underscores the potential for a misleading covariate balance evaluation when dealing with unreliable covariates, given that treatment effect estimates may be biased even when the covariate balance is perceived as adequate.

Construction of efficient classes of circular balanced repeated measurements designs with R

Pharmacology, medicine, psychology, and the animal sciences all employ repeated measurement designs (RMDs). However, RMDs may experience carryover effects, which are the primary cause of bias in treatment effect estimation. In order to eliminate the carryover effects for odd v (the number of treatments), minimal circular balanced and strongly balanced repeated measurement designs (RMDs) are the ones that should be used. The minimal circular partially balanced and weakly balanced RMDs are used for even v. In order to obtain these important classes of minimal circular RMDs in periods of equal, two, and three different sizes, an R-based algorithm is developed in this article. The newly developed algorithm has made so simple the construction of balanced RMDs and their generalized classes. As a result, it is a novel piece of research.

Using Balancing Weights to Target the Treatment Effect on the Treated when Overlap is Poor.

Inverse probability weights are commonly used in epidemiology to estimate causal effects in observational studies. Researchers often focus on either the average treatment effect or the average treatment effect on the treated with inverse probability weighting estimators. However, poor overlap in the baseline covariates between the treated and control groups can produce extreme weights that can result in biased treatment effect estimates. One alternative to inverse probability weights are overlap weights, which target the population with the most overlap on observed covariates. Although estimates based on overlap weights produce less bias in such contexts, the causal estimand can be difficult to interpret. An alternative to model-based inverse probability weights are balancing weights, which directly target imbalances during the estimation process, rather than model fit. Here, we explore whether balancing weights allow analysts to target the average treatment effect on the treated in cases where inverse probability weights lead to biased estimates due to poor overlap. We conduct three simulation studies and an empirical application. We find that balancing weights often allow the analyst to still target the average treatment effect on the treated even when overlap is poor. We show that although overlap weights remain a key tool, more familiar estimands can sometimes be targeted by using balancing weights.

Methods for comparative effectiveness based on time to confirmed disability progression with irregular observations in multiple sclerosis.

Real-world data sources offer opportunities to compare the effectiveness of treatments in practical clinical settings. However, relevant outcomes are often recorded selectively and collected at irregular measurement times. It is therefore common to convert the available visits to a standardized schedule with equally spaced visits. Although more advanced imputation methods exist, they are not designed to recover longitudinal outcome trajectories and typically assume that missingness is non-informative. We, therefore, propose an extension of multilevel multiple imputation methods to facilitate the analysis of real-world outcome data that is collected at irregular observation times. We illustrate multilevel multiple imputation in a case study evaluating two disease-modifying therapies for multiple sclerosis in terms of time to confirmed disability progression. This survival outcome is derived from repeated measurements of the Expanded Disability Status Scale, which is collected when patients come to the healthcare center for a clinical visit and for which longitudinal trajectories can be estimated. Subsequently, we perform a simulation study to compare the performance of multilevel multiple imputation to commonly used single imputation methods. Results indicate that multilevel multiple imputation leads to less biased treatment effect estimates and improves the coverage of confidence intervals, even when outcomes are missing not at random.

Humidified atmosphere in a time-lapse embryo culture system does not improve ongoing pregnancy rate: a retrospective propensity score model study derived from 496 first ICSI cycles.

To investigate whether high relative humidity conditions (HC), when using a time-lapse system (TLS) with sequential culture media, are beneficial to embryo culture, improving ongoing pregnancy rates. We included patients undergoing their first ICSI cycle treatment from April 2021 to May 2022. Patients assigned to dry conditions (DC) or HC were 278 and 218, respectively. We used a GERI TLS, three chambers configured in humidity conditions and three in dry conditions. The effect of HC on ongoing pregnancy rate was assessed by the propensity matched sample, to reduce potential differences between women undergoing either HC or DC and reduce biased estimation of treatment effect. After adjusting for several confounding variables and applying the propensity score (PS), no significant differences were observed in the rates of normal (2PN) and abnormal (1PN and 3PN) fertilization, blastulation, top-quality blastocysts, frozen blastocysts, ongoing pregnancies, and miscarriages. The 2-cell (t2) and 4-cell (t4) stages and cell divisions between such stages occurred earlier and were more synchronous in the in DC. These results suggest that HC conditions do not improve the rate of ongoing pregnancy and several embryological outcomes, under the conditions used in this study based on a time-lapse system and sequential culture with day 3 medium change-over.

Missing data matter: an empirical evaluation of the impacts of missing EHR data in comparative effectiveness research.

The impacts of missing data in comparative effectiveness research (CER) using electronic health records (EHRs) may vary depending on the type and pattern of missing data. In this study, we aimed to quantify these impacts and compare the performance of different imputation methods. We conducted an empirical (simulation) study to quantify the bias and power loss in estimating treatment effects in CER using EHR data. We considered various missing scenarios and used the propensity scores to control for confounding. We compared the performance of the multiple imputation and spline smoothing methods to handle missing data. When missing data depended on the stochastic progression of disease and medical practice patterns, the spline smoothing method produced results that were close to those obtained when there were no missing data. Compared to multiple imputation, the spline smoothing generally performed similarly or better, with smaller estimation bias and less power loss. The multiple imputation can still reduce study bias and power loss in some restrictive scenarios, eg, when missing data did not depend on the stochastic process of disease progression. Missing data in EHRs could lead to biased estimates of treatment effects and false negative findings in CER even after missing data were imputed. It is important to leverage the temporal information of disease trajectory to impute missing values when using EHRs as a data resource for CER and to consider the missing rate and the effect size when choosing an imputation method.

Propensity score analysis with local balance.

Most propensity score (PS) analysis methods rely on a correctly specified parametric PS model, which may result in biased estimation of the average treatment effect (ATE) when the model is misspecified. More flexible nonparametric models for treatment assignment alleviate this issue, but they do not always guarantee covariate balance. Methods that force balance in the means of covariates and their transformations between the treatment groups, termed global balance in this article, do not always lead to unbiased estimation of ATE. Their estimated propensity scores only ensure global balance but not the balancing property, which is defined as the conditional independence between treatment assignment and covariates given the propensity score. The balancing property implies not only global balance but also local balance-the mean balance of covariates in propensity score stratified sub-populations. Local balance implies global balance, but the reverse is false. We propose the propensity score with local balance (PSLB) methodology, which incorporates nonparametric propensity score models and optimizes local balance. Extensive numerical studies showed that the proposed method can substantially outperform existing methods that estimate the propensity score by optimizing global balance, when the model is misspecified. The proposed method is implemented in the R package PSLB.

Propensity Score Matching with Cross-Classified Data Structures: A Comparison of Methods

In the current study, we compare propensity score (PS) matching methods for data with a cross-classified structure, where each individual is clustered within more than one group, but the groups are not hierarchically organized. Through a Monte Carlo simulation study, we compared sequential cluster matching (SCM), preferential within cluster matching (PWCM), greedy matching (GM), and optimal full matching (OFM), using propensity scores from four different models. The results indicated that the four matching methods performed well when PSs were estimated with logistic regression containing both level-1 and level-2 covariates. When the level-2 covariates were omitted in the logistic regression PS model, matching methods resulted in biased treatment effect estimates. However, omission of level-2 covariates did not result in biased estimates when the PS model was a logistic cross-classified random effects model (CCREM). SCM and PWCM outperformed GM and OFM with a logistic CCREM that included level-1 and level-2 covariates.

Dyadic retention in a phase 3 mild cognitive impairment clinical trial

AbstractBackgroundExiting a study early (dropout) may bias treatment effect estimates, while dropout and missing visits reduces study power. Simply increasing planned sample sizes cannot remove bias due to non‐ignorable missingness (Fleming, 2011). Preventing missing data is best (National Research Council, 2010). Hence, quantifying associations with missingness may identify ways to improve retention. Alzheimer’s disease (AD) clinical trials require study participation by a participant and study partner. Previous research examined dyadic retention in AD dementia trials (Grill et al., 2013; Bernstein, Grill & Gillen, 2021), yet less is understood in mild cognitive impairment (MCI) trials. Here, we examined MCI trial dropout rates and missingness patterns by dyad type.MethodWe performed retrospective analyses of the Alzheimer’s Disease Cooperative Study phase 3 MCI trial of donepezil and vitamin E. Scheduled study visits occurred at baseline, month 3, and semi‐annually until month 36. Study completion included the progression to a possible or probable AD diagnosis or month 36 visit. Dropout was withdrawal of consent or lost‐to‐follow‐up. Deaths and study completions were censored. Dyad types were spousal, adult child, or other. We estimated Kaplan‐Meier‐based dropout rates and summarized frequencies of on‐study missing visit patterns by baseline dyad type and stratified by randomized treatment (placebo, donepezil, vitamin E).ResultFigure 1 displays cumulative dropout rates by dyad type and treatment (panels A, B, C). Among 259 participants randomized to placebo, estimated study dropout rates were 24% for spousal, 16% for adult child, and 14% for other dyads. Table 1 summarizes on‐study missing visit patterns by dyad type and treatment; Figure 2 displays placebo data for specific patterns of missing data. While on study, 21% of spousal, 23% of adult child, and 27% of other dyads missed at least one scheduled visit, primarily monotonic missingness.ConclusionDyad type may be associated with some types of missingness in MCI clinical trials. Implementing dyad‐type‐specific strategies that encourage staying on study and attending all scheduled visits may improve retention, thus reducing bias, in future MCI trials.

Use of Data Enhancement for Emerging Study Methodologies That Compare Clinical Trial and Real-World Data

External comparators (ECs) are a useful emerging research methodology to complement clinical trials by providing real-world (RW) context on comparator treatments and are increasingly of interest to regulators and payers in rare diseases and their sub-populations. Indeed, in recent publications from FDA and NICE, there is an insistence on assessing the fitness-for-purpose of RW data, including both data quality (e.g., completeness, accuracy of key study variables) as well as relevance (e.g., availability of trial-like outcomes), to ensure robust and reliable results can be generated in RW evidence studies including ECs. In light of these requirements, we aimed to understand the suitability of existing haematology-oncology RW data sources (DSs) for comparison with clinical trial data, with the intention of bringing together qualified partners to establish a research collaboration network specialized in this emerging research methodology. Literature reviews, desk research, and recent/ongoing in-house clinical studies were reviewed to identify European DSs involved in RW haematology-oncology data generation. Of 332 European DSs identified, primary market research was conducted with 46 to ascertain fitness-for-purpose, including patient counts, variable availability, data quality, and operational aspects of data access. As detailed in Table 1 below, among the 46 sources, approximately 9,560 patients per year were recorded across 6 haematology-oncology malignancies - DLBCL, FL, CLL, MM, MCL, and MZL - spanning 11 European countries. We found that data on patient and disease characteristics, labs, clinical outcomes, and treatment sequence were generally readily available (collected in approximately 100%, 87%, 91%, and 91% of sources, respectively). Data quality and completeness was reasonable, but mixed for response measures and prognostic factors, particularly at later lines of therapy. On the other hand, genetic markers, QoL and HCRU measures, were often not readily available (collected in only approximately 46%, 20% and 11% of sources, respectively). Regarding operational aspects, hospitals were more willing than registries and claims databases to share patient-level data with external researchers. Most DSs reported data collection could be conducted via electronic extraction (e.g., electronic medical records, 65%) or a mix of electronic and manual methods (e.g., case report forms, 48%), which would enable the enhancement of structured EMR data with unstructured data (e.g., captured in clinical notes or pathology reports. Though data availability across the assessed European haematology-oncology DSs is sufficient to support ECs, artificial intelligence was identified as a useful tool to help enhance RW data readily captured in within EMRs - such as genetic markers, response measures and prognostic factors - and improve the comparability of clinical trial and RW patients. Specifically, emerging machine learning (ML) methods can improve propensity score matching, increase statistical power and reduce bias in treatment effect estimates or can offer alternatives to propensity score matching, such as G-computation and double debiased ML. Moreover, natural language processing (NLP) techniques can be used to extract unstructured data from clinical notes or pathology reports, instead of relying on more burdensome manual chart review methods. Although there is significant promise in emerging technologies for improving the data quality and operational efficiency of future haematology-oncology EC studies, the use of techniques such as NLP and ML to generate RW evidence may have limitations when used on small sample sizes in the context of rare diseases and, especially for the consumption of regulatory and payer audiences, will require data validation and compliance with information governance models to drive the fidelity and broad applicability of these techniques. Figure 1View largeDownload PPTFigure 1View largeDownload PPT Close modal

Propensity Score Methods and Difference-in-Differences with an Exogenous Time-Varying Confounder: Evaluation of Methods

Quasi-experimental designs (QEDs) are used to estimate a treatment effect without randomization. Confounders have a causal relationship with the outcome and probability of treatment adoption and if unaccounted for can bias treatment effect estimates. A variable considered a confounder prior to treatment can change after treatment has occurred (i.e., a time-varying confounder) not as a result of treatment (what we call an exogenous time-varying confounder). If the post-treatment value causally affects the outcome to change and this post-treatment value of the exogenous time-varying confounder is unaccounted for, then the treatment effect may be biased. We review the Rubin Causal Model and QED assumptions and the effect an exogenous time-varying confounder has on the ability of QEDs to produce an appropriate counterfactual. We conduct a simulation study evaluating propensity score and difference-in-differences based methods for estimating a treatment effect with an exogenous time-varying confounder. Propensity score weighted two-way fixed effects, inverse probability weighted, or doubly robust difference-in-differences methods, each with propensity scores estimated using post-implementation values of the exogenous time-varying confounder, proved least biased when the exogenous time-varying confounder changed differentially for members of the treatment and control groups.

A meta-analytic framework to adjust for bias in external control studies.

While randomized controlled trials (RCTs) are the gold standard for estimating treatment effects in medical research, there is increasing use of and interest in using real-world data for drug development. One such use case is the construction of external control arms for evaluation of efficacy in single-arm trials, particularly in cases where randomization is either infeasible or unethical. However, it is well known that treated patients in non-randomized studies may not be comparable to control patients-on either measured or unmeasured variables-and that the underlying population differences between the two groups may result in biased treatment effect estimates as well as increased variability in estimation. To address these challenges for analyses of time-to-event outcomes, we developed a meta-analytic framework that uses historical reference studies to adjust a log hazard ratio estimate in a new external control study for its additional bias and variability. The set of historical studies is formed by constructing external control arms for historical RCTs, and a meta-analysis compares the trial controls to the external control arms. Importantly, a prospective external control study can be performed independently of the meta-analysis using standard causal inference techniques for observational data. We illustrate our approach with a simulation study and an empirical example based on reference studies for advanced non-small cell lung cancer. In our empirical analysis, external control patients had lower survival than trial controls (hazard ratio: 0.907), but our methodology is able to correct for this bias. An implementation of our approach is available in the R package ecmeta.

Surrogate end points in cardio-thoracic trials: a call for better reporting and improved interpretation of trial findings.

Evidence to inform regulatory, funding and clinical decisions about health interventions should come from well-conducted clinical trials. Despite the challenges associated to their design, conduct and generalizability, randomized controlled trials (RCTs) are typically considered the studies providing the least biased estimates of treatment effects [1]. The selection of outcomes to be evaluated in RCTs and what specific measures to use for them—the trial end points—is a complex yet pivotal step in the process. Outcomes should allow to reasonably monitor the trial objectives and provide meaningful information for the different stakeholders and evidence-based decision-making at different levels [2]. For instance, in an RCT evaluating a robot-assisted versus video-assisted thoracic surgery in the early-stage lung cancer, the clinicians might prioritize overall or disease-free survival, whilst the patients might be highly interested in the effect of treatment on their functional status or health-related quality of life. Both concepts are typically measured through patient-reported outcome measures, which can be either generic (e.g. EQ-5D-5 l [3]) or condition specific (e.g. EORTC QLQ-LC29 [4]). Healthcare managers and policymakers might look for information on the cost-effectiveness profile of the experimental approach, or the budget impact expected from its implementation within 3–5 years [5].

Sponsorship Bias in Clinical Trials in the Dental Application of Probiotics: A Meta-Epidemiological Study.

Many experimental and clinical trials have investigated the dental application of probiotics, although the evidence concerning the effects of probiotic supplements is conflicting. We aimed to examine whether sponsorship in trials about dental applications of probiotics is associated with biased estimates of treatment effects. Overall, 13 meta-analyses involving 48 randomized controlled trials (23 with high risk of sponsorship bias, 25 with low risk) with continuous outcomes were included. Effect sizes were calculated from differences in means of first reported continuous outcomes, divided by the pooled standard deviation. For each meta-analysis, the difference in standardized mean differences between high-risk and low-risk trials was estimated by random effects meta-regression. Differences in standardized mean differences (DSMDs) were then calculated via meta-analyses in a random effects meta-analysis model. A combined DSMD of greater than zero indicated that high-risk trials showed more significant treatment effects than low-risk trials. The results show that trials with a high risk of sponsorship bias showed more significant intervention effects than did low-risk trials (combined DSMD, 0.06; 95% confidence interval, 0.3 to 0.9; p < 0.001), with low heterogeneity among meta-analyses (I2 = 0%; between-meta-analyses variance τ2 = 0.00). Based on our study, high-risk clinical trials with continuous outcomes reported more favorable intervention effects than did low-risk trials in general.

Should RECOVERY have used response adaptive randomisation? Evidence from a simulation study

BackgroundThe Randomised Evaluation of COVID-19 Therapy (RECOVERY) trial is aimed at addressing the urgent need to find effective treatments for patients hospitalised with suspected or confirmed COVID-19. The trial has had many successes, including discovering that dexamethasone is effective at reducing COVID-19 mortality, the first treatment to reach this milestone in a randomised controlled trial. Despite this, it continues to use standard or ‘fixed’ randomisation to allocate patients to treatments. We assessed the impact of implementing response adaptive randomisation within RECOVERY using an array of performance measures, to learn if it could be beneficial going forward. This design feature has recently been implemented within the REMAP-CAP platform trial.MethodsTrial data was simulated to closely match the data for patients allocated to standard care, dexamethasone, hydroxychloroquine, or lopinavir-ritonavir in the RECOVERY trial from March-June 2020, representing four out of five arms tested throughout this period. Trials were simulated in both a two-arm trial setting using standard care and dexamethasone, and a four-arm trial setting utilising all above treatments. Two forms of fixed randomisation and two forms of response-adaptive randomisation were tested. In the two-arm setting, response-adaptive randomisation was implemented across both trial arms, whereas in the four-arm setting it was implemented in the three non-standard care arms only. In the two-arm trial, randomisation strategies were performed at the whole trial level as well as within three pre-specified patient subgroups defined by patients’ respiratory support level.ResultsAll response-adaptive randomisation strategies led to more patients being given dexamethasone and a lower mortality rate in the trial. Subgroup specific response-adaptive randomisation reduced mortality rates even further. In the two-arm trial, response-adaptive randomisation reduced statistical power compared to FR, with subgroup level adaptive randomisation exhibiting the largest power reduction. In the four-arm trial, response-adaptive randomisation increased statistical power in the dexamethasone arm but reduced statistical power in the lopinavir arm. Response-adaptive randomisation did not induce any meaningful bias in treatment effect estimates nor did it cause any inflation in the type 1 error rate.ConclusionsUsing response-adaptive randomisation within RECOVERY could have increased the number of patients receiving the optimal COVID-19 treatment during the trial, while reducing the number of patients needed to attain the same study power as the original study. This would likely have reduced patient deaths during the trial and lead to dexamethasone being declared effective sooner. Deciding how to balance the needs of patients within a trial and future patients who have yet to fall ill is an important ethical question for the trials community to address. Response-adaptive randomisation deserves to be considered as a design feature in future trials of COVID-19 and other diseases.

Unmeasured confounding in nonrandomized studies: quantitative bias analysis in health technology assessment.

Evidence generated from nonrandomized studies (NRS) is increasingly submitted to health technology assessment (HTA) agencies. Unmeasured confounding is a primary concern with this type of evidence, as it may result in biased treatment effect estimates, which has led to much criticism of NRS by HTA agencies. Quantitative bias analysesare a group of methods that have been developed in the epidemiological literature to quantify the impact of unmeasured confounding and adjust effect estimates from NRS. Key considerations for application in HTA proposed in this article reflect the need to balance methodological complexity with ease of application and interpretation, and the need to ensure the methods fit within the existing frameworks used to assess nonrandomized evidence by HTA bodies.

A FLEXIBLE SENSITIVITY ANALYSIS APPROACH FOR UNMEASURED CONFOUNDING WITH MULTIPLE TREATMENTS AND A BINARY OUTCOME WITH APPLICATION TO SEER-MEDICARE LUNG CANCER DATA.

In the absence of a randomized experiment, a key assumption for drawing causal inference about treatment effects is the ignorable treatment assignment. Violations of the ignorability assumption may lead to biased treatment effect estimates. Sensitivity analysis helps gauge how causal conclusions will be altered in response to the potential magnitude of departure from the ignorability assumption. However, sensitivity analysis approaches for unmeasured confounding in the context of multiple treatments and binary outcomes are scarce. We propose a flexible Monte Carlo sensitivity analysis approach for causal inference in such settings. We first derive the general form of the bias introduced by unmeasured confounding, with emphasis on theoretical properties uniquely relevant to multiple treatments. We then propose methods to encode the impact of unmeasured confounding on potential outcomes and adjust the estimates of causal effects in which the presumed unmeasured confounding is removed. Our proposed methods embed nested multiple imputation within the Bayesian framework, which allow for seamless integration of the uncertainty about the values of the sensitivity parameters and the sampling variability, as well as use of the Bayesian Additive Regression Trees for modeling flexibility. Expansive simulations validate our methods and gain insight into sensitivity analysis with multiple treatments. We use the SEER-Medicare data to demonstrate sensitivity analysis using three treatments for early stage non-small cell lung cancer. The methods developed in this work are readily available in the R package SAMTx.

Accounting for publication bias using a bivariate trim and fill meta-analysis procedure.

In research synthesis, publication bias (PB) refers to the phenomenon that the publication of a study is associated with the direction and statistical significance of its results. Consequently, it may lead to biased (commonly optimistic) estimates of treatment effects. Visualization tools such as funnel plots have been widely used to investigate PB in univariate meta-analyses. The trim and fill procedure is a nonparametric method to identify and adjust for PB. It is popular among applied scientists due to its simplicity. However, most visualization tools and PB correction methods focus on univariate outcomes. For a meta-analysis with multiple outcomes, the conventional univariate trim and fill method can only account for different outcomes separately and thus may lead to inconsistent conclusions. In this article, we propose a bivariate trim and fill procedure to simultaneously account for PB in the presence of two outcomes that are possibly associated. Based on a recently developed galaxy plot for bivariate meta-analysis, the proposed procedure uses a data-driven imputation algorithm to detect and adjust PB. The method relies on the symmetry of the galaxy plot and assumes that some studies are suppressed based on a linear combination of outcomes. The method projects bivariate outcomes along a particular direction, uses the univariate trim and fill method to estimate the number of trimmed and filled studies, and yields consistent conclusions about PB. The proposed approach is validated using simulated data and is applied to a meta-analysis of the efficacy and safety of antidepressant drugs.

Real-world data: Assessing electronic health records and medical claims data to support regulatory decision-making for drug and biological products.

Real-world data: Assessing electronic health records and medical claims data to support regulatory decision-making for drug and biological products.