Apples and oranges? baseline imbalance in hysteropexy vs hysterectomy comparisons.

Assessing the impact of attrition in randomized controlled trials

The impact of trial baseline imbalances should be considered in systematic reviews: a methodological case study

The propensity score has been proposed, and for the most part accepted, as a tool to allow for the evaluation of medical interventions in the presence of baseline imbalances arising in the context of observational studies. The lack of an analogous tool to allow for the evaluation of medical interventions in the presence of potentially systematic baseline imbalances in randomized trials has required the use of ad hoc methods. This, in turn, leads to challenges to the conclusions. For example, much of the controversy surrounding recommendations for or against mammography for some age groups stems from the fact that all the randomized trials to study mammography had baseline imbalances, to some extent, in important prognostic covariates. While some of these trials used cluster randomization, baseline imbalances are prevalent also in individually randomized trials. We provide a systematic approach for evaluating medical interventions in the presence of potentially systematic baseline imbalances in individually randomized trials with allocation concealment. Specifically, we define the reverse propensity score as the probability, conditional on all previous allocations and the allocation procedure (restrictions on the randomization), that a given patient will receive a given treatment. We demonstrate how the reverse propensity score allows for both detection of and correction for selection bias, or systematic baseline imbalances.

Randomized clinical trials attempt to reduce bias and create similar groups at baseline to infer causal effects. In meta-analyses, baseline imbalance may threaten the validity of the treatment effects. This meta-epidemiological study examined baseline imbalance in comparisons of exercise and antihypertensive medicines. Baseline data for systolic blood pressure, diastolic blood pressure, and age were extracted from a network meta-analysis of 391 randomized trials comparing exercise types and antihypertensive medicines. Fixed-effect meta-analyses were used to determine the presence of baseline imbalance and/or inconsistency. Meta-regression analyses were conducted on sample size, the risk of bias for allocation concealment, and whether data for all randomized participants were presented at baseline. In one exercise comparison, the resistance group was 0.3 years younger than the control group (95% confidence interval 0.6 to 0.1). Substantial inconsistency was observed in other exercise comparisons. Less data were available for medicines, but there were no occurrences of baseline imbalance and only a few instances of inconsistency. Several moderator analyses identified significant associations. We identified baseline imbalance as well as substantial inconsistency in exercise comparisons. Researchers should consider conducting meta-analyses of key prognostic variables at baseline to ensure balance across trials.

ObjectivesThe NINDS rt-PA Stroke Study is frequently cited in support of alteplase for acute ischemic stroke within 3 h of symptom onset. Multiple post-hoc reanalyses of this trial have been published to adjust for a baseline imbalance in stroke severity. We performed a risk of selection bias assessment and reanalyzed trial data to determine if the etiology of this baseline imbalance was more likely due to random chance or randomization errors.MethodsA risk of selection bias assessment was conducted using signaling questions from the Cochrane Risk of Bias 2 (ROB 2) tool. Four sensitivity analyses were conducted on the trial data based on the randomization process: assessment of imbalances in allocation in unique strata; adherence to a pre-specified restriction on randomization between time strata at each randomization center; assessment of differences in baseline computed tomography (CT) results in unique strata; and comparison of baseline characteristics between allocation groups within each time strata. A multivariable logistic regression model was used to compare reported treatment effects with revised treatment effects after adjustment of baseline imbalances identified in the sensitivity analyses.ResultsBased on criteria from the ROB 2 tool, the risk of bias arising from the randomization process was high. Sensitivity analyses found 11 of 16 unique strata deviated from the expected 1:1 allocation ratio. Three randomization centers violated an apriori rule regarding a maximum difference in allocation between the time strata. Three unique strata had imbalances in baseline CT results that prognostically favored alteplase. Four imbalances in baseline characteristics were identified in the 91–180-min time stratum that all prognostically favored alteplase and were consistent with a larger alteplase treatment effect size compared to the 0–90-min time stratum. After adjustments for baseline imbalances, all reported treatment effects were reduced. Three out of seven originally positive reported results were revised to non-significant.ConclusionThis risk of selection bias assessment revealed a high risk of selection bias in the NINDS rt-PA Stroke Study. Sensitivity analyses conducted based on the randomization process supported this assessment. Baseline imbalances in the trial were more likely due to randomization errors than random chance. Adjusted analyses accounting for baseline imbalances revealed a reduction in reported treatment effects supporting the presence of selection bias in the trial. Treatment decisions and guideline recommendations based on the original treatment effect reported in the NINDS rt-PA Stroke Study should be done cautiously.

Ensuring the comparability of comparison groups: is randomization enough?

We describe methods for meta-analysis of randomised trials where a continuous outcome is of interest, such as blood pressure, recorded at both baseline (pre treatment) and follow-up (post treatment). We used four examples for illustration, covering situations with and without individual participant data (IPD) and with and without baseline imbalance between treatment groups in each trial. Given IPD, meta-analysts can choose to synthesise treatment effect estimates derived using analysis of covariance (ANCOVA), a regression of just final scores, or a regression of the change scores. When there is baseline balance in each trial, treatment effect estimates derived using ANCOVA are more precise and thus preferred. However, we show that meta-analysis results for the summary treatment effect are similar regardless of the approach taken. Thus, without IPD, if trials are balanced, reviewers can happily utilise treatment effect estimates derived from any of the approaches. However, when some trials have baseline imbalance, meta-analysts should use treatment effect estimates derived from ANCOVA, as this adjusts for imbalance and accounts for the correlation between baseline and follow-up; we show that the other approaches can give substantially different meta-analysis results. Without IPD and with unavailable ANCOVA estimates, reviewers should limit meta-analyses to those trials with baseline balance. Trowman's method to adjust for baseline imbalance without IPD performs poorly in our examples and so is not recommended. Finally, we extend the ANCOVA model to estimate the interaction between treatment effect and baseline values and compare options for estimating this interaction given only aggregate data.

BackgroundBaseline imbalances have been identified in randomized trials of evolocumab and alirocumab. Our aim was to quantitatively assess (1) the presence of systematic baseline differences, and (2) the relationship of baseline differences with effects on low-density lipoprotein-cholesterol (LDL-c) and clinical outcomes in the trials.MethodsWe performed a meta-epidemiological study. PubMed, Embase, regulatory reports, ClinicalTrials.gov and company websites were searched for trials. Seven baseline characteristics (mean age, LDL-c, BMI, percentage males, diabetics, smokers, and hypertensives) and five outcomes (LDL-c, major adverse cardiac events, serious adverse events, any adverse events, all-cause mortality) were extracted. We calculated (1) range and distribution of baseline imbalances (sign-test), (2) pooled baseline differences and heterogeneity (meta-analysis), (3) differences in SDs around continuous variables (sign-test and pooling), and (4) the relationship of baseline differences with outcomes (meta-regression). The comparisons of PCSK9-inhibitor groups with either placebo or ezetimibe were analysed separately and combined.ResultsWe identified 43 trials with 63,193 participants. Baseline characteristics were frequently missing. Many trials showed small baseline imbalances, but some large imbalances. Only baseline BMI showed a statistically significant lower pooled mean for the drug versus placebo groups (MD -0.16; 95% CI -0.24 to -0.09). Heterogeneity in baseline imbalances was present in six placebo- and five ezetimibe-comparisons. Heterogeneity was statistically significant for BMI, males, diabetics and hypertensives in the combined comparisons. There was a statistically significant preponderance for larger SDs in the PCSK9-inhibitor versus control groups (sign-test age 0.014; LDL-c 0.014; BMI 0.049). Meta-regression showed clinically relevant relationships of baseline imbalances in age, BMI and diabetics with the risk of any adverse events and the risk of mortality. Two relationships were statistically significant: A higher mean BMI in the drug versus control group with a decreased risk of mortality (beta − 0.56; 95% CI -1.10 to -0.02), and a higher proportion of diabetics with an increased risk of any adverse events (beta 0.02; 95% 0.01 to 0.04).ConclusionsHeterogeneous baseline imbalances and systematically different SDs were present in evolocumab and alirocumab trials, so study groups cannot be assumed to be comparable. These findings raise concerns about the design and conduct of the randomization procedures.

BackgroundThe optimal timing of resection for synchronous colorectal liver metastases is still controversial. Retrospective cohort studies always had baseline imbalances in comparing simultaneous resection with staged strategy. Significantly more patients with mild conditions received simultaneous resections. Previous published meta-analyses based on these studies did not correct these biases, resulting in low reliability. Our meta-analysis was conducted to compensate for this deficiency and find candidates for each surgical strategy.MethodsA systemic search for major databases and relevant journals from January 2000 to April 2013 was performed. The primary outcomes were postoperative mortality, morbidity, overall survival and disease-free survival. Other outcomes such as number of patients need blood transfusion and length of hospital stay were also assessed. Baseline analyses were conducted to find and correct potential confounding factors.Results22 studies with a total of 4494 patients were finally included. After correction of baseline imbalance, simultaneous and staged resections were similar in postoperative mortality (RR = 1.14, P = 0.52), morbidity (RR = 1.02, P = 0.85), overall survival (HR = 0.96, P = 0.50) and disease-free survival (HR = 0.97, P = 0.87). Only in pulmonary complications, simultaneous resection took a significant advantage (RR = 0.23, P = 0.003). The number of liver metastases was the major factor interfering with selecting surgical strategies. With >3 metastases, simultaneous and staged strategies were almost the same in morbidity (49.4% vs. 50.9%). With ≤3 metastases, staged resection caused lower morbidity (13.8% vs. 17.2%), not statistically significant.ConclusionsThe number of liver metastases was the major confounding factor for postoperative morbidity, especially in staged resections. Without baseline imbalances, simultaneous took no statistical significant advantage in safety and efficacy. Considering the inherent limitations of this meta-analysis, the results should be interpret and applied prudently.

Background: This paper addresses one threat to the internal validity of a randomized controlled trial (RCT), selection bias. Many authors argue that random allocation is used to ensure baseline equality between study conditions in a given study and that statistically significant differences at pretest mean that randomisation has failed. Purpose: The purpose of this study was to clarify the role of random allocation in an RCT study. Is the role of random allocation to protect against selection bias? And does it have a further role, namely to ensure baseline equality and the absence of statistically significant differences between study conditions at pretest? Setting: The participants for this study were 229 children in 1st and 2nd grade and data were collected as part of an RCT evaluation of a volunteer reading programme piloted in Ireland, Wizards of Words (WoW). Intervention: Not applicable. Research design: The allocation procedure adopted in this study was stratified and blocked random allocation. Data collection and analysis: Data were collected using standardised and criterion-referenced tests of reading ability. Data were collected by qualified Speech and Language Therapists. Independent-samples t-tests were used to analyse pretest data. Findings: The role of random allocation is to protect against selection bias, and statistically significant baseline differences can result even when random allocation has been successful. Whether or not random allocation has been successful is determined by the generation of the random allocation sequence and the steps taken to ensure its concealment. The size of differences between study conditions at pretest can be important for the analysis of posttest data but does not by itself determine whether random allocation was successful. In addition, there are serious concerns about the appropriateness of tests of significance when comparing two study conditions at pretest. Keywords: baseline equality; baseline imbalance; random allocation; selection bias; test of statistical significance

DEPERROR: Risks of systematic errors in drug and non-drug randomized clinical trials assessing intervention effects in patients with unipolar depression

Our objective was to compare the pattern of organ dysfunctions and outcomes of critically ill patients with systemic lupus erythematosus (SLE) with patients with other systemic rheumatic diseases (SRD). We studied 116 critically ill SRD patients, 59 SLE and 57 other-SRD patients. The SLE group was younger and included more women. Respiratory failure (61%) and shock (39%) were the most common causes of ICU admission for other-SRD and SLE groups, respectively. ICU length-of-stay was similar for the two groups. The 60-day survival adjusted for the groups' baseline imbalances was not different (P = 0.792). Total SOFA scores were equal for the two groups at admission and during ICU stay, although respiratory function was worse in the other-SRD group at admission and renal and hematological functions were worse in the SLE group at admission. The incidence of severe respiratory dysfunction (respiratory SOFA >2) at admission was higher in the other-SRD group, whereas severe hematological dysfunction (hematological SOFA >2) during ICU stay was higher in the SLE group. SLE patients were younger and displayed a decreased incidence of respiratory failure compared to patients with other-SRDs. However, the incidences of renal and hematological failure and the presence of shock at admission were higher in the SLE group. The 60-day survival rates were similar.

Interventions to reduce childhood stunting burden require clinical trials with a primary outcome of linear growth. When growth is measured longitudinally, there are several options for including baseline measurements in the analysis. This study compares the performance of several methods. Randomized controlled trials evaluating a hypothetical intervention to improve length‐for‐age z‐score (LAZ) from birth through 24 months of age were simulated. The intervention effect was evaluated using linear regression and five methods for handling baseline measurements: comparing final measurements only (FINAL), comparing final measurement adjusted for baseline (ADJUST), comparing the change in the measurement over time (DELTA), adjusting for baseline when comparing the changes over time (DELTA+ADJUST) and adjusting for baseline in two‐step residuals approach (RESIDUALS). We calculated bias, precision and power of each method for scenarios with and without a baseline imbalance in LAZ. Using a 0.15 effect size at 18 months, FINAL and DELTA required 1200 and 1500 enroled participants, respectively, to reach 80% power, whereas ADJUST, DELTA+ADJUST and RESIDUALS only required 900 participants. The adjusted models also produced unbiased estimates when there was a baseline imbalance, whereas the FINAL and DELTA methods produced biased estimates, as large as 0.07 lower and higher, respectively, than the true effect. Adjusted methods required smaller sample size and produced more precise results than both DELTA and FINAL methods in all test scenarios. If randomization fails, and there is an imbalance in LAZ at baseline, DELTA and FINAL methods can produce biased estimates, but adjusted models remain unbiased. These results warn against using the FINAL or DELTA methods.

BackgroundAnalysis of variance (ANOVA), change-score analysis (CSA) and analysis of covariance (ANCOVA) respond differently to baseline imbalance in randomized controlled trials. However, no empirical studies appear to have quantified the differential bias and precision of estimates derived from these methods of analysis, and their relative statistical power, in relation to combinations of levels of key trial characteristics. This simulation study therefore examined the relative bias, precision and statistical power of these three analyses using simulated trial data.Methods126 hypothetical trial scenarios were evaluated (126 000 datasets), each with continuous data simulated by using a combination of levels of: treatment effect; pretest-posttest correlation; direction and magnitude of baseline imbalance. The bias, precision and power of each method of analysis were calculated for each scenario.ResultsCompared to the unbiased estimates produced by ANCOVA, both ANOVA and CSA are subject to bias, in relation to pretest-posttest correlation and the direction of baseline imbalance. Additionally, ANOVA and CSA are less precise than ANCOVA, especially when pretest-posttest correlation ≥ 0.3. When groups are balanced at baseline, ANCOVA is at least as powerful as the other analyses. Apparently greater power of ANOVA and CSA at certain imbalances is achieved in respect of a biased treatment effect.ConclusionsAcross a range of correlations between pre- and post-treatment scores and at varying levels and direction of baseline imbalance, ANCOVA remains the optimum statistical method for the analysis of continuous outcomes in RCTs, in terms of bias, precision and statistical power.

Reply