Are morbidity and mortality estimates from randomized controlled trials externally valid? A comparison of outcomes among infants enrolled into an RCT or a cohort study in Botswana

Neil Thivalapill,Jean Leidner,Gbolahan Ajibola,Mompati Mmalane,Rebecca Zash,Shahin Lockman,Joseph Makhema,Kathleen Powis,Roger L Shapiro

doi:10.1186/s12874-021-01343-5

Abstract

BackgroundThe external validity of the randomized controlled trial (RCT) refers to the extent to which the results of the RCT apply to the relevant, non-trial population and is impacted by its eligibility criteria, its organization, and its delivery of the intervention. Here, we compared the outcomes of mortality and hospitalization between an RCT and a cohort study that concurrently enrolled HIV-exposed uninfected (HEU) newborns in Botswana.MethodsThe Mpepu Study (the RCT) was a clinical trial which determined that co-trimoxazole (CTX) provided no survival benefit for HEUs, allowing both arms of the RCT to be used. The Maikaelelo study (the cohort study) was a prospective observational study that enrolled HEU newborns with telephone follow-up and no in-person visits. Rates of death and hospitalization in the pooled population, were modeled using cox-proportional hazards models for time to death or time to first hospitalization, with study setting (RCT vs. cohort study) as an independent variable. The causal effect of study setting on morbidity and mortality was obtained through a treatment effects approach.ResultsIn total, 4,010 infants were included; 1,306 were enrolled into the cohort study and 2,704 were enrolled into the RCT. No significant differences in mortality were observed between the two study settings (HR: 1.28, 95% CI: 0.76, 2.13), but RCT participants had a lower risk of hospitalization (HR: 0.72, 95% CI: 0.58, 0.89) that decreased with age. However, RCT participants had a higher risk of hospitalization within the first six months of life. The causal risk difference in hospitalizations attributable to the RCT setting was -0.03 (95% CI: -0.05, -0.01).ConclusionsChildren in an RCT with rigorous application of national standard of care guidelines experienced a significantly lower risk of hospitalization than children participating in a cohort study that did not alter clinical care. Future research is needed to further investigate outcome disparities when real-world results fail to mirror those achieved in a clinical trial.Trial registrationThe Mpepu Trial was funded by the U.S. National Institutes of Health (No. NCT01229761) and the Maikaelelo Study was funded primarily by the U.S. Centers for Disease Control and Prevention (32AI007433-21).

Highlights

The external validity of the randomized controlled trial (RCT) refers to the extent to which the results of the Randomized controlled trials (RCTs) apply to the relevant, non-trial population and is impacted by its eligibility criteria, its organization, and its delivery of the intervention
While the goal of this work is to determine whether the rates of morbidity and mortality obtained in the RCT setting were valid in the nonRCT context, we cannot comment on whether the effect estimate of CTX was valid given that the cohort study did not assess the efficacy of this intervention
Given that the cohort study was never intended to measure the effect of CTX as was the case for the RCT, it is impossible to determine whether these domains of the RCT influenced its effect estimate of CTX on mortality and morbidity in HIV-exposed uninfected (HEU) infants

Summary

Introduction

The external validity of the randomized controlled trial (RCT) refers to the extent to which the results of the RCT apply to the relevant, non-trial population and is impacted by its eligibility criteria, its organization, and its delivery of the intervention. Randomized controlled trials (RCTs) are often considered the highest grade of evidence in clinical research because the randomization process aims to balance both known and unknown confounders [1] In doing so, these trials seek to measure the true causal effect of an intervention on a given outcome [2]. The PRECIS-2 tool considers nine of these domains, each of which is scored on whether the trial answers questions about pragmatism under realworld conditions or efficacy under ideal conditions [6] These domains include the eligibility criteria, the recruitment of study subjects, the setting in which the study was conducted, the organization of the study, the delivery of the intervention and the adherence to it, the follow-up methods used, the primary outcome measured, and the methods of the primary analysis

Objectives

Methods

Results

Discussion

Conclusion