Graphing and reporting heterogeneous treatment effects through reference classes

James A Watson,Chris C Holmes

doi:10.1186/s13063-020-04306-1

Abstract

BackgroundExploration and modelling of heterogeneous treatment effects as a function of baseline covariates is an important aspect of precision medicine in randomised controlled trials (RCTs). Randomisation generally guarantees the internal validity of an RCT, but heterogeneity in treatment effect can reduce external validity. Estimation of heterogeneous treatment effects is usually done via a predictive model for individual outcomes, where one searches for interactions between treatment allocation and important patient baseline covariates. However, such models are prone to overfitting and multiple testing and typically demand a transformation of the outcome measurement, for example, from the absolute risk in the original RCT to log-odds of risk in the predictive model.MethodsWe show how reference classes derived from baseline covariates can be used to explore heterogeneous treatment effects via a two-stage approach. We first estimate a risk score which captures on a single dimension some of the heterogeneity in outcomes of the trial population. Heterogeneity in the treatment effect can then be explored via reweighting schemes along this axis of variation. This two-stage approach bypasses the search for interactions with multiple covariates, thus protecting against multiple testing. It also allows for exploration of heterogeneous treatment effects on the original outcome scale of the RCT. This approach would typically be applied to multivariable models of baseline risk to assess the stability of average treatment effects with respect to the distribution of risk in the population studied.Case studyWe illustrate this approach using the single largest randomised treatment trial in severe falciparum malaria and demonstrate how the estimated treatment effect in terms of absolute mortality risk reduction increases considerably in higher risk strata.Conclusions‘Local’ and ‘tilting’ reweighting schemes based on ranking patients by baseline risk can be used as a general approach for exploring, graphing and reporting heterogeneity of treatment effect in RCTs.Trial registrationISRCTN clinical trials registry: ISRCTN50258054. Prospectively registered on 22 July 2005.

Highlights

Exploration and modelling of heterogeneous treatment effects as a function of baseline covariates is an important aspect of precision medicine in randomised controlled trials (RCTs)
Multivariable risk-based ranking of trial individuals In the following we consider RCT data of the form {xi, yi, ti}Ni=1, where xi is a vector of baseline patient covariates for the ith patient, and yi is their observed outcome after receiving a randomised treatment allocation indicated by ti
A consequence of this variation in baseline risk and its influence on the reported treatment effect is that average treatment effect (ATE) can be misleading when used to guide treatment recommendations at the individual level

Summary

Methods

Multivariable risk-based ranking of trial individuals In the following we consider RCT data of the form {xi, yi, ti}Ni=1, where xi is a vector of baseline patient covariates for the ith patient, and yi is their observed outcome after receiving a randomised treatment allocation indicated by ti. Local kernels target a specific individual focussed at their quantile of risk qi for the ith subject by considering the treatment outcome of other individuals in a local neighbourhood of risk-adjacent individuals, with q’s close to qi These local reference classes are parameterised by their bandwidth (radius) γ ∈[ 0, 1], which defines the proportion of subjects in the window ‘close’ to subject i, which are used to estimate the ITE of the ith subject. Estimation of a risk-based CATE using reference class forecasting with exponential tilting Local reweighting schemes provide a principled approach for determining an ITE for a given subject in the trial up to a certain accuracy, with a certain bias-variance trade-off (see the section). The estimation of ITEs using an Epanechnikov kernel with varying bandwidth gives very similar results (panel c2)

Background

Findings

Discussion