PCN59 CHARACTERISING ERRORS INCURRED BY MODEL SELECTION BASED ON AIC FOR EXTRAPOLATION OF SURVIVAL IN HEALTH TECHNOLOGY ASSESSMENT - A SIMULATION STUDY

Abstract

Analysts are frequently required to extrapolate time-to-event outcomes beyond the observed follow-up period within clinical trials for health technology assessment (HTA). A typical example within oncology is extrapolation of overall survival beyond the trial observation period. Increasingly, decision-makers are asked to assess the performance and validity of multiple models for relatively immature trial data. Current guidance focuses on likelihood and information criteria to rank competing models. However, substantial discretion remains for the analysts in selecting the parametric distributions while avoiding overfitting the data. Our study characterises the impact of decision-impacting errors which result from reliance on these criteria alone within simulated trial data with varying degrees of follow-up. We defined errors as selection of models that did not contain the true mean within the 95% confidence interval of the predicted mean. We simulated data with sample sizes comparable to phase 3 oncology trials (N = 300). Time-to-death was drawn randomly from prespecified distributions, and maximum follow-up period was varied, within the simulation, relative to the true median of the distribution. Maximum-likelihood models, assuming a range of potential distributions were fitted to the data. Model selection was based upon minimum Akaike Information Criterion (AIC). Errors were measured relative to the underlying true distribution. Increased follow-up resulted in decreased rates of error across all underlying true distributions. Focusing on data derived from a Weibull distribution, the 95% CI for the mean of the minimal AIC model contained the true mean in approximately 50% to 19% of cases for follow-up times of 75% to 200% respectively of the median survival time. Decreasing error was associated with increasing shape parameter. This study demonstrates the challenges faced by analysts and decision-makers when relying on information criteria alone in the selection of parametric distributions for extrapolation of time-to-event outcomes in HTA.