PNS15 IMPROVING UNDERSTANDING OF UNCERTAINTY IN ECONOMIC MODELS THROUGH MACHINE LEARNING : POSSIBILITIES AND RECOMMENDATIONS

Abstract

Probabilistic sensitivity analysis (PSA) conducted with economic models are typically summarized by a cost-effectiveness acceptability curve or a scatterplot on the cost-effectiveness plane. This method ignores considerable data which could be mined to gain better insights into what drives parametric uncertainty in a model. We applied classification and regression trees (CART), a common algorithm found in machine-learning, to PSA to examine whether and how certain parameters can best be identified as drivers of model uncertainty. We applied CART to the output of PSA from several published Markov models in oncology which were adapted for our purpose. Each model’s parameter space included variables for costs, utilities, natural history, and patient characteristics, which were mined to explain outcomes for incremental cost, QALYs, life-years, and incremental cost-effectiveness ratios (ICER). The R library rpart was used to apply CART. We report our results qualitatively and offer recommendations for use. CART results were more interpretable and plausible when the number of simulations of the PSA was high (over 5000) and a thoughtful pre-selection of plausible model parameters to include in analysis was made. Incrementally selecting parameters by first applying CART to incremental costs and QALYs before applying to ICERs also improved interpretability. Application of CART to the entire parameter space without previous selection of parameters led to more complex or overfitted results that were more difficult to interpret. Comparison of most prominent drivers of parametric uncertainty identified by CART were not generally in agreement with those identified by one-way sensitivity analyses, suggesting that model uncertainty is likely driven by variation in several parameters simultaneously. CART when applied appropriately can enhance our understanding of parametric uncertainty in economic models. With greater experience using machine-learning techniques there is potential to enhance HTA decision-making especially where concerns on model uncertainty is high.