PRM116 - Do Models With Better Goodness Of Fit Ensure Less Biased Treatment Effect Estimates?

Abstract

There is widespread misconception that goodness-of-fit statistics are critical in determining whether treatment effects estimated from a model are unbiased. We aimed to examine whether models with better goodness-of-fit ensure less biased treatment effect estimates. We constructed a simulated dataset of 10,000 patients to compare two treatments on a continuous outcome of a prognostic score (mean: 0.82; range:-6.2to8.0) wherein a higher score indicates a higher risk of mortality within one year. By design in our simulation, the new treatment would lower the prognostic score by 0.1 compared to the traditional treatment. We then generated two study designs using this dataset: 1) a randomized controlled trial (RCT) with randomly assigned treatment; 2) an observational study design wherein patients receiving the new treatment tended to be sicker on observed (e.g. older age, higher comorbidity score) and unobserved covariates. Treatment effect estimates were obtained from the observational study design using: 1) multivariable linear regression accounting for observed covariates, and 2) instrumental variable two-stage least squares (2SLS) regression accounting for observed and unobserved confounders and were compared with 3) the regression results from the RCT. Bootstrapping with 1000 resamples and standardized errors (SE) were used to estimate out-of-sample prediction for each of the three models. Across the three models, multivariable linear regression had the highest goodness-of-fit (adjusted R-squared 0.215) and best out-of-sample prediction (SE: 2.15), but very biased treatment effect estimates due to unmeasured confounders (bias: 1.119, 95%CI: 1.101to1.137). The RCT regression had the lowest R-squared (0.001) and 2SLS had a R-squared of 0.122. Both RCT (bias: 0.003, 95%CI: -0.017to0.024) and 2SLS (bias: 0.004, 95%CI: -0.039to0.046) had unbiased estimates. Models with better goodness-of-fit would have better predictive power but do not ensure less biased treatment effect estimates. Correct study design and model specification are important for unbiased treatment effect estimates.