Generalizability of an Automatic Explanation Method for Machine Learning Prediction Results on Asthma-Related Hospital Visits in Patients With Asthma: Quantitative Analysis

Gang Luo,Claudia L Nau,William W Crawford,Michael Schatz,Corinna Koebnick,Robert S Zeiger

doi:10.2196/24153

Abstract

BackgroundAsthma exerts a substantial burden on patients and health care systems. To facilitate preventive care for asthma management and improve patient outcomes, we recently developed two machine learning models, one on Intermountain Healthcare data and the other on Kaiser Permanente Southern California (KPSC) data, to forecast asthma-related hospital visits, including emergency department visits and hospitalizations, in the succeeding 12 months among patients with asthma. As is typical for machine learning approaches, these two models do not explain their forecasting results. To address the interpretability issue of black-box models, we designed an automatic method to offer rule format explanations for the forecasting results of any machine learning model on imbalanced tabular data and to suggest customized interventions with no accuracy loss. Our method worked well for explaining the forecasting results of our Intermountain Healthcare model, but its generalizability to other health care systems remains unknown.ObjectiveThe objective of this study is to evaluate the generalizability of our automatic explanation method to KPSC for forecasting asthma-related hospital visits.MethodsThrough a secondary analysis of 987,506 data instances from 2012 to 2017 at KPSC, we used our method to explain the forecasting results of our KPSC model and to suggest customized interventions. The patient cohort covered a random sample of 70% of patients with asthma who had a KPSC health plan for any period between 2015 and 2018.ResultsOur method explained the forecasting results for 97.57% (2204/2259) of the patients with asthma who were correctly forecasted to undergo asthma-related hospital visits in the succeeding 12 months.ConclusionsFor forecasting asthma-related hospital visits, our automatic explanation method exhibited an acceptable generalizability to KPSC.International Registered Report Identifier (IRRID)RR2-10.2196/resprot.5039

Highlights

IntroductionPatient outcomes deteriorate, whereas health care costs increase
The objective of this study is to evaluate the generalizability of our automatic explanation method to Kaiser Permanente Southern California (KPSC) in forecasting asthma-related hospital visits
For forecasting asthma-related hospital visits, our automatic explanation method exhibited an acceptable generalizability to KPSC

Summary

Introduction

Patient outcomes deteriorate, whereas health care costs increase To address this problem, we recently considered many candidate features and developed two extreme gradient boosting (XGBoost) [23] machine learning models, one on Intermountain Healthcare data [24] and the other on Kaiser Permanente Southern California (KPSC) data [25], to forecast asthma-related hospital visits in the succeeding 12 months among patients with asthma with a higher accuracy. To address the interpretability issue of black-box models, we designed an automatic method to offer rule format explanations for any machine learning model’s forecasting results on imbalanced tabular data and to suggest customized interventions with no accuracy loss [26]. Our method worked well for explaining our Intermountain Healthcare model’s forecasting results [26], but its generalizability to other health care systems remains unknown

Objectives

Results

Conclusion