Survival Augmented Patient Preference Incorporated Reinforcement Learning to Evaluate Tailoring Variables for Personalized Healthcare

Yingchao Zhong,Chang Wang,Lu Wang

doi:10.3390/stats4040046

Abstract

In this paper, we consider personalized treatment decision strategies in the management of chronic diseases, such as chronic kidney disease, which typically consists of sequential and adaptive treatment decision making. We investigate a two-stage treatment setting with a survival outcome that could be right censored. This can be formulated through a dynamic treatment regime (DTR) framework, where the goal is to tailor treatment to each individual based on their own medical history in order to maximize a desirable health outcome. We develop a new method, Survival Augmented Patient Preference incorporated reinforcement Q-Learning (SAPP-Q-Learning) to decide between quality of life and survival restricted at maximal follow-up. Our method incorporates the latent patient preference into a weighted utility function that balances between quality of life and survival time, in a Q-learning model framework. We further propose a corresponding m-out-of-n Bootstrap procedure to accurately make statistical inferences and construct confidence intervals on the effects of tailoring variables, whose values can guide personalized treatment strategies.

Highlights

For chronic illnesses, patients often have to navigate a series of treatment decisions.It has been increasingly recognized that due to patient heterogeneity based on genetics, environmental factors, various other factors and the interplay between the factors, a good treatment plan needs to be both personalized and adaptive to a patient’s changing clinical course
Dynamic treatment regimes (DTRs) are algorithmic solutions to this clinical problem, where a dynamic treatment regime (DTR) consists of a sequence of treatment decision rules that adapt over time in response to an individual’s clinical response and health outcome trajectory
We explore, synthesize, and adapt existing methods in the literature to create a new method for estimating the optimal treatment regime and constructing stagespecific confidence intervals that fit our scenario

Summary

Introduction

Patients often have to navigate a series of treatment decisions. It has been increasingly recognized that due to patient heterogeneity based on genetics, environmental factors, various other factors and the interplay between the factors, a good treatment plan needs to be both personalized and adaptive to a patient’s changing clinical course. Along with the development of data science, machine learning flavored methods were developed for DTR estimation, including tree-based and list-based methods ([7,8,9,10]), classification type methods ([11,12]), and stochastic tree search methods [13]

Methods

Results

Conclusion