Dynamic Treatment Regimes Research Articles

Bayesian set of best dynamic treatment regimes: Construction and sample size calculation for SMARTswithbinary outcomes.

Sequential, multiple assignment, randomized trials (SMARTs) compare sequences of treatment decision rules called dynamic treatment regimes (DTRs). In particular, the Adaptive Treatment for Alcohol and Cocaine Dependence (ENGAGE) SMART aimed to determine the best DTRs for patients with a substance use disorder. While many authors have focused on a single pairwise comparison, addressing the main goal involves comparisons of >2 DTRs. For complex comparisons, there is a paucity of methods for binary outcomes. We fill this gap by extending the multiple comparisons with the best (MCB) methodology to the Bayesian binary outcome setting. The set of best is constructed based on simultaneous credible intervals. A substantial challenge for power analysis is the correlation between outcome estimators for distinct DTRs embedded in SMARTs due to overlapping subjects. We address this using Robins' G-computation formula to take a weighted average of parameter draws obtained via simulation from the parameter posteriors. We use non-informative priors and work with the exact distribution of parameters avoiding unnecessary normality assumptions and specification of the correlation matrix of DTR outcome summary statistics. We conduct simulation studies for both the construction of a set of optimal DTRs using the Bayesian MCB procedure and the sample size calculation for two common SMART designs. We illustrate our method on the ENGAGE SMART. The R package SMARTbayesR for power calculations is freely available on the Comprehensive R Archive Network (CRAN) repository. An RShiny app is available at https://wilart.shinyapps.io/shinysmartbayesr/.

Optimal dynamic treatment regime estimation using information extraction from unstructured clinical text.

The wide-scale adoption of electronic health records (EHRs) provides extensive information to support precision medicine and personalized health care. In addition to structured EHRs, we leverage free-text clinical information extraction (IE) techniques to estimate optimal dynamic treatment regimes (DTRs), a sequence of decision rules that dictate how to individualize treatments to patients based on treatment and covariate history. The proposed IE of patient characteristics closely resembles "The clinical Text Analysis and Knowledge Extraction System" and employs named entity recognition, boundary detection, and negation annotation. It also utilizes regular expressions to extract numerical information. Combining the proposed IE with optimal DTR estimation, we extract derived patient characteristics and use tree-based reinforcement learning (T-RL) to estimate multistage optimal DTRs. IE significantly improved the estimation in counterfactual outcome models compared to using structured EHR data alone, which often include incomplete data, data entry errors, and other potentially unobserved risk factors. Moreover, including IE in optimal DTR estimation provides larger study cohorts and a broader pool of candidate tailoring variables. We demonstrate the performance of our proposed method via simulations and an application using clinical records to guide blood pressure control treatments among critically ill patients with severe acute hypertension. This joint estimation approach improves the accuracy of identifying the optimal treatment sequence by 14-24% compared to traditional inference without using IE, based on our simulations over various scenarios. In the blood pressure control application, we successfully extracted significant blood pressure predictors that are unobserved or partially missing from structuredEHR.

Learning and Assessing Optimal Dynamic Treatment Regimes Through Cooperative Imitation Learning

Dynamic Treatment Regimes (DTRs) are sets of sequential decision rules that can be adapted over time to treat patients with a specific pathology. DTR consists of alternative treatment paths and any of these treatments can be adapted depending on the patient’s characteristics. Reinforcement Learning (RL) and Imitation Learning (IL) approaches have been deployed for obtaining optimal treatment for a patient but, these approaches rely only on positive trajectories (i.e., treatments that concluded with positive responses of the patient). In contrast, negative trajectories (i.e., samples of non-responding treatments) are discarded, although these have valuable information content. We propose a Cooperative Imitation Learning (CIL) method that exploits information from both negative and positive trajectories to learn the optimal DTR. The proposed method reduces the chance of selecting any treatment which results in a negative outcome (negative response of the patient) during the medical examination. To validate our approach, we have considered a well-known DTR which is defined for the treatment of patients with alcohol addiction. Results show that our approach outperforms those that rely only on positive trajectories.

Clinical Trials for Personalized Medicine: Design and Data Analysis

In this article, we provide an overview of statistical methods for design and analysis of clinical trials for personalized medicine. In a basic setting where marker-defined subgroups are well defined based on a baseline biomarker, enrichment, marker-stratified, and strategy-based clinical trial designs, as well as related data analysis methods, are discussed. Clinical trials with development and validation of predictive signatures or systems are also discussed. Lastly, we provide a brief overview for some recent topics, master protocol clinical trials, the development of dynamic treatment regimens and AI-based diagnostics, as active fields in which further contribution of biometrics is warranted.

Variable selection in regression-based estimation of dynamic treatment regimes.

Dynamic treatment regimes (DTRs) consist of a sequence of decision rules, one per stage of intervention, that aim to recommend effective treatments for individual patients according to patient information history. DTRs can be estimated from models which include interactions between treatment and a (typically small) number of covariates which are often chosen a priori. However, with increasingly large and complex data being collected, it can be difficult to know which prognostic factors might be relevant in the treatment rule. Therefore, a more data-driven approach to select these covariates might improve the estimated decision rules and simplify models to make them easier to interpret. We propose a variable selection method for DTR estimation using penalized dynamic weighted least squares. Our method has the strong heredity property, that is, an interaction term can be included in the model only if the corresponding main terms have also been selected. We show our method has both the double robustness property and the oracle property theoretically; and the newly proposed method compares favorably with other variable selection approaches in numerical studies. We further illustrate the proposed method on data from the Sequenced Treatment Alternatives to Relieve Depressionstudy.

Reinforcement Learning in Healthcare: A Survey

As a subfield of machine learning, reinforcement learning (RL) aims at optimizing decision making by using interaction samples of an agent with its environment and the potentially delayed feedbacks. In contrast to traditional supervised learning that typically relies on one-shot, exhaustive, and supervised reward signals, RL tackles sequential decision-making problems with sampled, evaluative, and delayed feedbacks simultaneously. Such a distinctive feature makes RL techniques a suitable candidate for developing powerful solutions in various healthcare domains, where diagnosing decisions or treatment regimes are usually characterized by a prolonged period with delayed feedbacks. By first briefly examining theoretical foundations and key methods in RL research, this survey provides an extensive overview of RL applications in a variety of healthcare domains, ranging from dynamic treatment regimes in chronic diseases and critical care, automated medical diagnosis, and many other control or scheduling problems that have infiltrated every aspect of the healthcare system. In addition, we discuss the challenges and open issues in the current research and highlight some potential solutions and directions for future research.

Survival Augmented Patient Preference Incorporated Reinforcement Learning to Evaluate Tailoring Variables for Personalized Healthcare

In this paper, we consider personalized treatment decision strategies in the management of chronic diseases, such as chronic kidney disease, which typically consists of sequential and adaptive treatment decision making. We investigate a two-stage treatment setting with a survival outcome that could be right censored. This can be formulated through a dynamic treatment regime (DTR) framework, where the goal is to tailor treatment to each individual based on their own medical history in order to maximize a desirable health outcome. We develop a new method, Survival Augmented Patient Preference incorporated reinforcement Q-Learning (SAPP-Q-Learning) to decide between quality of life and survival restricted at maximal follow-up. Our method incorporates the latent patient preference into a weighted utility function that balances between quality of life and survival time, in a Q-learning model framework. We further propose a corresponding m-out-of-n Bootstrap procedure to accurately make statistical inferences and construct confidence intervals on the effects of tailoring variables, whose values can guide personalized treatment strategies.

Step-adjusted tree-based reinforcement learning for evaluating nested dynamic treatment regimes using test-and-treat observational data.

Dynamic treatment regimes (DTRs) include a sequence of treatment decision rules, in which treatment is adapted over time in response to the changes in an individual's disease progression and health care history. In medical practice, nested test-and-treat strategies are common to improve cost-effectiveness. For example, for patients at risk of prostate cancer, only patients who have high prostate-specific antigen (PSA) need a biopsy, which is costly and invasive, to confirm the diagnosis and help determine the treatment if needed. A decision about treatment happens after the biopsy, and is thus nested within the decision of whether to do the test. However, current existing statistical methods are not able to accommodate such a naturally embedded property of the treatment decision within the test decision. Therefore, we developed a new statistical learning method, step-adjusted tree-based reinforcement learning, to evaluate DTRs within such a nested multistage dynamic decision framework using observational data. At each step within each stage, we combined the robust semiparametric estimation via augmented inverse probability weighting with a tree-based reinforcement learning method to deal with the counterfactual optimization. The simulation studies demonstrated robust performance of the proposed methods under different scenarios. We further applied our method to evaluate the necessity of prostate biopsy and identify the optimal test-and-treat regimes for prostate cancer patients using data from the Johns Hopkins University prostate cancer active surveillance dataset.

Restricted sub-tree learning to estimate an optimal dynamic treatment regime using observational data.

Dynamic treatment regimes (DTRs), consisting of a sequence of tailored treatment decision rules that span multiple stages of care, present a unique opportunity in our drive toward personalized medicine. Given that estimation of optimal DTRs is often exploratory and communication with clinicians is vital, robust and flexible methods that yield interpretable results are needed. Tree-based methods utilizing a purity measure defined on the full set of covariates have enjoyed much success in meeting this goal. Often, however, it is necessary for clinical, practical, or ethical reasons to restrict certain covariates that should be used when making treatment decisions. Herein we present restricted sub-tree learning (ReST-L), a flexible and robust, sub-tree-based method to estimate an optimal multi-stage multi-treatment DTR that enables restrictions to the set of prespecified candidate tailoring variables. ReST-L employs a purity measure derived from an augmented inverse probability weighted estimator for the counterfactual mean outcome, using observational data to build multi-stage decision trees that are restricted in sub-tree spaces defined by the corresponding prescriptive covariates. We show that ReST-L is able to correctly estimate the optimal DTR searching over a large number of variables with relatively small sample sizes and improves upon competing estimation methods. We demonstrate the utility of ReST-L to estimate a two-stage fluid resuscitation strategy for patients admitted to an intensive care unit with acute emergent sepsis.

Individualized resuscitation strategy for septic shock formalized by finite mixture modeling and dynamic treatment regimen

BackgroundSeptic shock comprises a heterogeneous population, and individualized resuscitation strategy is of vital importance. The study aimed to identify subclasses of septic shock with non-supervised learning algorithms, so as to tailor resuscitation strategy for each class.MethodsPatients with septic shock in 25 tertiary care teaching hospitals in China from January 2016 to December 2017 were enrolled in the study. Clinical and laboratory variables were collected on days 0, 1, 2, 3 and 7 after ICU admission. Subclasses of septic shock were identified by both finite mixture modeling and K-means clustering. Individualized fluid volume and norepinephrine dose were estimated using dynamic treatment regime (DTR) model to optimize the final mortality outcome. DTR models were validated in the eICU Collaborative Research Database (eICU-CRD) dataset.ResultsA total of 1437 patients with a mortality rate of 29% were included for analysis. The finite mixture modeling and K-means clustering robustly identified five classes of septic shock. Class 1 (baseline class) accounted for the majority of patients over all days; class 2 (critical class) had the highest severity of illness; class 3 (renal dysfunction) was characterized by renal dysfunction; class 4 (respiratory failure class) was characterized by respiratory failure; and class 5 (mild class) was characterized by the lowest mortality rate (21%). The optimal fluid infusion followed the resuscitation/de-resuscitation phases with initial large volume infusion and late restricted volume infusion. While class 1 transitioned to de-resuscitation phase on day 3, class 3 transitioned on day 1. Classes 1 and 3 might benefit from early use of norepinephrine, and class 2 can benefit from delayed use of norepinephrine while waiting for adequate fluid infusion.ConclusionsSeptic shock comprises a heterogeneous population that can be robustly classified into five phenotypes. These classes can be easily identified with routine clinical variables and can help to tailor resuscitation strategy in the context of precise medicine.

Adaptive randomization in a two-stage sequential multiple assignment randomized trial.

Sequential multiple assignment randomized trials (SMARTs) are systematic and efficient media for comparing dynamic treatment regimes (DTRs), where each patient is involved in multiple stages of treatment with the randomization at each stage depending on the patient's previous treatment history and interim outcomes. Generally, patients enrolled in SMARTs are randomized equally to ethically acceptable treatment options regardless of how effective those treatments were during the previous stages, which results in some undesirable consequences in practice, such as low recruitment, less retention, and lower treatment adherence. In this article, we propose a response-adaptive SMART (RA-SMART) design where the allocation probabilities are imbalanced in favor of more promising treatments based on the accumulated information on treatment efficacy from previous patients and stages. The operating characteristics of the RA-SMART design relative to SMART design, including the consistency and efficiency of estimated response rate under each DTR, the power of identifying the optimal DTR, and the number of patients treated with the optimal and the worst DTRs, are assessed through extensive simulation studies. Some practical suggestions are discussed in the conclusion.

Individualized Mechanical power-based ventilation strategy for acute respiratory failure formalized by finite mixture modeling and dynamic treatment regimen.

BackgroundMechanical ventilation (MV) is the key to the successful treatment of acute respiratory failure (ARF) in the intensive care unit (ICU). The study aims to formalize the concept of individualized MV strategy with finite mixture modeling (FMM) and dynamic treatment regime (DTR).MethodsARF patients requiring MV for over 48 h from 2008 to 2019 were included. FMM was conducted to identify classes of ARF. Static and dynamic mechanical power (MP_static and MP_dynamic) and relevant clinical variables were calculated/collected from hours 0 to 48 at an interval of 8 h. was calculated as the difference between actual and optimal MP.FindingsA total of 8768 patients were included for analysis with a mortality rate of 27%. FFM identified three classes of ARF, namely, the class 1 (baseline), class 2 (critical) and class 3 (refractory respiratory failure). The effect size of MP_static on mortality is the smallest in class 1 (HR for every 5 Joules/min increase: 1.29; 95% CI: 1.15 to 1.45; p < 0.001) and the largest in class 3 (HR for every 5 Joules/min increase: 1.83; 95% CI: 1.52 to 2.20; p < 0.001).InterpretationMP has differing therapeutic effects for subtypes of ARF. Optimal MP estimated by DTR model may help to improve survival outcome.FundingThe study was funded by Health Science and Technology Plan of Zhejiang Province (2021KY745), Key Research & Development project of Zhejiang Province (2021C03071) and Yilu "Gexin" - Fluid Therapy Research Fund Project (YLGX-ZZ-2,020,005).

Inverse reinforcement learning in contextual MDPs

We consider the task of Inverse Reinforcement Learning in Contextual Markov Decision Processes (MDPs). In this setting, contexts, which define the reward and transition kernel, are sampled from a distribution. In addition, although the reward is a function of the context, it is not provided to the agent. Instead, the agent observes demonstrations from an optimal policy. The goal is to learn the reward mapping, such that the agent will act optimally even when encountering previously unseen contexts, also known as zero-shot transfer. We formulate this problem as a non-differential convex optimization problem and propose a novel algorithm to compute its subgradients. Based on this scheme, we analyze several methods both theoretically, where we compare the sample complexity and scalability, and empirically. Most importantly, we show both theoretically and empirically that our algorithms perform zero-shot transfer (generalize to new and unseen contexts). Specifically, we present empirical experiments in a dynamic treatment regime, where the goal is to learn a reward function which explains the behavior of expert physicians based on recorded data of them treating patients diagnosed with sepsis.

Estimating the Effectiveness of Rotavirus Vaccine Schedules.

Important questions exist regarding the comparative effectiveness of alternative childhood vaccine schedules; however, optimal approaches to studying this complex issue are unclear. We applied methods for studying dynamic treatment regimens to estimate the comparative effectiveness of different rotavirus vaccine (RV) schedules for preventing acute gastroenteritis-related emergency department (ED) visits or hospitalization. We studied the effectiveness of six separate protocols: one- and two-dose monovalent rotavirus vaccine (RV1); one-, two-, and three-dose pentavalent rotavirus vaccine (RV5); and no RV vaccine. We used data on all infants to estimate the counterfactual cumulative risk for each protocol. Infants were censored when vaccine receipt deviated from the protocol. Inverse probability of censoring-weighted estimation addressed potentially informative censoring by protocol deviations. A nonparametric group-based bootstrap procedure provided statistical inference. The method yielded similar 2-year effectiveness estimates for the full-series protocols; weighted risk difference estimates comparing unvaccinated children to those adherent to either full-series (two-dose RV1, three-dose RV5) corresponded to four fewer hospitalizations and 12 fewer ED visits over the 2-year period per 1,000 children. We observed dose-response relationships, such that additional doses further reduced risk of acute gastroenteritis. Under a theoretical intervention to fully vaccinate all children, the 2-year risk differences comparing full to observed adherence were 0.04% (95% CI = 0.03%, 0.05%) for hospitalizations and 0.17% (95% CI = 0.14%, 0.19%) for ED visits. The proposed approach can generate important evidence about the consequences of delaying or skipping vaccine doses, and the impact of interventions to improve vaccine schedule adherence.

Efficient adjustment sets in causal graphical models with hidden variables

Summary We study the selection of adjustment sets for estimating the interventional mean under a point exposure dynamic treatment regime, that is, a treatment rule that depends on the subject’s covariates. We assume a nonparametric causal graphical model with, possibly, hidden variables and at least one adjustment set comprised of observable variables. We provide the definition of a valid adjustment set for a point exposure dynamic treatment regime, which generalizes the existing definition for a static intervention. We show that there exists an adjustment set, referred to as optimal minimal, that yields the nonparametric estimator of the interventional mean with the smallest asymptotic variance among those that are based on observable minimal adjustment sets. An observable minimal adjustment set is a valid adjustment set such that all its variables are observable and the removal of any of its variables destroys its validity. We provide similar optimality results for the class of observable minimum adjustment sets, that is, valid observable adjustment sets of minimum cardinality among the observable adjustment sets. Moreover, we show that if either no variables are hidden or if all the observable variables are ancestors of either treatment, outcome or the variables that are used to decide treatment, a globally optimal adjustment set exists. We provide polynomial-time algorithms to compute the globally optimal, optimal minimal and optimal minimum adjustment sets. Because static interventions can be viewed as a special case of dynamic regimes, all our results also apply for static interventions.

A scoping review of studies using observational data to optimise dynamic treatment regimens

BackgroundDynamic treatment regimens (DTRs) formalise the multi-stage and dynamic decision problems that clinicians often face when treating chronic or progressive medical conditions. Compared to randomised controlled trials, using observational data to optimise DTRs may allow a wider range of treatments to be evaluated at a lower cost. This review aimed to provide an overview of how DTRs are optimised with observational data in practice.MethodsUsing the PubMed database, a scoping review of studies in which DTRs were optimised using observational data was performed in October 2020. Data extracted from eligible articles included target medical condition, source and type of data, statistical methods, and translational relevance of the included studies.ResultsFrom 209 PubMed abstracts, 37 full-text articles were identified, and a further 26 were screened from the reference lists, totalling 63 articles for inclusion in a narrative data synthesis. Observational DTR models are a recent development and their application has been concentrated in a few medical areas, primarily HIV/AIDS (27, 43%), followed by cancer (8, 13%), and diabetes (6, 10%). There was substantial variation in the scope, intent, complexity, and quality between the included studies. Statistical methods that were used included inverse-probability weighting (26, 41%), the parametric G-formula (16, 25%), Q-learning (10, 16%), G-estimation (4, 6%), targeted maximum likelihood/minimum loss-based estimation (4, 6%), regret regression (3, 5%), and other less common approaches (10, 16%). Notably, studies that were primarily intended to address real-world clinical questions (18, 29%) tended to use inverse-probability weighting and the parametric G-formula, relatively well-established methods, along with a large amount of data. Studies focused on methodological developments (45, 71%) tended to be more complicated and included a demonstrative real-world application only.ConclusionsAs chronic and progressive conditions become more common, the need will grow for personalised treatments and methods to estimate the effects of DTRs. Observational DTR studies will be necessary, but so far their use to inform clinical practice has been limited. Focusing on simple DTRs, collecting large and rich clinical datasets, and fostering tight partnerships between content experts and data analysts may result in more clinically relevant observational DTR studies.

Bayesian data fusion: Probabilistic sensitivity analysis for unmeasured confounding using informative priors based on secondary data.

Bayesian causal inference offers a principled approach to policy evaluation of proposed interventions on mediators or time-varying exposures. Building on the Bayesian g-formula method introduced by Keil etal., we outline a general approach for the estimation of population-level causal quantities involving dynamic and stochastic treatment regimes, including regimes related to mediation estimands such as natural direct and indirect effects. We further extend this approach to propose a Bayesian data fusion (BDF), an algorithm for performing probabilistic sensitivity analysis when a confounder unmeasured in a primary data set is available in an external data source. When the relevant relationships are causally transportable between the two source populations, BDF corrects confounding bias and supports causal inference and decision-making within the main study population without sharing of the individual-level external data set. We present results from a simulation study comparing BDF to two common frequentist correction methods for unmeasured mediator-outcome confounding bias in the mediation setting. We use these methods to analyze data on the role of stage at cancer diagnosis in contributing to Black-White colorectal cancer survivaldisparities.

Multicategory Angle-Based Learning for Estimating Optimal Dynamic Treatment Regimes With Censored Data

An optimal dynamic treatment regime (DTR) consists of a sequence of decision rules in maximizing long-term benefits, which is applicable for chronic diseases such as HIV infection or cancer. In this article, we develop a novel angle-based approach to search the optimal DTR under a multicategory treatment framework for survival data. The proposed method targets to maximize the conditional survival function of patients following a DTR. In contrast to most existing approaches which are designed to maximize the expected survival time under a binary treatment framework, the proposed method solves the multicategory treatment problem given multiple stages for censored data. Specifically, the proposed method obtains the optimal DTR via integrating estimations of decision rules at multiple stages into a single multicategory classification algorithm without imposing additional constraints, which is also more computationally efficient and robust. In theory, we establish Fisher consistency and provide the risk bound for the proposed estimator under regularity conditions. Our numerical studies show that the proposed method outperforms competing methods in terms of maximizing the conditional survival probability. We apply the proposed method to two real datasets: Framingham heart study data and acquired immunodeficiency syndrome clinical data. Supplementary materials for this article are available online.

Ambiguous Dynamic Treatment Regimes: A Reinforcement Learning Approach

A main research goal in various studies is to use an observational data set and provide a new set of counterfactual guidelines that can yield causal improvements. Dynamic Treatment Regimes (DTRs) are widely studied to formalize this process. However, available methods in finding optimal DTRs often rely on assumptions that are violated in real-world applications (e.g., medical decision-making or public policy), especially when (a) the existence of unobserved confounders cannot be ignored, and (b) the unobserved confounders are time-varying (e.g., affected by previous actions). When such assumptions are violated, one often faces ambiguity regarding the underlying causal model that is needed to be assumed to obtain an optimal DTR. This ambiguity is inevitable, since the dynamics of unobserved confounders and their causal impact on the observed part of the data cannot be understood from the observed data. Motivated by a case study of finding superior treatment regimes for patients who underwent transplantation in our partner hospital and faced a medical condition known as New Onset Diabetes After Transplantation (NODAT), we extend DTRs to a new class termed Ambiguous Dynamic Treatment Regimes (ADTRs), in which the casual impact of treatment regimes is evaluated based on a "cloud" of potential causal models. We then connect ADTRs to Ambiguous Partially Observable Mark Decision Processes (APOMDPs) proposed by Saghafian (2018), and develop two Reinforcement Learning methods termed Direct Augmented V-Learning (DAV-Learning) and Safe Augmented V-Learning (SAV-Learning), which enable using the observed data to efficiently learn an optimal treatment regime. We establish theoretical results for these learning methods, including (weak) consistency and asymptotic normality. We further evaluate the performance of these learning methods both in our case study and in simulation experiments.

Parkinson’s Disease Classification and Medication Adherence Monitoring Using Smartphone-based Gait Assessment and Deep Reinforcement Learning Algorithm

For the diagnosis and classification of Parkinson’s patients, the unified scale of Parkinson’s patients (Unified Parkinson’s Disease Rating Scale – UPDRS) is used, which requires the patient to perform a series of tests among which the biomarkers of speech, the facial expression, the hand movement and walking analysis are considered, after which the doctor diagnoses the patient whether or not he has Parkinson’s disease according to the score obtained. The work proposes a system for monitoring patients with the use of cell phones and their automatic classification according to the data collected by them. The system starts from the budget that Parkinson’s patients have different abnormalities when walking if they do not follow the required medication. The cell phone collects the data passively while the patient has his cell phone in his pocket. After that, the data preprocessor helps to extract the walking cycles that this Parkinson-related biomarker contains. The algorithm proposed for classification and Medication Adherence Monitoring is the Deep Reinforcement Learning. With this work we demonstrate the feasibility of using cell phones to monitor the biomarker walking in Parkinson’s patients and the possibility of Passive Medication Adherence Monitoring and Dynamic Treatment Regimes.