Dynamic Treatment Regimes Research Articles

Proper Inference for Value Function in High-Dimensional Q-Learning for Dynamic Treatment Regimes

Proper Inference for Value Function in High-Dimensional Q-Learning for Dynamic Treatment Regimes

A smoothed Q‐learning algorithm for estimating optimal dynamic treatment regimes

AbstractIn this paper, we propose a smoothed Q‐learning algorithm for estimating optimal dynamic treatment regimes. In contrast to the Q‐learning algorithm in which nonregular inference is involved, we show that, under assumptions adopted in this paper, the proposed smoothed Q‐learning estimator is asymptotically normally distributed even when the Q‐learning estimator is not and its asymptotic variance can be consistently estimated. As a result, inference based on the smoothed Q‐learning estimator is standard. We derive the optimal smoothing parameter and propose a data‐driven method for estimating it. The finite sample properties of the smoothed Q‐learning estimator are studied and compared with several existing estimators including the Q‐learning estimator via an extensive simulation study. We illustrate the new method by analyzing data from the Clinical Antipsychotic Trials of Intervention Effectiveness–Alzheimer's Disease (CATIE‐AD) study.

Prospects and challenges for clinical decision support in the era of big data.

Recently, there has been burgeoning interest in developing more effective and robust clinical decision support systems (CDSSs) for oncology. This has been primarily driven by the demands for more personalized and precise medical practice in oncology in the era of so-called Big Data (BD); an era that promises to harness the power of large-scale data flow to revolutionize cancer treatment. This interest in BD analytics has created new opportunities as well as new unmet challenges. These include: routine aggregation and standardization of clinical data; patient privacy; transformation of current analytical approaches to handle such noisy and heterogeneous data; and expanded use of advanced statistical learning methods based on confluence of modern statistical methods and machine learning algorithms. In this review, we present the current status of CDSSs in oncology, the prospects and current challenges of BD analytics, and the promising role of integrated modern statistics and machine learning algorithms in predicting complex clinical endpoints, individualizing treatment rules, and optimizing dynamic personalized treatment regimens. We discuss issues pertaining to these topics and present application examples from an aggregate of experiences. We also discuss the role of human factors in improving the utilization and acceptance of such enhanced CDSSs and how to mitigate possible sources of human error to achieve optimal performance and wider acceptance.

Power analysis in a SMART design: sample size estimation for determining the best embedded dynamic treatment regime.

Sequential, multiple assignment, randomized trial (SMART) designs have become increasingly popular in the field of precision medicine by providing a means for comparing more than two sequences of treatments tailored to the individual patient, i.e., dynamic treatment regime (DTR). The construction of evidence-based DTRs promises a replacement to ad hoc one-size-fits-all decisions pervasive in patient care. However, there are substantial statistical challenges in sizing SMART designs due to the correlation structure between the DTRs embedded in the design (EDTR). Since a primary goal of SMARTs is the construction of an optimal EDTR, investigators are interested in sizing SMARTs based on the ability to screen out EDTRs inferior to the optimal EDTR by a given amount which cannot be done using existing methods. In this article, we fill this gap by developing a rigorous power analysis framework that leverages the multiple comparisons with the best methodology. Our method employs Monte Carlo simulation to compute the number of individuals to enroll in an arbitrary SMART. We evaluate our method through extensive simulation studies. We illustrate our method by retrospectively computing the power in the Extending Treatment Effectiveness of Naltrexone (EXTEND) trial. An R package implementing our methodology is available to download from the Comprehensive R Archive Network.

Proper Inference for Value Function in High-Dimensional Q-Learning for Dynamic Treatment Regimes

ABSTRACTDynamic treatment regimes are a set of decision rules and each treatment decision is tailored over time according to patients’ responses to previous treatments as well as covariate history. There is a growing interest in development of correct statistical inference for optimal dynamic treatment regimes to handle the challenges of nonregularity problems in the presence of nonrespondents who have zero-treatment effects, especially when the dimension of the tailoring variables is high. In this article, we propose a high-dimensional Q-learning (HQ-learning) to facilitate the inference of optimal values and parameters. The proposed method allows us to simultaneously estimate the optimal dynamic treatment regimes and select the important variables that truly contribute to the individual reward. At the same time, hard thresholding is introduced in the method to eliminate the effects of the nonrespondents. The asymptotic properties for the parameter estimators as well as the estimated optimal value function are then established by adjusting the bias due to thresholding. Both simulation studies and real data analysis demonstrate satisfactory performance for obtaining the proper inference for the value function for the optimal dynamic treatment regimes. Supplementary materials for this article are available online.

Cumulative incidence regression for dynamic treatment regimens.

Recently dynamic treatment regimens (DTRs) have drawn considerable attention, as an effective tool for personalizing medicine. Sequential Multiple Assignment Randomized Trials (SMARTs) are often used to gather data for making inference on DTRs. In this article, we focus on regression analysis of DTRs from a two-stage SMART for competing risk outcomes based on cumulative incidence functions (CIFs). Even though there are extensive works on the regression problem for DTRs, no research has been done on modeling the CIF for SMART trials. We extend existing CIF regression models to handle covariate effects for DTRs. Asymptotic properties are established for our proposed estimators. The models can be implemented using existing software by an augmented-data approximation. We show the improvement provided by our proposed methods by simulation and illustrate its practical utility through an analysis of a SMART neuroblastoma study, where disease progression cannot be observed after death.

Interpretable Dynamic Treatment Regimes

ABSTRACTPrecision medicine is currently a topic of great interest in clinical and intervention science. A key component of precision medicine is that it is evidence-based, that is, data-driven, and consequently there has been tremendous interest in estimation of precision medicine strategies using observational or randomized study data. One way to formalize precision medicine is through a treatment regime, which is a sequence of decision rules, one per stage of clinical intervention, that map up-to-date patient information to a recommended treatment. An optimal treatment regime is defined as maximizing the mean of some cumulative clinical outcome if applied to a population of interest. It is well-known that even under simple generative models an optimal treatment regime can be a highly nonlinear function of patient information. Consequently, a focal point of recent methodological research has been the development of flexible models for estimating optimal treatment regimes. However, in many settings, estimation of an optimal treatment regime is an exploratory analysis intended to generate new hypotheses for subsequent research and not to directly dictate treatment to new patients. In such settings, an estimated treatment regime that is interpretable in a domain context may be of greater value than an unintelligible treatment regime built using “black-box” estimation methods. We propose an estimator of an optimal treatment regime composed of a sequence of decision rules, each expressible as a list of “if-then” statements that can be presented as either a paragraph or as a simple flowchart that is immediately interpretable to domain experts. The discreteness of these lists precludes smooth, that is, gradient-based, methods of estimation and leads to nonstandard asymptotics. Nevertheless, we provide a computationally efficient estimation algorithm, prove consistency of the proposed estimator, and derive rates of convergence. We illustrate the proposed methods using a series of simulation examples and application to data from a sequential clinical trial on bipolar disorder. Supplementary materials for this article are available online.

A joint longitudinal and survival model for dynamic treatment regimes in Presence of Competing Risk Analysis

BackgroundIn oncology setup, the personalized medicine with the Dynamic Treatment Regime(DTR) is the attractive tool for treatment management. It is based on sequence rule by changing the required treatment by looking into patients dynamic condition repeatedly. This repeatedly measured condition generates the longitudinal data and the time-to-event data jointly. The time-to-event data is generated with competing risks. Now handling the joint longitudinal and survival data with competing risks is itself challenging. It becomes more challenging to decide the best effective treatment strategy while we work with DTR approach in presences of competing risks through joint longitudinal and survival model. MethodsThis article is dedicated towards development of statistical methodology to handle competing risk analysis for DTR in repeatedly measured survival data. In this study a simulated dataset is used to resemble the observed data distribution seen in a motivating cancer trial data. ResultsThis algorithm is prepared through Bayesian analysis to obtain the best effective therapeutic regimen. The OpenBUGS function is prepared, which provides the therapeutic effect comparison and there after estimations between different treatment sequences. ConclusionThis developed method is easy to handle for personalized medicine context in oncology for supportive decision rule.

Constructing dynamic treatment regimes over indefinite time horizons

Existing methods for estimating optimal dynamic treatment regimes are limited to cases where a utility function is optimized over a fixed time period. We develop an estimation procedure for the optimal dynamic treatment regime over an indefinite time period and derive associated large-sample results. The proposed method can be used to estimate the optimal dynamic treatment regime in chronic disease settings. We illustrate this by simulating a dataset corresponding to a cohort of patients with diabetes that mimics the third wave of the National Health and Nutrition Examination Survey, and examining the performance of the proposed method in controlling the level of haemoglobin A1c.

Variants of double robust estimators for two-stage dynamic treatment regimes

Certain conditions and illnesses may necessitate multiple stages of treatment and thus require unique study designs to compare the efficacy of these interventions. Such studies are characterized by two or more stages of treatment punctuated by decision points where intermediate outcomes inform the choice for the next stage of treatment. The algorithm that dictates what treatments to take based on intermediate outcomes is referred to as a dynamic regime. This paper describes an efficient method of building double robust estimators of the treatment effect of different regimes. A double robust estimator utilizes both modeling of the outcome and weighting based on the modeled probability of receiving treatment in such a way that the resulting estimator is a consistent estimate of the desired population parameter under the condition that at least one of those models is correct. This new and more efficient double robust estimator is compared to another double robust estimator as well as classical regression and inverse probability weighted estimators. The methods are applied to estimate the regime effects in the STAR*D anti-depression treatment trial.

TREE-BASED REINFORCEMENT LEARNING FOR ESTIMATING OPTIMAL DYNAMIC TREATMENT REGIMES.

Dynamic treatment regimes (DTRs) are sequences of treatment decision rules, in which treatment may be adapted over time in response to the changing course of an individual. Motivated by the substance use disorder (SUD) study, we propose a tree-based reinforcement learning (T-RL) method to directly estimate optimal DTRs in a multi-stage multi-treatment setting. At each stage, T-RL builds an unsupervised decision tree that directly handles the problem of optimization with multiple treatment comparisons, through a purity measure constructed with augmented inverse probability weighted estimators. For the multiple stages, the algorithm is implemented recursively using backward induction. By combining semiparametric regression with flexible tree-based learning, T-RL is robust, efficient and easy to interpret for the identification of optimal DTRs, as shown in the simulation studies. With the proposed method, we identify dynamic SUD treatment regimes for adolescents.

Optical metamaterials for decontamination of translucent liquids and gases

Metamaterials with periodical structures like quartz or glass small granules irradiated by UV-C to act against microbial contamination of translucent liquids and gases are studied. Investigations of the modifications of individual metamaterial elements when UV evanescent waves are dispersed in the optical contact zone, as a function of granule geometry, were performed. Different situations were investigated, when quartz (SiO2), glass, or black (plastic) materials with dimensions of about 0.5–3 mm are separately placed into a quartz tube of about 2.7 cm diameter and 90 cm length, named ‘core tube’. Quartz granules transmit within the 240–400 nm region of the Hg lamp and ensure an effective decontamination of translucent liquids and gases. Our approach is based upon the increased transfer of UV radiation via evanescent waves in case of unordered metamaterials present in contaminated fluids. We made a series of estimations of the decontamination rate of this type of metamaterials versus ordered metamaterials consisting of spherical elements. Experiments have convincingly demonstrated that both quartz and glass metamaterials can effectively annihilate Coliform (including Escherichia coli) or Enterococcus bacteria, as well as yeast and Kombucha cultures. The decontamination efficiency was assessed both in dynamic and static treatment regimes. Control experiments were performed in the absence of metamaterials and/or UV-C irradiation.

Estimating the causal effect of treatment regimes for organ transplantation.

Patients awaiting cadaveric organ transplantation face a difficult decision if offered a low-quality organ: accept the organ or remain on the waiting list and hope a better organ is offered in the future. A dynamic treatment regime (DTR) for transplantation is a rule that determines whether a patient should decline an offered organ. Existing methods can estimate the effect of DTRs on survival outcomes, but these were developed for applications where treatment is abundantly available. For transplantation, organ availability is limited, and existing methods can only estimate the effect of a DTR assuming a single patient follows the DTR. We show for transplantation that the effect of a DTR depends on whether other patients follow the DTR. To estimate the anticipated survival if the entire population awaiting transplantation were to adopt a DTR, we develop a novel inverse probability weighted estimator (IPCW) which re-weights patients based on the probability of following their transplant history in the counterfactual world in which all patients follow the DTR of interest. We estimate this counterfactual probability using hot deck imputation to fill in data that is not observed for patients who are artificially censored by IPCW once they no longer follow the DTR of interest. We show via simulation that our proposed method has good finite-sample properties, and we apply our method to a lung transplantation observational registry.

A Bayesian Machine Learning Approach for Optimizing Dynamic Treatment Regimes

ABSTRACTMedical therapy often consists of multiple stages, with a treatment chosen by the physician at each stage based on the patient’s history of treatments and clinical outcomes. These decisions can be formalized as a dynamic treatment regime. This article describes a new approach for optimizing dynamic treatment regimes, which bridges the gap between Bayesian inference and existing approaches, like Q-learning. The proposed approach fits a series of Bayesian regression models, one for each stage, in reverse sequential order. Each model uses as a response variable the remaining payoff assuming optimal actions are taken at subsequent stages, and as covariates the current history and relevant actions at that stage. The key difficulty is that the optimal decision rules at subsequent stages are unknown, and even if these decision rules were known the relevant response variables may be counterfactual. However, posterior distributions can be derived from the previously fitted regression models for the optimal decision rules and the counterfactual response variables under a particular set of rules. The proposed approach averages over these posterior distributions when fitting each regression model. An efficient sampling algorithm for estimation is presented, along with simulation studies that compare the proposed approach with Q-learning. Supplementary materials for this article are available online.

Quantile-Optimal Treatment Regimes

ABSTRACTFinding the optimal treatment regime (or a series of sequential treatment regimes) based on individual characteristics has important applications in areas such as precision medicine, government policies, and active labor market interventions. In the current literature, the optimal treatment regime is usually defined as the one that maximizes the average benefit in the potential population. This article studies a general framework for estimating the quantile-optimal treatment regime, which is of importance in many real-world applications. Given a collection of treatment regimes, we consider robust estimation of the quantile-optimal treatment regime, which does not require the analyst to specify an outcome regression model. We propose an alternative formulation of the estimator as a solution of an optimization problem with an estimated nuisance parameter. This novel representation allows us to investigate the asymptotic theory of the estimated optimal treatment regime using empirical process techniques. We derive theory involving a nonstandard convergence rate and a nonnormal limiting distribution. The same nonstandard convergence rate would also occur if the mean optimality criterion is applied, but this has not been studied. Thus, our results fill an important theoretical gap for a general class of policy search methods in the literature. The article investigates both static and dynamic treatment regimes. In addition, doubly robust estimation and alternative optimality criterion such as that based on Gini’s mean difference or weighted quantiles are investigated. Numerical simulations demonstrate the performance of the proposed estimator. A data example from a trial in HIV+ patients is used to illustrate the application. Supplementary materials for this article are available online.

Augmented outcome-weighted learning for estimating optimal dynamic treatment regimens.

Dynamic treatment regimens (DTRs) are sequential treatment decisions tailored by patient's evolving features and intermediate outcomes at each treatment stage. Patient heterogeneity and the complexity and chronicity of many diseases call for learning optimal DTRs that can best tailor treatment according to each individual's time-varying characteristics (eg, intermediate response over time). In this paper, we propose a robust and efficient approach referred to as Augmented Outcome-weighted Learning (AOL) to identify optimal DTRs from sequential multiple assignment randomized trials. We improve previously proposed outcome-weighted learning to allow for negative weights. Furthermore, to reduce the variability of weights for numeric stability and improve estimation accuracy, in AOL, we propose a robust augmentation to the weights by making use of predicted pseudooutcomes from regression models for Q-functions. We show that AOL still yields Fisher-consistent DTRs even if the regression models are misspecified and that an appropriate choice of the augmentation guarantees smaller stochastic errors in value function estimation for AOL than the previous outcome-weighted learning. Finally, we establish the convergence rates for AOL. The comparative advantage of AOL over existing methods is demonstrated through extensive simulation studies and an application to a sequential multiple assignment randomized trial for major depressive disorder.

Reward ignorant modeling of dynamic treatment regimes.

Personalized medicine optimizes patient outcome by tailoring treatments to patient-level characteristics. This approach is formalized by dynamic treatment regimes (DTRs): decision rules that take patient information as input and output recommended treatment decisions. The DTR literature has seen the development of increasingly sophisticated causal inference techniques that attempt to address the limitations of our typically observational datasets. Often overlooked, however, is that in practice most patients may be expected to receive optimal or near-optimal treatment, and so the outcome used as part of a typical DTR analysis may provide limited information. In light of this, we propose considering a more standard analysis: ignore the outcome and elicit an optimal DTR by modeling the observed treatment as a function of relevant covariates. This offers a far simpler analysis and, in some settings, improved optimal treatment identification. To distinguish this approach from more traditional DTR analyses, we term it reward ignorant modeling, and also introduce the concept of multimethod analysis, whereby different analysis methods are used in settings with multiple treatment decisions. We demonstrate this concept through a variety of simulation studies, and through analysis of data from the International Warfarin Pharmacogenetics Consortium, which also serve as motivation for this work.

Optimal two-stage dynamic treatment regimes from a classification perspective with censored survival data.

Clinicians often make multiple treatment decisions at key points over the course of a patient's disease. A dynamic treatment regime is a sequence of decision rules, each mapping a patient's observed history to the set of available, feasible treatment options at each decision point, and thus formalizes this process. An optimal regime is one leading to the most beneficial outcome on average if used to select treatment for the patient population. We propose a method for estimation of an optimal regime involving two decision points when the outcome of interest is a censored survival time, which is based on maximizing a locally efficient, doubly robust, augmented inverse probability weighted estimator for average outcome over a class of regimes. By casting this optimization as a classification problem, we exploit well-studied classification techniques such as support vector machines to characterize the class of regimes and facilitate implementation via a backward iterative algorithm. Simulation studies of performance and application of the method to data from a sequential, multiple assignment randomized clinical trial in acute leukemia are presented.

HIGH-DIMENSIONAL A-LEARNING FOR OPTIMAL DYNAMIC TREATMENT REGIMES.

Precision medicine is a medical paradigm that focuses on finding the most effective treatment decision based on individual patient information. For many complex diseases, such as cancer, treatment decisions need to be tailored over time according to patients' responses to previous treatments. Such an adaptive strategy is referred as a dynamic treatment regime. A major challenge in deriving an optimal dynamic treatment regime arises when an extraordinary large number of prognostic factors, such as patient's genetic information, demographic characteristics, medical history and clinical measurements over time are available, but not all of them are necessary for making treatment decision. This makes variable selection an emerging need in precision medicine. In this paper, we propose a penalized multi-stage A-learning for deriving the optimal dynamic treatment regime when the number of covariates is of the non-polynomial (NP) order of the sample size. To preserve the double robustness property of the A-learning method, we adopt the Dantzig selector which directly penalizes the A-leaning estimating equations. Oracle inequalities of the proposed estimators for the parameters in the optimal dynamic treatment regime and error bounds on the difference between the value functions of the estimated optimal dynamic treatment regime and the true optimal dynamic treatment regime are established. Empirical performance of the proposed approach is evaluated by simulations and illustrated with an application to data from the STAR*D study.

PRM17 - Assessing Type and Impact of Biases Potentially Occurring when Analyzing Real World Evidence: The Case of Second Line Treatment for Ovarian Cancer

Real world evidence (RWE) pose additional methodological challenges for evaluating causality, including confounding, missing/misclassified data, no clear treatment assignment, dynamic treatment regimens, and switching. We aimed to assess the type and impact of biases potentially occurring when analyzing RWE using the case of ovarian cancer treatment. We compared overall survival (OS) with and without second-line treatment in ovarian cancer patients (n=1581) using retrospective IMS Oncology electronic medical records. We identified potential confounding and other biases using directed acyclic graphs (DAG). To assess the biases, we applied several analytic approaches starting with simple Cox regression with and without baseline variables, including time-dependent covariates to reduce immortal time bias, applying the “target trial” approach, including pseudo-populations and marginal structural models with inverse probability of censoring weighting (IPCW) (causal). We compared hazard ratios (HR) and 95% confidence intervals (95%CI) to assess the bias associated with each of these approaches compared to the causal analysis. The crude and baseline-adjusted analyses yielded a HR for second-line versus no second-line therapy of 0.565 (95%CI 0.495-0.645) and 0.535 (95%CI 0.468-0.613), respectively. Including treatment as a time-dependent covariate to account for immortal time bias, the corresponding crude and adjusted HR increased to 1.665 (95%CI 1.459-1.901) and 1.683 (95%CI 1.407-2.014). Applying a causal (counterfactual) analysis using IPCW and replication yielded a HR of 1.067 (95%CI 1.020-1.115), which matched the results of a published randomized clinical trial. When using routine RWE data, DAGs can guide the identification of potential biases and variables that need to be controlled for. In our analysis, potential biases were substantial with different directions. Only the application of the target trial and replication approach in combination with a causal analysis matched data from clinical trials.