Dynamic Treatment Regimes Research Articles

Who will benefit from antidepressants in the acute treatment of bipolar depression? A reanalysis of the STEP-BD study by Sachs et al. 2007, using Q-learning.

BackgroundThere is substantial uncertainty regarding the efficacy of antidepressants in the treatment of bipolar disorders.MethodsTraditional randomized controlled trials and statistical methods are not designed to discover if, when, and to whom an intervention should be applied; thus, other methodological approaches are needed that allow for the practice of personalized, evidence-based medicine with patients with bipolar depression.ResultsDynamic treatment regimes operationalize clinical decision-making as a sequence of decision rules, one per stage of clinical intervention, that map patient information to a recommended treatment. Using data from the acute depression randomized care (RAD) pathway of the Systematic Treatment Enhancement Program for Bipolar Disorder (STEP-BD) study, we estimate an optimal dynamic treatment regime via Q-learning.ConclusionsThe estimated optimal treatment regime presents some evidence that patients in the RAD pathway of STEP-BD who experienced a (hypo)manic episode before the depressive episode may do better to forgo adding an antidepressant to a mandatory mood stabilizer.Electronic supplementary materialThe online version of this article (doi:10.1186/s40345-014-0018-5) contains supplementary material, which is available to authorized users.

Penalized Q-Learning for Dynamic Treatment Regimens.

A dynamic treatment regimen incorporates both accrued information and long-term effects of treatment from specially designed clinical trials. As these trials become more and more popular in conjunction with longitudinal data from clinical studies, the development of statistical inference for optimal dynamic treatment regimens is a high priority. In this paper, we propose a new machine learning framework called penalized Q-learning, under which valid statistical inference is established. We also propose a new statistical procedure: individual selection and corresponding methods for incorporating individual selection within penalized Q-learning. Extensive numerical studies are presented which compare the proposed methods with existing methods, under a variety of scenarios, and demonstrate that the proposed approach is both inferentially and computationally superior. It is illustrated with a depression clinical trial study.

IqLearn: Interactive Q-Learning in R.

Chronic illness treatment strategies must adapt to the evolving health status of the patient receiving treatment. Data-driven dynamic treatment regimes can offer guidance for clinicians and intervention scientists on how to treat patients over time in order to bring about the most favorable clinical outcome on average. Methods for estimating optimal dynamic treatment regimes, such as Q-learning, typically require modeling nonsmooth, nonmonotone transformations of data. Thus, building well-fitting models can be challenging and in some cases may result in a poor estimate of the optimal treatment regime. Interactive Q-learning (IQ-learning) is an alternative to Q-learning that only requires modeling smooth, monotone transformations of the data. The R package iqLearn provides functions for implementing both the IQ-learning and Q-learning algorithms. We demonstrate how to estimate a two-stage optimal treatment policy with iqLearn using a generated data set bmiData which mimics a two-stage randomized body mass index reduction trial with binary treatments at each stage.

New Statistical Learning Methods for Estimating Optimal Dynamic Treatment Regimes

New Statistical Learning Methods for Estimating Optimal Dynamic Treatment Regimes

Penalized Q-learning for dynamic treatment regimens

Penalized Q-learning for dynamic treatment regimens

Use of personalized Dynamic Treatment Regimes (DTRs) and Sequential Multiple Assignment Randomized Trials (SMARTs) in mental health studies.

SummaryDynamic treatment regimens (DTRs) are sequential decision rules tailored at each point where a clinical decision is made based on each patient’s time-varying characteristics and intermediate outcomes observed at earlier points in time. The complexity, patient heterogeneity, and chronicity of mental disorders call for learning optimal DTRs to dynamically adapt treatment to an individual’s response over time. The Sequential Multiple Assignment Randomized Trial (SMARTs) design allows for estimating causal effects of DTRs. Modern statistical tools have been developed to optimize DTRs based on personalized variables and intermediate outcomes using rich data collected from SMARTs; these statistical methods can also be used to recommend tailoring variables for designing future SMART studies. This paper introduces DTRs and SMARTs using two examples in mental health studies, discusses two machine learning methods for estimating optimal DTR from SMARTs data, and demonstrates the performance of the statistical methods using simulated data.

Q- and A-learning Methods for Estimating Optimal Dynamic Treatment Regimes.

In clinical practice, physicians make a series of treatment decisions over the course of a patient's disease based on his/her baseline and evolving characteristics. A dynamic treatment regime is a set of sequential decision rules that operationalizes this process. Each rule corresponds to a decision point and dictates the next treatment action based on the accrued information. Using existing data, a key goal is estimating the optimal regime, that, if followed by the patient population, would yield the most favorable outcome on average. Q- and A-learning are two main approaches for this purpose. We provide a detailed account of these methods, study their performance, and illustrate them using data from a depression study.

Response to Reader Reaction

Response to Reader Reaction

Introduction to the special issue: The Tenth International Conference on Health Policy

This special issue of Health Services and Outcomes Research Methodology contains papers, based on presentations at the tenth International Conference on Health Policy Statistics (ICHPS) held October 5–7, 2013 at The Palmer House Hilton Hotel in Chicago. This special issue contains 6 high quality papers, which reflects the very high standard of research presented at the conference. Over 260 statisticians, methodologists, and health policy experts attended the tenth ICHPS, which was co-chaired by Xiao-Hua Andrew Zhou of the University of Washington and Donald Hedeker of the University of Illinois at Chicago. The meeting’s theme, ‘‘Statistically Engaged Health Care Policy,’’ reflected the growing importance of the collaboration and engagement of statisticians in evaluating and improving health care policy in the US and throughout the world. Major presentations were from a range of topics, including comparative effectiveness research, causal inference, longitudinal data, missing data, propensity score methods, data synthesis, personalized medicine and dynamic treatment regimes, and methods for patientreported outcomes. Keynote addresses were given by Seth Eisen, Office of Research and Development of the Department of Veteran Affairs, on ‘‘The Reincarnation of Health Policy Statisticians,’’ and Sharon-Lise Normand, Department of Health Care Policy at the Harvard Medical School, on ‘‘Statistics and Health Policy in 2013—Progresses and Retreats.’’ Also, the Plenary address was delivered by Robert Gibbons, Departments of Medicine and Health

Sequential multiple assignment randomized trial (SMART) with adaptive randomization for quality improvement in depression treatment program.

Implementation study is an important tool for deploying state-of-the-art treatments from clinical efficacy studies into a treatment program, with the dual goals of learning about effectiveness of the treatments and improving the quality of care for patients enrolled into the program. In this article, we deal with the design of a treatment program of dynamic treatment regimens (DTRs) for patients with depression post-acute coronary syndrome. We introduce a novel adaptive randomization scheme for a sequential multiple assignment randomized trial of DTRs. Our approach adapts the randomization probabilities to favor treatment sequences having comparatively superior Q-functions used in Q-learning. The proposed approach addresses three main concerns of an implementation study: it allows incorporation of historical data or opinions, it includes randomization for learning purposes, and it aims to improve care via adaptation throughout the program. We demonstrate how to apply our method to design a depression treatment program using data from a previous study. By simulation, we illustrate that the inputs from historical data are important for the program performance measured by the expected outcomes of the enrollees, but also show that the adaptive randomization scheme is able to compensate poorly specified historical inputs by improving patient outcomes within a reasonable horizon. The simulation results also confirm that the proposed design allows efficient learning of the treatments by alleviating the curse of dimensionality.

Proceedings of the University of Pennsylvania 6th annual conference on statistical issues in clinical trials: Dynamic treatment regimes.

This issue of Clinical Trials contains the proceedings of the sixth in a series of symposia held at the University of Pennsylvania on current statistical issues in clinical trials. This symposium, held in spring 2013, focused on dynamic treatment regimes, a topic highly relevant to the ever-increasing concerns about evaluation of treatment strategies in chronic diseases.

University of Pennsylvania 6th annual conference on statistical issues in clinical trials: Dynamic treatment regimes (afternoon session).

University of Pennsylvania 6th annual conference on statistical issues in clinical trials: Dynamic treatment regimes (afternoon session).

University of Pennsylvania 6th annual conference on statistical issues in clinical trials: Dynamic treatment regimes (morning session).

University of Pennsylvania 6th annual conference on statistical issues in clinical trials: Dynamic treatment regimes (morning session).

Pattern-mixture-type Estimation and Testing of Neuroblastoma Treatment Regimes.

Sequentially randomized designs are commonly used in biomedical research, particularly in clinical trials, to assess and compare the effects of different treatment regimes. In such designs, eligible patients are first randomized to one of the initial therapies, then patients with some intermediate response (e.g. without progressive diseases) are randomized to one of the maintenance therapies. The goal is to evaluate dynamic treatment regimes consisting of an initial therapy, the intermediate response, and a maintenance therapy. In this article, we demonstrate the use of pattern-mixture model (commonly used for analyzing missing data) for estimating the effects of treatment regimes based on familiar survival analysis techniques such as Nelson-Aalen and parametric models. Moreover, we demonstrate how to use estimates from pattern-mixture models to test for the differences across treatment regimes in a weighted log-rank setting. We investigate the properties of the proposed estimators and test in a Monte Carlo simulation study. Finally we demonstrate the methods using the long-term survival data from the high risk neuroblastoma study.

Inference about the expected performance of a data-driven dynamic treatment regime.

A dynamic treatment regime (DTR) comprises a sequence of decision rules, one per stage of intervention, that recommends how to individualize treatment to patients based on evolving treatment and covariate history. These regimes are useful for managing chronic disorders, and fit into the larger paradigm of personalized medicine. The Value of a DTR is the expected outcome when the DTR is used to assign treatments to a population of interest. The Value of a data-driven DTR, estimated using data from a Sequential Multiple Assignment Randomized Trial, is both a data-dependent parameter and a non-smooth function of the underlying generative distribution. These features introduce additional variability that is not accounted for by standard methods for conducting statistical inference, for example, the bootstrap or normal approximations, if applied without adjustment. Our purpose is to provide a feasible method for constructing valid confidence intervals (CIs) for this quantity of practical interest. We propose a conceptually simple and computationally feasible method for constructing valid CIs for the Value of an estimated DTR based on subsampling. The method is self-tuning by virtue of an approach called the double bootstrap. We demonstrate the proposed method using a series of simulated experiments. The proposed method offers considerable improvement in terms of coverage rates of the CIs over the standard bootstrap approach. In this article, we have restricted our attention to Q-learning for estimating the optimal DTR. However, other methods can be employed for this purpose; to keep the discussion focused, we have not explored these alternatives. Subsampling-based CIs provide much better performance compared to standard bootstrap for the Value of an estimated DTR.

Optimization of behavioral dynamic treatment regimens based on the sequential, multiple assignment, randomized trial (SMART).

A behavioral intervention is a program aimed at modifying behavior for the purpose of treating or preventing disease, promoting health, and/or enhancing well-being. Many behavioral interventions are dynamic treatment regimens, that is, sequential, individualized multicomponent interventions in which the intensity and/or type of treatment is varied in response to the needs and progress of the individual participant. The multiphase optimization strategy (MOST) is a comprehensive framework for development, optimization, and evaluation of behavioral interventions, including dynamic treatment regimens. The objective of optimization is to make dynamic treatment regimens more effective, efficient, scalable, and sustainable. An important tool for optimization of dynamic treatment regimens is the sequential, multiple assignment, randomized trial (SMART). The purpose of this article is to discuss how to develop optimized dynamic treatment regimens within the MOST framework. The article discusses the preparation, optimization, and evaluation phases of MOST. It is shown how MOST can be used to develop a dynamic treatment regimen to meet a prespecified optimization criterion. The SMART is an efficient experimental design for gathering the information needed to optimize a dynamic treatment regimen within MOST. One signature feature of the SMART is that randomization takes place at more than one point in time. MOST and SMART can be used to develop optimized dynamic treatment regimens that will have a greater public health impact.

Estimation of optimal dynamic treatment regimes.

Recent advances in medical research suggest that the optimal treatment rules should be adaptive to patients over time. This has led to an increasing interest in studying dynamic treatment regime, a sequence of individualized treatment rules, one per stage of clinical intervention, which maps present patient information to a recommended treatment. There has been a recent surge of statistical work for estimating optimal dynamic treatment regimes from randomized and observational studies. The purpose of this article is to review recent methodological progress and applied issues associated with estimating optimal dynamic treatment regimes. We discuss sequential multiple assignment randomized trials, a clinical trial design used to study treatment sequences. We use a common estimator of an optimal dynamic treatment regime that applies to sequential multiple assignment randomized trials data as a platform to discuss several practical and methodological issues. We provide a limited survey of practical issues associated with modeling sequential multiple assignment randomized trials data. We review some existing estimators of optimal dynamic treatment regimes and discuss practical issues associated with these methods including model building, missing data, statistical inference, and choosing an outcome when only non-responders are re-randomized. We mainly focus on the estimation and inference of dynamic treatment regimes using sequential multiple assignment randomized trials data. Dynamic treatment regimes can also be constructed from observational data, which may be easier to obtain in practice; however, care must be taken to account for potential confounding.

SMART designs in cancer research: Past, present, and future.

Cancer affects millions of people worldwide each year. Patients require sequences of treatment based on their response to previous treatments to combat cancer and fight metastases. Physicians provide treatment based on clinical characteristics, changing over time. Guidelines for these individualized sequences of treatments are known as dynamic treatment regimens (DTRs) where the initial treatment and subsequent modifications depend on the response to previous treatments, disease progression, and other patient characteristics or behaviors. To provide evidence-based DTRs, the Sequential Multiple Assignment Randomized Trial (SMART) has emerged over the past few decades. To examine and learn from past SMARTs investigating cancer treatment options, to discuss potential limitations preventing the widespread use of SMARTs in cancer research, and to describe courses of action to increase the implementation of SMARTs and collaboration between statisticians and clinicians. There have been SMARTs investigating treatment questions in areas of cancer, but the novelty and perceived complexity has limited its use. By building bridges between statisticians and clinicians, clarifying research objectives, and furthering methods work, there should be an increase in SMARTs addressing relevant cancer treatment questions. Within any area of cancer, SMARTs develop DTRs that can guide treatment decisions over the disease history and improve patient outcomes.

Personalizing medicine: a review of adaptive treatment strategies

Much of current pharmacological practice focuses on identifying the single 'best' treatment (or course of treatments) for a particular disease. Recently, however, focus has begun to shift towards a more patient-centric rather than disease-centric approach, where personal characteristics are used to identify the optimal treatment for an individual. Adaptive treatment strategies (also known as dynamic treatment regimes) are part of a rapidly expanding area of research whereby such personalized treatments can be identified. These methods can lead to improved results over standard 'one size fits all' approaches, as well as provide a route to formalizing a common practice of using ad hoc approaches when deciding or updating management plans. Here, we provide an introduction to adaptive treatment strategies, explaining their background, their purpose, and how they can be employed in practice.

Optimization of individualized dynamic treatment regimes for recurrent diseases.

Patients with cancer or other recurrent diseases may undergo a long process of initial treatment, disease recurrences, and salvage treatments. It is important to optimize the multi-stage treatment sequence in this process to maximally prolong patients' survival. Comparing disease-free survival for each treatment stage over penalizes disease recurrences but under penalizes treatment-related mortalities. Moreover, treatment regimes used in practice are dynamic; that is, the choice of next treatment depends on a patient's responses to previous therapies. In this article, using accelerated failure time models, we develop a method to optimize such dynamic treatment regimes. This method utilizes all the longitudinal data collected during the multi-stage process of disease recurrences and treatments, and identifies the optimal dynamic treatment regime for each individual patient by maximizing his or her expected overall survival. We illustrate the application of this method using data from a study of acute myeloid leukemia, for which the optimal treatment strategies for different patient subgroups are identified.