Dynamic Treatment Regimes Research Articles

Personalizing medicine: a review of adaptive treatment strategies

Much of current pharmacological practice focuses on identifying the single 'best' treatment (or course of treatments) for a particular disease. Recently, however, focus has begun to shift towards a more patient-centric rather than disease-centric approach, where personal characteristics are used to identify the optimal treatment for an individual. Adaptive treatment strategies (also known as dynamic treatment regimes) are part of a rapidly expanding area of research whereby such personalized treatments can be identified. These methods can lead to improved results over standard 'one size fits all' approaches, as well as provide a route to formalizing a common practice of using ad hoc approaches when deciding or updating management plans. Here, we provide an introduction to adaptive treatment strategies, explaining their background, their purpose, and how they can be employed in practice.

Optimization of individualized dynamic treatment regimes for recurrent diseases.

Patients with cancer or other recurrent diseases may undergo a long process of initial treatment, disease recurrences, and salvage treatments. It is important to optimize the multi-stage treatment sequence in this process to maximally prolong patients' survival. Comparing disease-free survival for each treatment stage over penalizes disease recurrences but under penalizes treatment-related mortalities. Moreover, treatment regimes used in practice are dynamic; that is, the choice of next treatment depends on a patient's responses to previous therapies. In this article, using accelerated failure time models, we develop a method to optimize such dynamic treatment regimes. This method utilizes all the longitudinal data collected during the multi-stage process of disease recurrences and treatments, and identifies the optimal dynamic treatment regime for each individual patient by maximizing his or her expected overall survival. We illustrate the application of this method using data from a study of acute myeloid leukemia, for which the optimal treatment strategies for different patient subgroups are identified.

Set‐valued dynamic treatment regimes for competing outcomes

Dynamic treatment regimes (DTRs) operationalize the clinical decision process as a sequence of functions, one for each clinical decision, where each function maps up-to-date patient information to a single recommended treatment. Current methods for estimating optimal DTRs, for example Q-learning, require the specification of a single outcome by which the "goodness" of competing dynamic treatment regimes is measured. However, this is an over-simplification of the goal of clinical decision making, which aims to balance several potentially competing outcomes, for example, symptom relief and side-effect burden. When there are competing outcomes and patients do not know or cannot communicate their preferences, formation of a single composite outcome that correctly balances the competing outcomes is not possible. This problem also occurs when patient preferences evolve over time. We propose a method for constructing DTRs that accommodates competing outcomes by recommending sets of treatments at each decision point. Formally, we construct a sequence of set-valued functions that take as input up-to-date patient information and give as output a recommended subset of the possible treatments. For a given patient history, the recommended set of treatments contains all treatments that produce non-inferior outcome vectors. Constructing these set-valued functions requires solving a non-trivial enumeration problem. We offer an exact enumeration algorithm by recasting the problem as a linear mixed integer program. The proposed methods are illustrated using data from the CATIE schizophrenia study.

Dynamic Treatment Regimes.

A dynamic treatment regime consists of a sequence of decision rules, one per stage of intervention, that dictate how to individualize treatments to patients based on evolving treatment and covariate history. These regimes are particularly useful for managing chronic disorders, and fit well into the larger paradigm of personalized medicine. They provide one way to operationalize a clinical decision support system. Statistics plays a key role in the construction of evidence-based dynamic treatment regimes - informing best study design as well as efficient estimation and valid inference. Due to the many novel methodological challenges it offers, this area has been growing in popularity among statisticians in recent years. In this article, we review the key developments in this exciting field of research. In particular, we discuss the sequential multiple assignment randomized trial designs, estimation techniques like Q-learning and marginal structural models, and several inference techniques designed to address the associated non-standard asymptotics. We reference software, whenever available. We also outline some important future directions.

Discussion of “Dynamic treatment regimes: Technical challenges and applications”

We congratulate Drs. Laber, Lizotte, Qian, Pelham and Murphy on an outstanding review of dynamic treatment regimes (DTRs). This group has done pioneering work in advancing the theory and applications of DTRs. In a DTR, the treatment type and level are repeatedly adjusted according to an individual’s need. An important part of designing DTRs is to choose tailoring variables. A tailoring variable is a variable that is used to decide how to adjust the treatment. Laber et al. say that the current state of the art for choosing tailoring variable is to construct low-dimensional summaries of information that is available at the time of treatment on a subject’s status, where the summaries are motivated by clinical judgment, exploratory analyses and convenience. Laber et al. state that an important open problem is the development of formal feature extraction and construction techniques for choosing tailoring variables for DTRs. In this discussion, we discuss a method for choosing tailoring variables when the data available is a simple randomized clinical trial in which the treatment regimes after the initial randomization are set by protocol. This includes trials in which the treatment regime is changed over time based on a patient’s status in a set way specified by the protocol. In such trials, at a given time past the initial treatment assignment, there is no variation in how a patient is treated at that time given a patient’s status and past treatment history, i.e., every patient who has the same status and treatment history at time t past initial treatment assignment receives the treatment at time t that is specified by the protocol. Because of this lack of variation in treatment regimes after the initial treatment assignment, such simple randomized clinical trials cannot be used to evaluate DTRs. However, we will show that such simple randomized clinical trials may provide useful data for designing DTRs to evaluate in a later Sequential, Multiple, Assignment Randomized Trial (SMART).

Dynamic treatment regimes: technical challenges and applications.

Dynamic treatment regimes are of growing interest across the clinical sciences because these regimes provide one way to operationalize and thus inform sequential personalized clinical decision making. Formally, a dynamic treatment regime is a sequence of decision rules, one per stage of clinical intervention. Each decision rule maps up-to-date patient information to a recommended treatment. We briefly review a variety of approaches for using data to construct the decision rules. We then review a critical inferential challenge that results from nonregularity, which often arises in this area. In particular, nonregularity arises in inference for parameters in the optimal dynamic treatment regime; the asymptotic, limiting, distribution of estimators are sensitive to local perturbations. We propose and evaluate a locally consistent Adaptive Confidence Interval (ACI) for the parameters of the optimal dynamic treatment regime. We use data from the Adaptive Pharmacological and Behavioral Treatments for Children with ADHD Trial as an illustrative example. We conclude by highlighting and discussing emerging theoretical problems in this area.

Comment on "Dynamic treatment regimes: technical challenges and applications"

Inference for parameters associated with optimal dynamic treatment regimes is challenging as these estimators are nonregular when there are non-responders to treatments. In this discussion, we comment on three aspects of alleviating this nonregularity. We first discuss an alternative approach for smoothing the quality functions. We then discuss some further details on our existing work to identify non-responders through penalization. Third, we propose a clinically meaningful value assessment whose estimator does not suffer from nonregularity.

Discussion of “Dynamic treatment regimes: Technical challenges and applications”

The contributions of the review paper on the technical challenges and applications of dynamic treatment regimes are briefly discussed and potential extensions to continuous action spaces and high dimensional problems are indicated.

Rejoinder of “Dynamic treatment regimes: Technical challenges and applications”

Rejoinder of “Dynamic treatment regimes: Technical challenges and applications”

Comment on “Dynamic treatment regimes: Technical challenges and applications”

Comment on “Dynamic treatment regimes: Technical challenges and applications”

Discussion of “Dynamic treatment regimes: Technical challenges and applications”

Discussion of “Dynamic treatment regimes: Technical challenges and applications”

A decade of Clinical Trials

My colleague and then Society for Clinical Trials (SCT) board member Kay Dickersin called me in spring of 2003 to ask a favor: would I consider taking on the editorship of the society journal, Controlled Clinical Trials? I protested vigorously; too much time, I said, not enough help, no support, I had papers to write, etc, etc. All true, but I knew I would say yes. Editing had long been in my blood, and to be able to continue the strong journal tradition established by Curt Meinert, the founding editor, Janet Wittes and then Jim Neaton was too tempting. Now, 10 years and over 660 published articles later, I am glad I did. But had I known then what I was about to face, I might have demurred. Why? The society was in negotiation with the publisher Elsevier to renew the journal contract. A decision was soon to be made – with my concurrence – that we should jump ship. This meant that I was not about to take over Controlled Clinical Trials, but rather a new journal with a new name, no Medline listing, no impact factor, no institutional subscriptions, no articles in the queue, no staff, little money, and an empty Excel spreadsheet to manage articles that would somehow appear from somewhere. What we did have was a dedicated editorial board – all of whom moved to the new journal – and a society membership that provided the initial subscriber base and source of articles. Looking at the contents of the first issue, one sees what that meant: a lively and eclectic collection of articles and commentaries that still inform and entertain, every one written by someone in the SCT or a colleague. There was a mix of methodology; one on dynamic treatment regimens by Lavori and Dawson [1], three on data monitoring committees, two from the DAMOCLES group (Data Monitoring Committees: Lessons, Ethics, Statistics) [2–4], a delightful retrospective and accompanying editorial on the Salk polio vaccine trial [5,6], and a very nice article on the groundbreaking model of ‘coverage under evidence development’ embodied by the National Emphysema Treatment Trial [7], followed by four terrific commentaries. But my favorite, 10 years later, was a profile of a founding SCT member, Paul Meier [8], by my colleague and erstwhile associate editor Harry Marks [9], neither of whom survived the ensuing decade. That interview, which I commend to all readers who missed it, was a tribute to both Paul and Harry’s wide-ranging scholarship, knowledge, and humanity, and provided rich material for Paul’s obituaries after his death in 2011. That first issue was a harbinger of a decade spent exploring in the journal the multifaceted nature and the scholarship around the methods of human experimentation, which encompasses the fields of statistics, medicine, biology, ethics, epidemiology, economics, history, law, policy, sociology, study design, inference, philosophy, informatics, and yet other fields, with a soupcon of passion. It is hard not to enjoy playing in that sandbox. But as I was warned by a sage colleague at the time, it is not the inaugural issue that tries editors’ souls, but the second one. And so it was. This forced me to contemplate some existential questions, like ‘What is the minimum number of articles that can legitimately be considered a journal ‘‘issue?’’’. The answer had a surprisingly concrete foundation; the number of pages whose thickness was sufficient to fit the words Clinical Trials on the spine of the bound journal. That turned out to be about five articles, the precise number comprising the second issue, and several thereafter. I confess this led to occasional musing about the joys of heavier gauge paper. A medical editor colleague, puzzled when I told him I was taking on this job, asked what there was left to learn about clinical trials. The 600+ articles published since then in this journal represent just a first chapter of an answer. Amazingly, the need to develop further the theory and application of randomized clinical trials is now a pressing national priority. With health-care costs crushing the nation’s economy without concomitant return, and a clinical trial model involving the creation of a parallel care system that is similarly unsustainable,

Q-Learning: Flexible Learning About Useful Utilities

Dynamic treatment regimes are fast becoming an important part of medicine, with the corresponding change in emphasis from treatment of the disease to treatment of the individual patient. Because of the limited number of trials to evaluate personally tailored treatment sequences, inferring optimal treatment regimes from observational data has increased importance. Q-learning is a popular method for estimating the optimal treatment regime, originally in randomized trials but more recently also in observational data. Previous applications of Q-learning have largely been restricted to continuous utility end-points with linear relationships. This paper is the first attempt at both extending the framework to discrete utilities and implementing the modelling of covariates from linear to more flexible modelling using the generalized additive model (GAM) framework. Simulated data results show that the GAM adapted Q-learning typically outperforms Q-learning with linear models and other frequently-used methods based on propensity scores in terms of coverage and bias/MSE. This represents a promising step toward a more fully general Q-learning approach to estimating optimal dynamic treatment regimes.

Inference for optimal dynamic treatment regimes using an adaptive m-out-of-n bootstrap scheme.

A dynamic treatment regime consists of a set of decision rules that dictate how to individualize treatment to patients based on available treatment and covariate history. A common method for estimating an optimal dynamic treatment regime from data is Q-learning which involves nonsmooth operations of the data. This nonsmoothness causes standard asymptotic approaches for inference like the bootstrap or Taylor series arguments to breakdown if applied without correction. Here, we consider the m-out-of-n bootstrap for constructing confidence intervals for the parameters indexing the optimal dynamic regime. We propose an adaptive choice of m and show that it produces asymptotically correct confidence sets under fixed alternatives. Furthermore, the proposed method has the advantage of being conceptually and computationally much simple than competing methods possessing this same theoretical property. We provide an extensive simulation study to compare the proposed method with currently available inference procedures. The results suggest that the proposed method delivers nominal coverage while being less conservative than alternatives. The proposed methods are implemented in the qLearn R-package and have been made available on the Comprehensive R-Archive Network (http://cran.r-project.org/). Analysis of the Sequenced Treatment Alternatives to Relieve Depression (STAR*D) study is used as an illustrative example.

Seven deadly sins in trauma outcomes research

Because randomized clinical trials in trauma outcomes research are expensive and complex, they have rarely been the basis for the clinical care of trauma patients. Most published findings are derived from retrospective and occasionally prospective observational studies that may be particularly susceptible to bias. The sources of bias include some common to other clinical domains, such as heterogeneous patient populations with competing and interdependent short- and long-term outcomes. Other sources of bias are unique to trauma, such as rapidly changing multisystem responses to injury that necessitate highly dynamic treatment regimens such as blood product transfusion. The standard research design and analysis strategies applied in published observational studies are often inadequate to address these biases. Drawing on recent experience in the design, data collection, monitoring, and analysis of the 10-site observational PRospective Observational Multicenter Major Trauma Transfusion (PROMMTT) study, 7 common and sometimes overlapping biases are described through examples and resolution strategies. Sources of bias in trauma research include ignoring (1) variation in patients' indications for treatment (indication bias), (2) the dependency of intervention delivery on patient survival (survival bias), (3) time-varying treatment, (4) time-dependent confounding, (5) nonuniform intervention effects over time, (6) nonrandom missing data mechanisms, and (7) imperfectly defined variables. This list is not exhaustive. The mitigation strategies to overcome these threats to validity require epidemiologic and statistical vigilance. Minimizing the highlighted types of bias in trauma research will facilitate clinical translation of more accurate and reproducible findings and improve the evidence-base that clinicians apply in their care of injured patients.

Robust estimation of optimal dynamic treatment regimes for sequential treatment decisions

A dynamic treatment regime is a list of sequential decision rules for assigning treatment based on a patient's history. Q- and A-learning are two main approaches for estimating the optimal regime, i.e., that yielding the most beneficial outcome in the patient population, using data from a clinical trial or observational study. Q-learning requires postulated regression models for the outcome, while A-learning involves models for that part of the outcome regression representing treatment contrasts and for treatment assignment. We propose an alternative to Q- and A-learning that maximizes a doubly robust augmented inverse probability weighted estimator for population mean outcome over a restricted class of regimes. Simulations demonstrate the method's performance and robustness to model misspecification, which is a key concern.

Q‐learning for estimating optimal dynamic treatment rules from observational data

The area of dynamic treatment regimes (DTR) aims to make inference about adaptive, multistage decision-making in clinical practice. A DTR is a set of decision rules, one per interval of treatment, where each decision is a function of treatment and covariate history that returns a recommended treatment. Q-learning is a popular method from the reinforcement learning literature that has recently been applied to estimate DTRs. While, in principle, Q-learning can be used for both randomized and observational data, the focus in the literature thus far has been exclusively on the randomized treatment setting. We extend the method to incorporate measured confounding covariates, using direct adjustment and a variety of propensity score approaches. The methods are examined under various settings including non-regular scenarios. We illustrate the methods in examining the effect of breastfeeding on vocabulary testing, based on data from the Promotion of Breastfeeding Intervention Trial.

Evaluating Joint Effects of Induction–Salvage Treatment Regimes on Overall Survival in Acute Leukaemia

Typical oncology practice often includes not only an initial, frontline treatment, but also subsequent treatments given if the initial treatment fails. The physician chooses a treatment at each stage based on the patient's baseline covariates and history of previous treatments and outcomes. Such sequentially adaptive medical decision-making processes are known as dynamic treatment regimes, treatment policies, or multi-stage adaptive treatment strategies. Conventional analyses in terms of frontline treatments that ignore subsequent treatments may be misleading, because they actually are an evaluation of more than front-line treatment effects on outcome. We are motivated by data from a randomized trial of four combination chemotherapies given as frontline treatments to patients with acute leukemia. Most patients in the trial also received a second-line treatment, chosen adaptively and subjectively rather than by randomization, either because the initial treatment was ineffective or the patient's cancer later recurred. We evaluate effects on overall survival time of the 16 two-stage strategies that actually were used. Our methods include a likelihood-based regression approach in which the transition times of all possible multi-stage outcome paths are modeled, and estimating equations with inverse probability of treatment weighting to correct for bias. While the two approaches give different numerical estimates of mean survival time, they lead to the same substantive conclusions when comparing the two-stage regimes.

Evaluation of Viable Dynamic Treatment Regimes in a Sequentially Randomized Trial of Advanced Prostate Cancer

We present new statistical analyses of data arising from a clinical trial designed to compare two-stage dynamic treatment regimes (DTRs) for advanced prostate cancer. The trial protocol mandated that patients be initially randomized among four chemotherapies, and that those who responded poorly be re-randomized to one of the remaining candidate therapies. The primary aim was to compare the DTRs’ overall success rates, with success defined by the occurrence of successful responses in each of two consecutive courses of the patient’s therapy. Of the 150 study participants, 47 did not complete their therapy as per the algorithm. However, 35 of them did so for reasons that precluded further chemotherapy, that is, toxicity and/or progressive disease. Consequently, rather than comparing the overall success rates of the DTRs in the unrealistic event that these patients had remained on their assigned chemotherapies, we conducted an analysis that compared viable switch rules defined by the per-protocol rules but with the additional provision that patients who developed toxicity or progressive disease switch to a non-prespecified therapeutic or palliative strategy. This modification involved consideration of bivariate per-course outcomes encoding both efficacy and toxicity. We used numerical scores elicited from the trial’s principal investigator to quantify the clinical desirability of each bivariate per-course outcome, and defined one endpoint as their average over all courses of treatment. Two other simpler sets of scores as well as log survival time were also used as endpoints. Estimation of each DTR-specific mean score was conducted using inverse probability weighted methods that assumed that missingness in the 12 remaining dropouts was informative but explainable in that it only depended on past recorded data. We conducted additional worst- and best-case analyses to evaluate sensitivity of our findings to extreme departures from the explainable dropout assumption.

Estimating the Optimal Dynamic Antipsychotic Treatment Regime: Evidence from the Sequential Multiple-Assignment Randomized Clinical Antipsychotic Trials of Intervention and Effectiveness Schizophrenia Study

Summary Treatment of schizophrenia is notoriously difficult and typically requires personalized adaption of treatment because of lack of efficacy of treatment, poor adherence or intolerable side effects. The Clinical Antipsychotic Trials in Intervention Effectiveness schizophrenia study is a sequential multiple-assignment randomized trial comparing the typical antipsychotic medication perphenazine with several newer atypical antipsychotics. The paper describes the marginal structural modelling method for estimating optimal dynamic treatment regimes and applies the approach to this schizophrenia study. Missing data and valid estimation of confidence intervals are also addressed.