Existing Reinforcement Learning Methods Research Articles

Automatic index selection with learned cost estimator

Choosing useful indexes for a relational database is important for efficient query optimization. However, current methods tend to use the cost estimated by the optimizer in the database management system (DBMS) to measure the benefits of the index and, due to inaccurate cost estimates, do not bring to the most optimal solution. In addition, existing reinforcement learning methods treat the creation of different indexes as independent actions, ignoring the relationships among indexes, which may bring to unnecessary training costs. To address these problems, we propose DeepIndex, an automatic index selector with a learning-based cost estimator to improve the quality of index selection. To accurately estimate the benefit of an index, we design a learning-based cost estimator to predict the execution time of queries on certain indexes. In particular, we treat query plans as graphs and develop a model based on graph convolutional network (GCN) to learn features from queries and indexes. After that, we design a reinforcement-learning-based index selection model considering the relationships among indexes and combine our cost estimator to select indexes. Extensive experiments on two benchmark workloads JOB and TPC-DS show that our model performs better than state-of-the-art models with the lowest storage costs, and our lowest relative execution cost outperforms the baselines by 38.59% on JOB and 10.42% on TPC-DS.

Clustering experience replay for the effective exploitation in reinforcement learning

Reinforcement learning is a useful tool for training an agent to effectively achieve the desired goal in the sequential decision-making problem. It trains the agent to make decision by exploiting the experience in the transitions resulting from the different decisions. To exploit this experience, most reinforcement learning methods replay the explored transitions by uniform sampling. But in this way, it is easy to ignore the last explored transitions. Another way to exploit this experience defines the priority of each transition by the estimation error in training and then replays the transitions according to their priorities. But it only updates the priorities of the transitions replayed at the current training time step, thus the transitions with low priorities will be ignored. In this paper, we propose a clustering experience replay, called CER, to effectively exploit the experience hidden in all explored transitions in the current training. CER clusters and replays the transitions by a divide-and-conquer framework based on time division as follows. Firstly, it divides the whole training process into several periods. Secondly, at the end of each period, it uses k-means to cluster the transitions explored in this period. Finally, it constructs a conditional probability density function to ensure that all kinds of transitions will be sufficiently replayed in the current training. We construct a new method, TD3_CER, to implement our clustering experience replay on TD3. Through the theoretical analysis and experiments, we illustrate that our TD3_CER is more effective than the existing reinforcement learning methods. The source code can be downloaded from https://github.com/grcai/CER-Master.

Novel two-dimensional off-policy [formula omitted]-learning method for output feedback optimal tracking control of batch process with unknown dynamics

Reinforcement learning (RL) is an artificial intelligence algorithm that can learn adaptive optimal control law online. In view of the fact that the previous control approaches were usually overly dependent on the model parameters of system, and most existing RL methods are based on state feedback, their application in actual industrial production is limited. Additionally, developing accurate process system models and ensuring the closed-loop system’s control performance is more challenging, as modern businesses place a premium on product quality and economic efficiency. As a result, this work introduces a novel data-driven two-dimensional (2D) off-policy Q-learning method based on output feedback is used to achieve optimal tracking control for batch process. First, the error between the actual output and the given set-point is extended to the system to ensure the good tracking performance. Second, by analyzing the relationship between the value function and the Q-function obtained from the 2D system’s performance index, the 2D Bellman equation is obtained in terms of output feedback that is independent of the model parameters. The optimal control problem can be effectively solved by the proposed method in this paper when the policy iteration is executed using only the measurement data of system along the batch and time directions. Following that, the proposed approach’s unbiasedness and convergence are strictly confirmed. Finally, the simulation results for the injection molding process demonstrate that the proposed method is capable of determining the optimal control law as the number of batches is growing increasingly.

Adapting Reinforcement Learning Treatment Policies Using Limited Data to Personalize Critical Care

Reinforcement learning (RL) demonstrates promise for developing effective treatment policies in critical care settings. However, existing RL methods often require large and comprehensive patient data sets and do not readily lend themselves to settings in which certain patient subpopulations are severely underrepresented. In this study, we develop a new method, noisy Bayesian policy updates (NBPU), for selecting high-performing reinforcement learning–based treatment policies for underrepresented patient subpopulations using limited observations. Our method uses variational inference to learn a probability distribution over treatment policies based on a reference patient subpopulation for which sufficient data are available. It then exploits limited data from an underrepresented patient subpopulation to update this probability distribution and adapts its recommendations to this subpopulation. We demonstrate our method’s utility on a data set of ICU patients receiving intravenous blood anticoagulant medication. Our results show that NBPU outperforms state-of-the-art methods in terms of both selecting effective treatment policies for patients with nontypical clinical characteristics and predicting the corresponding policies’ performance for these patients.

Learning-based airborne sensor task assignment in unknown dynamic environments

In sensor management, the existing researches rely on traditional system modeling and strive to maximize the information superiority. In fact, on the one hand, complex environmental disturbance, incomplete information or uncooperative behavior in air combat missions often bring out unknown system evolution; on the other hand, to take full advantage of sensor effectiveness is of course essential, but more importantly, the detection security is the primary guarantee. This paper proposes the airborne sensor task assignment problem in unknown dynamic environments. Different from traditional methods that minimize the estimation error covariance or information entropy based on system dynamic model, our scheme needs to maximize agent survival while maintaining the necessary sensor detection without such model support. In assignment implementation, it is not straightforward to apply existing reinforcement learning methods, but design the state space and rewards ingeniously to meet the actual combat requirements. First, instead of selecting the locations of agents and targets as fundamental and infinite state variables, we consider the situation variables, such as target threat ranking together with cumulative radiation and information acquisition indication of sensors, which are all discrete state variables to reduce computational burden. Second, the reward structure is also designed based on the complex constraints of the mission, which is to encourage lower assignment risk and relatively full utilization of sensing, while penalizing too dangerous continuance assignment and inadequate assignment revenue. Simulations show that our proposed scheme achieves the desirable mission completion rate and the acceptable target tracking accuracy.

Kernel Temporal Difference based Reinforcement Learning for Brain Machine Interfaces.

Brain-machine interfaces (BMIs) enable people with disabilities to control external devices with their motor intentions through a decoder. Compared with supervised learning, reinforcement learning (RL) is more promising for the disabled because it can assist them to learn without actual limb movement. Current RL decoders deal with tasks with immediate reward delivery. But for tasks where the reward is only given by the end of the trial, existing RL methods may take a long time to train and are prone to becoming trapped in the local minima. In this paper, we propose to embed temporal difference method (TD) into Quantized Attention-Gated Kernel Reinforcement Learning (QAGKRL) to solve this temporal credit assignment problem. This algorithm utilizes a kernel network to ensure the global linear structure and adopts a softmax policy to efficiently explore the state-action mapping through TD error. We simulate a center-out task where the agent needs several steps to first reach a periphery target and then return to the center to get the external reward. Our proposed algorithm is tested on simulated data and compared with two state-of-the-art models. We find that introducing the TD method to QAGKRL achieves a prediction accuracy of 96.2% ± 0.77% (mean ± std), which is significantly better the other two methods.Clinical Relevance-This paper proposes a novel kernel temporal difference RL method for the multi-step task with delayed reward delivery, which potentially enables BMI online continuous decoding.

A CNN-based policy for optimizing continuous action control by learning state sequences

Continuous action control is widespread in real-world applications. It controls an agent to take action in continuous space for transiting from one state to another until achieving the desired goal. The optimization of continuous action control is an important issue, which aims to find the optimal policy for the agent to achieve the desired goal with the lowest consumption in continuous action space. A useful tool for this issue is reinforcement learning where an optimal policy is learned for the agent by maximizing the cumulative reward of the state transitions. When updating the policy at each state, most existing reinforcement learning methods consider only the one-step transition of this state. However, for each state in continuous action control, the recognizable information is usually hidden in the sequence of its previous states, thus these methods cannot learn the policy effectively enough for continuous action control. In this paper, we propose a new policy, called convolutional deterministic policy, to solve this problem. Enlightened from the convolutional neural networks used in natural language processing, our convolutional deterministic policy uses convolutional neural networks to learn the recognizable information in the state sequences. Then for each collected state, we update the convolutional deterministic policy by not only the recognizable information in the one-step transition of this state but also the recognizable information in the sequence of its previous states. As a result, our convolutional deterministic policy can make the agent take better action. Based on an effective reinforcement learning method, TD3, the implementation of our convolutional deterministic policy is in CTD3. The theoretical analysis and the experiment illustrate that our CTD3 can learn the policy not only better than but also faster than the existing RL methods for continuous action control. The source code can be downloaded from https://github.com/grcai.

Applying reinforcement learning and tree search to the unit commitment problem

Recent advances in artificial intelligence have demonstrated the capability of reinforcement learning (RL) methods to outperform the state of the art in decision-making problems under uncertainty. Day-ahead unit commitment (UC), scheduling power generation based on forecasts, is a complex power systems task that is becoming more challenging in light of increasing uncertainty. While RL is a promising framework for solving the UC problem, the space of possible actions from a given state is exponential in the number of generators and it is infeasible to apply existing RL methods in power systems larger than a few generators. Here we present a novel RL algorithm, guided tree search, which does not suffer from an exponential explosion in the action space with increasing number of generators. The method augments a tree search algorithm with a policy that intelligently reduces the branching factor. Using data from the GB power system, we demonstrate that guided tree search outperforms an unguided method in terms of computational complexity, while producing solutions that show no performance loss in terms of operating costs. We compare solutions against mixed-integer linear programming (MILP) and find that guided tree search outperforms a solution using reserve constraints, the current industry approach. The RL solutions exhibit complex behaviours that differ qualitatively from MILP, demonstrating its potential as a decision support tool for human operators.

A Hybrid Multi-Task Learning Approach for Optimizing Deep Reinforcement Learning Agents

Driven by recent technological advancements within the field of artificial intelligence (AI), deep learning (DL) has been emerged as a promising representation learning technique across different machine learning (ML) classes, especially within the reinforcement learning (RL) arena. This new direction has given rise to the evolution of a new technological domain named deep reinforcement learning (DRL) that combines the high representational learning capabilities of DL with existing RL methods. Performance optimization achieved by RL-based intelligent agents designed with model-free-based approaches was majorly limited to systems with RL algorithms focused on learning a single task. The aforementioned approach was found to be quite data inefficient, whenever DRL agents needed to interact with more complex, data-rich environments. This is primarily due to the limited applicability of DRL algorithms to many scenarios across related tasks from the same distribution. One of the possible approaches to mitigate this issue is by adopting the method of multi-task learning. The objective of this research paper is to present a hybrid multi-task learning-oriented approach for the optimization of DRL agents operating within different but semantically similar environments with related tasks. The proposed framework will be built with multiple, individual actor-critic models functioning within independent environments and transferring knowledge among themselves through a global network to optimize performance. The empirical results obtained by the hybrid multi-task learning model on OpenAI Gym based Atari 2600 video gaming environment demonstrates that the proposed model enhances the performance of the DRL agent relatively in the range of 15% to 20% margin.

Vocabulary-Wide Credit Assignment for Training Image Captioning Models.

Reinforcement learning (RL) algorithms have been shown to be efficient in training image captioning models. A critical step in RL algorithms is to assign credits to appropriate actions. There are mainly two classes of credit assignment methods in existing RL methods for image captioning, assigning a single credit for the whole sentence and assigning a credit to every word in the sentence. In this article, we propose a new credit assignment method which is orthogonal to the above two. It assigns every word in vocabulary an appropriate credit at each generation step. It is called vocabulary-wide credit assignment. Based on this we propose a Vocabulary-Critical Sequence Training (VCST). VCST can be incorporated into existing RL methods for training image captioning models to achieve better results. Extensive experiments with many popular models validated the effectiveness of VCST.

Joint Scheduling of Deferrable Demand and Storage With Random Supply and Processing Rate Limits

We study the joint scheduling of deferrable demands (e.g., the charging of electric vehicles) and storage systems in the presence of random supply, demand arrivals, processing costs, and subject to processing rate limit constraint. We formulate the scheduling problem as a dynamic program so as to minimize the expected total cost, the sum of processing costs, and the noncompletion penalty (incurred when a task is not fully processed by its deadline). Under mild assumptions, we characterize an optimal index-based priority rule: Tasks with less laxity should be processed first, and for two tasks with the same laxity, the task with a later deadline has the priority. Based on the established optimal control policy characterizations (on resource allocation among multitasks and storage operation), we propose to apply data-driven reinforcement learning (RL) methods to make energy procurement decisions. Numerical results show that the proposed approach significantly outperforms existing RL methods combined with the earliest deadline first priority rule (by reducing 26%–32% of system cost).

Fuzzy Logic-Driven Variable Time-Scale Prediction-Based Reinforcement Learning for Robotic Multiple Peg-in-Hole Assembly

Reinforcement learning (RL) has been increasingly used for single peg-in-hole assembly, where assembly skill is learned through interaction with the assembly environment in a manner similar to skills employed by human beings. However, the existing RL algorithms are difficult to apply to the multiple peg-in-hole assembly because the much more complicated assembly environment requires sufficient exploration, resulting in a long training time and less data efficiency. To this end, this article focuses on how to predict the assembly environment and how to use the predicted environment in assembly action control to improve the data efficiency of the RL algorithm. Specifically, first, the assembly environment is exactly predicted by a variable time-scale prediction (VTSP) defined as general value functions (GVFs), reducing the unnecessary exploration. Second, we propose a fuzzy logic-driven variable time-scale prediction-based reinforcement learning (FLDVTSP-RL) for assembly action control to improve the efficiency of the RL algorithm, in which the predicted environment is mapped to the impedance parameter in the proposed impedance action space by a fuzzy logic system (FLS) as the action baseline. To demonstrate the effectiveness of VTSP and the data efficiency of the FLDVTSP-RL methods, a dual peg-in-hole assembly experiment is set up; the results show that FLDVTSP-deep Q-learning (DQN) decreases the assembly time about 44% compared with DQN and FLDVTSP-deep deterministic policy gradient (DDPG) decreases the assembly time about 24% compared with DDPG. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —The complicated assembly environment of the multiple peg-in-hole assembly results in a contact state that cannot be recognized exactly from the force sensor. Therefore, contact-model-based methods that require tuning of the control parameters based on the contact state recognition cannot be applied directly in this complicated environment. Recently, reinforcement learning (RL) methods without contact state recognition have recently attracted scientific interest. However, the existing RL methods still rely on numerous explorations and a long training time, which cannot be directly applied to real-world tasks. This article takes inspiration from the manner in which human beings can learn assembly skills with a few trials, which relies on the variable time-scale predictions (VTSPs) of the environment and the optimized assembly action control strategy. Our proposed fuzzy logic-driven variable time-scale prediction-based reinforcement learning (FLDVTSP-RL) can be implemented in two steps. First, the assembly environment is predicted by the VTSP defined as general value functions (GVFs). Second, assembly action control is realized in an impedance action space with a baseline defined by the impedance parameter mapped from the predicted environment by the fuzzy logic system (FLS). Finally, a dual peg-in-hole assembly experiment is conducted; compared with deep Q-learning (DQN), FLDVTSP-DQN can decrease the assembly time about 44%; compared with deep deterministic policy gradient (DDPG), FLDVTSP-DDPG can decrease the assembly time about 24%.

A Survey of Multi-Task Deep Reinforcement Learning

Driven by the recent technological advancements within the field of artificial intelligence research, deep learning has emerged as a promising representation learning technique across all of the machine learning classes, especially within the reinforcement learning arena. This new direction has given rise to the evolution of a new technological domain named deep reinforcement learning, which combines the representational learning power of deep learning with existing reinforcement learning methods. Undoubtedly, the inception of deep reinforcement learning has played a vital role in optimizing the performance of reinforcement learning-based intelligent agents with model-free based approaches. Although these methods could improve the performance of agents to a greater extent, they were mainly limited to systems that adopted reinforcement learning algorithms focused on learning a single task. At the same moment, the aforementioned approach was found to be relatively data-inefficient, particularly when reinforcement learning agents needed to interact with more complex and rich data environments. This is primarily due to the limited applicability of deep reinforcement learning algorithms to many scenarios across related tasks from the same environment. The objective of this paper is to survey the research challenges associated with multi-tasking within the deep reinforcement arena and present the state-of-the-art approaches by comparing and contrasting recent solutions, namely DISTRAL (DIStill &amp; TRAnsfer Learning), IMPALA(Importance Weighted Actor-Learner Architecture) and PopArt that aim to address core challenges such as scalability, distraction dilemma, partial observability, catastrophic forgetting and negative knowledge transfer.

Online optimal and adaptive integral tracking control for varying discrete‐time systems using reinforcement learning

SummaryConventional closed‐form solution to the optimal control problem using optimal control theory is only available under the assumption that there are known system dynamics/models described as differential equations. Without such models, reinforcement learning (RL) as a candidate technique has been successfully applied to iteratively solve the optimal control problem for unknown or varying systems. For the optimal tracking control problem, existing RL techniques in the literature assume either the use of a predetermined feedforward input for the tracking control, restrictive assumptions on the reference model dynamics, or discounted tracking costs. Furthermore, by using discounted tracking costs, zero steady‐state error cannot be guaranteed by the existing RL methods. This article therefore presents an optimal online RL tracking control framework for discrete‐time (DT) systems, which does not impose any restrictive assumptions of the existing methods and equally guarantees zero steady‐state tracking error. This is achieved by augmenting the original system dynamics with the integral of the error between the reference inputs and the tracked outputs for use in the online RL framework. It is further shown that the resulting value function for the DT linear quadratic tracker using the augmented formulation with integral control is also quadratic. This enables the development of Bellman equations, which use only the system measurements to solve the corresponding DT algebraic Riccati equation and obtain the optimal tracking control inputs online. Two RL strategies are thereafter proposed based on both the value function approximation and the Q‐learning along with bounds on excitation for the convergence of the parameter estimates. Simulation case studies show the effectiveness of the proposed approach.

Guided Constrained Policy Optimization for Dynamic Quadrupedal Robot Locomotion

Deep reinforcement learning (RL) uses model-free techniques to optimize task-specific control policies. Despite having emerged as a promising approach for complex problems, RL is still hard to use reliably for real-world applications. Apart from challenges such as precise reward function tuning, inaccurate sensing and actuation, and non-deterministic response, existing RL methods do not guarantee behavior within required safety constraints that are crucial for real robot scenarios. In this regard, we introduce guided constrained policy optimization (GCPO), an RL framework based upon our implementation of constrained proximal policy optimization (CPPO) for tracking base velocity commands while following the defined constraints. We introduce schemes which encourage state recovery into constrained regions in case of constraint violations. We present experimental results of our training method and test it on the real ANYmal quadruped robot. We compare our approach against the unconstrained RL method and show that guided constrained RL offers faster convergence close to the desired optimum resulting in an optimal, yet physically feasible, robotic control behavior without the need for precise reward function tuning.

Safe deep reinforcement learning-based constrained optimal control scheme for active distribution networks

Reinforcement learning-based schemes are being recently applied for model-free voltage control in active distribution networks. However, existing reinforcement learning methods face challenges when it comes to continuous state and action spaces problems or problems with operation constraints. To address these limitations, this paper proposes an optimal voltage control scheme based on the safe deep reinforcement learning. In this scheme, the optimal voltage control problem is formulated as a constrained Markov decision process, in which both state and action spaces are continuous. To solve this problem efficiently, the deep deterministic policy gradient algorithm is utilized to learn the reactive power control policies, which determine the optimal control actions from the states. In contrast to existing reinforcement learning methods, deep deterministic policy gradient is naturally capable of addressing control problems with continuous state and action spaces. This is due to the utilization of deep neural networks to approximate both value function and policy. In addition, in order to handle the operation constraints in active distribution networks, a safe exploration approach is proposed to form a safety layer, which is composed directly on top the deep deterministic policy gradient actor network. This safety layer predicts the change in the constrained states and prevents the violation of active distribution networks operation constraints. Numerical simulations on modified IEEE test systems demonstrate that the proposed scheme successfully maintains all bus voltage within the allowed range, and reduces the system loss by 15% compared to the no control case.

Generation of Policy-Level Explanations for Reinforcement Learning

Though reinforcement learning has greatly benefited from the incorporation of neural networks, the inability to verify the correctness of such systems limits their use. Current work in explainable deep learning focuses on explaining only a single decision in terms of input features, making it unsuitable for explaining a sequence of decisions. To address this need, we introduce Abstracted Policy Graphs, which are Markov chains of abstract states. This representation concisely summarizes a policy so that individual decisions can be explained in the context of expected future transitions. Additionally, we propose a method to generate these Abstracted Policy Graphs for deterministic policies given a learned value function and a set of observed transitions, potentially off-policy transitions used during training. Since no restrictions are placed on how the value function is generated, our method is compatible with many existing reinforcement learning methods. We prove that the worst-case time complexity of our method is quadratic in the number of features and linear in the number of provided transitions, O(|F|2|tr samples|). By applying our method to a family of domains, we show that our method scales well in practice and produces Abstracted Policy Graphs which reliably capture relationships within these domains.

Learning the Dynamic Treatment Regimes from Medical Registry Data through Deep Q-network

This paper presents the deep reinforcement learning (DRL) framework to estimate the optimal Dynamic Treatment Regimes from observational medical data. This framework is more flexible and adaptive for high dimensional action and state spaces than existing reinforcement learning methods to model real-life complexity in heterogeneous disease progression and treatment choices, with the goal of providing doctors and patients the data-driven personalized decision recommendations. The proposed DRL framework comprises (i) a supervised learning step to predict expert actions, and (ii) a deep reinforcement learning step to estimate the long-term value function of Dynamic Treatment Regimes. Both steps depend on deep neural networks. As a key motivational example, we have implemented the proposed framework on a data set from the Center for International Bone Marrow Transplant Research (CIBMTR) registry database, focusing on the sequence of prevention and treatments for acute and chronic graft versus host disease after transplantation. In the experimental results, we have demonstrated promising accuracy in predicting human experts’ decisions, as well as the high expected reward function in the DRL-based dynamic treatment regimes.

System Design Perspective for Human-Level Agents Using Deep Reinforcement Learning: A Survey

Reinforcement learning (RL) has distinguished itself as a prominent learning method to augment the efficacy of autonomous systems. Recent advances in deep learning studies have complemented existing RL methods and led to a crucial breakthrough in the effort of applying RL to automation and robotics. Artificial agents based on deep RL can take selective and intelligent actions comparable with those of a human to maximize the feedback reward from the interactive environment. In this paper, we survey recent developments in the literature regarding deep RL methods for building human-level agents. As a result, prominent studies that involve modeling every aspect of a human-level agent will be examined. We also provide an overview of constructing a framework for prospective autonomous systems. Moreover, various toolkits and frameworks are suggested to facilitate the development of deep RL methods. Finally, we open a discussion that potentially raises a range of future research directions in deep RL.

Efficient Behavior Learning Based on State Value Estimation of Self and Others

The existing reinforcement learning methods have been seriously suffering from the curse of the dimension problem, especially when they are applied to multiagent dynamic environments. One of the typical examples is a case of RoboCup competitions since other agents and their behavior easily cause state and action space variations. This paper presents a method of modular learning in a multiagent environment by which the learning agent can acquire cooperative behavior with its teammates and competitive behavior against its opponents. The key ideas to resolve the issue are as follows. First, a two-layer hierarchical system with multilearning modules is adopted to reduce the size of the sensor and action spaces. The state space of the top layer consists of the state values from the lower level and the macro actions are used to reduce the size of the physical action space. Second, the state of the other, to what extent it is close to its own goal, is estimated by observation and used as a state variable in the top layer state space to realize the cooperative/competitive behavior. The method is applied to a four (defense team)-on-five (offense team) game task and the learning agent (a passer of the offense team) successfully acquired the teamwork plays (pass and shoot) within much shorter learning time.