Batch Mode Reinforcement Learning Research Articles

Digital twin based reinforcement learning for extracting network structures and load patterns in planning and operation of distribution systems

Low voltage distribution networks deliver power to the last mile of the network, but are often legacy assets from a time when low carbon technologies, e.g., electrified heat, storage, and electric vehicles, were not envisaged. Furthermore, exploiting emerging data from distribution networks to provide decision support for adapting planning and operational strategies with system transitions presents a challenge. To overcome these challenges, this paper proposes a novel application of digital twins based reinforcement learning to improve decision making by a distribution system operator, with key metrics of predictability, responsiveness, interoperability, and automation. The power system states, i.e., network configurations, technological combinations, and load patterns, are captured via a convolutional neural network, chosen for its pattern recognition capability with high-dimensional inputs. The convolutional neural networks are iteratively trained through the fitted Q-iteration algorithm, as a batch mode reinforcement learning, to adapt the planning and operational decisions with the dynamic system transitions. Case studies demonstrate the effectiveness of the proposed model by reducing 50% of the investment cost when the system transitions towards the winter and maintaining the power loss and loss of load within 5% compared to the benchmark optimisation. Doubled power consumption was observed in winter under future energy scenarios due to the electrification of heat. The trained model can accurately adapt optimal decisions according to the system changes while reducing the computational time of solving optimisation problems, for a range of scales of distribution systems, demonstrating its potential for scalable deployment by a system operator.

Kernel-Based Reinforcement Learning on Representative States

Markov decision processes (MDPs) are an established framework for solving sequential decision-making problems under uncertainty. In this work, we propose a new method for batch-mode reinforcement learning (RL) with continuous state variables. The method is an approximation to kernel-based RL on a set of k representative states. Similarly to kernel-based RL, our solution is a fixed point of a kernelized Bellman operator and can approximate the optimal solution to an arbitrary level of granularity. Unlike kernel-based RL, our method is fast. In particular, our policies can be computed in O(n) time, where n is the number of training examples. The time complexity of kernel-based RL is Ω(n2). We introduce our method, analyze its convergence, and compare it to existing work. The method is evaluated on two existing control problems with 2 to 4 continuous variables and a new problem with 64 variables. In all cases, we outperform state-of-the-art results and offer simpler solutions.

Efficient Batch-Mode Reinforcement Learning Using Extreme Learning Machines

As a class of batch-mode reinforcement learning (RL) methods for Markov decision problems with large or continuous state spaces, approximate policy iteration (API) has received increasing attention in the past decades. One open problem in the design of API algorithms is how to construct the basis functions or features for value function approximation (VFA). In this paper, we propose a novel batch-mode RL approach with randomly projected features for VFA. The proposed approach can be viewed as an extension of extreme learning machines (ELMs) to RL problems so it can be called ELM-API. The ELMs have been popularly studied in supervised learning problems, but there is not much work on the extension of ELMs to learning control problems. The proposed approach has advantages over the previous API algorithms in that the features for VFA can be quickly generated without complex parameter selection and the performance will be adaptive to different sample sets in batch-mode RL. In particular, the ELM-API approach can realize fast and efficient feature reconstruction when training sample sets are relatively small. Comprehensive simulation studies on two benchmark learning control problems were carried out to test the performance of API algorithms with different feature construction methods. It is shown that the ELM-API algorithm can obtain comparable or better performance than the previous API approaches. To further show the effectiveness of ELM-API in real-world applications, the simulation results on a more challenging high-dimensional lane-changing decision problem in dynamic traffic environment are also reported, which show the capability of the ELM-API algorithm in learning satisfactory lane-changing policies with high data efficiency.

Batch Mode TD($\lambda$ ) for Controlling Partially Observable Gene Regulatory Networks.

External control of gene regulatory networks (GRNs) has received much attention in recent years. The aim is to find a series of actions to apply to a gene regulation system making it avoid its diseased states. In this work, we propose a novel method for controlling partially observable GRNs combining batch mode reinforcement learning (Batch RL) and TD() algorithms. Unlike the existing studies inferring a computational model from gene expression data, and obtaining a control policy over the constructed model, our idea is to interpret the time series gene expression data as a sequence of observations that the system produced, and obtain an approximate stochastic policy directly from the gene expression data without estimation of the internal states of the partially observable environment. Thereby, we get rid of the most time consuming phases of the existing studies, inferring a model and running the model for the control. Results show that our method is able to provide control solutions for regulation systems of several thousands of genes only in seconds, whereas existing studies cannot solve control problems of even a few dozens of genes. Results also show that our approximate stochastic policies are almost as good as the policies generated by the existing studies.

Reinforcement Learning of Heuristic EV Fleet Charging in a Day-Ahead Electricity Market

This paper addresses the problem of defining a day-ahead consumption plan for charging a fleet of electric vehicles (EVs), and following this plan during operation. A challenge herein is the beforehand unknown charging flexibility of EVs, which depends on numerous details about each EV (e.g., plug-in times, power limitations, battery size, power curve, etc.). To cope with this challenge, EV charging is controlled during opertion by a heuristic scheme, and the resulting charging behavior of the EV fleet is learned by using batch mode reinforcement learning. Based on this learned behavior, a cost-effective day-ahead consumption plan can be defined. In simulation experiments, our approach is benchmarked against a multistage stochastic programming solution, which uses an exact model of each EVs charging flexibility. Results show that our approach is able to find a day-ahead consumption plan with comparable quality to the benchmark solution, without requiring an exact day-ahead model of each EVs charging flexibility.

Trajectory-Based Supplementary Damping Control for Power System Electromechanical Oscillations

This paper considers a trajectory-based approach to determine control signals superimposed to those of existing controllers so as to enhance the damping of electromechanical oscillations. This approach is framed as a discrete-time, multi-step optimization problem which can be solved by model-based and/or by learning-based methods. This paper proposes to apply a model-free tree-based batch mode reinforcement learning (RL) algorithm to perform such a supplementary damping control based only on information collected from observed trajectories of the power system. This RL-based supplementary damping control scheme is first implemented on a single generator and then several possibilities are investigated for extending it to multiple generators. Simulations are carried out on a 16-generator medium-size power system model, where also possible benefits of combining this RL-based control with model predictive control (MPC) are assessed.

Planning the Optimal Operation of a Multioutlet Water Reservoir with Water Quality and Quantity Targets

The integration of quality and quantity issues in the management of water resources systems is key to meet society’s long-term needs for freshwater while maintaining essential ecological services and economic benefits. Current water management practices are mostly targeted towards quantitative uses, and quality is usually addressed separately as an independent problem. One of the reasons for the lack of integration lies in the inadequacy of optimization techniques nowadays available to cope with the large, distributed, simulation models adopted to characterize the coupled ecological and biochemical processes in water bodies. In this paper we propose a novel approach based on the conjunctive use of a batch-mode Reinforcement Learning algorithm and a one-dimensional (1D) coupled hydrodynamic-ecological model to design the optimal operation of a multipurpose water reservoir accounting for both quantity and quality targets. We consider up to five operating objectives, including both in-reservoir and downstream water quality parameters, and design efficient operating policies conditioned upon not only the current storage but also water characteristics, such as temperature and total suspended solids at different depths. The approach is applied to a real world case study in Japan consisting of a water reservoir, Tono Dam, equipped with a selective withdrawal structure and used for flood protection, irrigation and recreational purposes. Results show that a potential control over in-reservoir and downstream water quality can be gained without impairing the hydraulic capacity of the reservoir by effectively exploiting—through the operating policy—the operational flexibility provided by the selective withdrawal structures.

Min Max Generalization for Deterministic Batch Mode Reinforcement Learning: Relaxation Schemes

We study the $\min \max$ optimization problem introduced in Fonteneau et al. [Towards min max reinforcement learning, ICAART 2010, Springer, Heidelberg, 2011, pp. 61--77] for computing policies for batch mode reinforcement learning in a deterministic setting with fixed, finite time horizon. First, we show that the $\min$ part of this problem is NP-hard. We then provide two relaxation schemes. The first relaxation scheme works by dropping some constraints in order to obtain a problem that is solvable in polynomial time. The second relaxation scheme, based on a Lagrangian relaxation where all constraints are dualized, can also be solved in polynomial time. We also theoretically prove and empirically illustrate that both relaxation schemes provide better results than those given in [Fonteneau et al., 2011, as cited above].

Batch mode reinforcement learning based on the synthesis of artificial trajectories

In this paper, we consider the batch mode reinforcement learning setting, where the central problem is to learn from a sample of trajectories a policy that satisfies or optimizes a performance criterion. We focus on the continuous state space case for which usual resolution schemes rely on function approximators either to represent the underlying control problem or to represent its value function. As an alternative to the use of function approximators, we rely on the synthesis of "artificial trajectories" from the given sample of trajectories, and show that this idea opens new avenues for designing and analyzing algorithms for batch mode reinforcement learning.

Tree-Based Batch Mode Reinforcement Learning

Reinforcement learning aims to determine an optimal control policy from interaction with a system or from observations gathered from a system. In batch mode, it can be achieved by approximating the...