Benchmarking regression methods for function approximation in reinforcement learning: heat pump control

Abstract

Motivated by the increasing interest in the application of machine learning techniques for power system control and demand response applications, this paper presents a benchmark of regression methods (extremely randomized trees (extra-trees), multi-layer perceptron (MLP), extreme gradient boosting, light gradient boosting machines, support vector regression (SVR) and extreme learning machines (ELMs)) available for function approximation in reinforcement learning (RL) techniques. In addition, we use Bayesian optimization to optimally select the hyperparameters of the regression algorithms. As a case study, the control of a heat pump (HP) in a grid-connected single-user microgrid powered by a photovoltaic (PV) installation is considered. The operation of the HP is controlled using fitted Q-iteration (FQI), a batch RL algorithm, with objective of maximizing PV self-consumption and minimizing electricity cost. Simulation results show that extra-trees are a suitable choice for application of FQI in real world applications where the importance on accuracy and computation times are equally weighted. Simulation results also show that a bag of ELMs performs better than (i) a single ELM with a 25.8% increment in PV self-consumption costing a factor of 3.8 increase in the computational time (ii) a MLP with a 5.4% increase in PV self-consumption and with a computation time of 25% that of the MLP. With SVR, a 32% decrease in PV self-consumption with a computation time increased by a factor 50 compared to extra-trees was obtained.