A Multi-Agent Reinforcement Learning Approach to Price and Comfort Optimization in HVAC-Systems

Christian Blad,Simon Bøgh,Carsten Kallesøe

doi:10.3390/en14227491

Christian Blad, Simon Bøgh + Show 1 more

Open Access

https://doi.org/10.3390/en14227491

Copy DOI

Journal: Energies	Publication Date: Nov 9, 2021
Citations: 4	License type: CC BY 4.0

Affiliation: Aalborg University, Grundfos (Denmark)

Abstract

This paper addresses the challenge of minimizing training time for the control of Heating, Ventilation, and Air-conditioning (HVAC) systems with online Reinforcement Learning (RL). This is done by developing a novel approach to Multi-Agent Reinforcement Learning (MARL) to HVAC systems. In this paper, the environment formed by the HVAC system is formulated as a Markov Game (MG) in a general sum setting. The MARL algorithm is designed in a decentralized structure, where only relevant states are shared between agents, and actions are shared in a sequence, which are sensible from a system’s point of view. The simulation environment is a domestic house located in Denmark and designed to resemble an average house. The heat source in the house is an air-to-water heat pump, and the HVAC system is an Underfloor Heating system (UFH). The house is subjected to weather changes from a data set collected in Copenhagen in 2006, spanning the entire year except for June, July, and August, where heat is not required. It is shown that: (1) When comparing Single Agent Reinforcement Learning (SARL) and MARL, training time can be reduced by 70% for a four temperature-zone UFH system, (2) the agent can learn and generalize over seasons, (3) the cost of heating can be reduced by 19% or the equivalent to 750 kWh of electric energy per year for an average Danish domestic house compared to a traditional control method, and (4) oscillations in the room temperature can be reduced by 40% when comparing the RL control methods with a traditional control method.

Highlights

In the USA and Europe, roughly 35% of the energy consumption in 2008 was used in HVAC systems [1]
It is shown that: (1) When comparing Single Agent Reinforcement Learning (SARL) and Multi-Agent Reinforcement Learning (MARL), training time can be reduced by 70% for a four temperature-zone Underfloor Heating system (UFH) system, (2) the agent can learn and generalize over seasons, (3) the cost of heating can be reduced by 19% or the equivalent to 750 kWh of electric energy per year for an average Danish domestic house compared to a traditional control method, and (4) oscillations in the room temperature can be reduced by 40% when comparing the RL control methods with a traditional control method
A test plan is established to validate that the MARL formulation reduces training time and improves scaling capabilities compared to a SARL formulation

Summary

Introduction

In the USA and Europe, roughly 35% of the energy consumption in 2008 was used in HVAC systems [1]. To reduce energy consumption and the carbon footprint from heat and energy production for HVAC systems, the regulation regarding insulation of buildings has increased [2] Another way to reduce energy consumption in buildings is to use more advanced control techniques, reducing energy waste, and increasing comfort. Smart controllers based on scheduling according to energy prices are proposed [5,6]. These algorithms require less commissioning than an MPC’s but are still comprehensive to commission. The central idea of value-based learning is to find the optimal action-value function, which needs to satisfy the Bellman optimality equation (Equation (1)) [8].

Objectives

Results

Discussion

Conclusion