Q-Learning-Based Model Predictive Control for Energy Management in Residential Aggregator

Abstract

This article presents a demand response scheduling model in a residential community using an energy management system aggregator. The aggregator manages a set of resources, including photovoltaic system, energy storage system, thermostatically controllable loads, and electrical vehicles. The solution aims to dynamically control the power demand and distributed energy resources to improve the matching performance between the renewable power generation and the consumption at the community level while trading electricity in both day-ahead and real-time markets to reduce the operational costs in the aggregator. The problem can be formulated as a mixed-integer linear programming problem in which the objective is to minimize the operation and the degradation costs related to the energy storage system and the electric vehicles batteries. To mitigate the uncertainties associated with system operation, a two-level model predictive control (MPC) integrating <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$Q$ </tex-math></inline-formula> -learning reinforcement learning model is designed to address different time-scale controllers. MPC algorithm allows making decisions for the day-ahead, based on predictions of uncertain parameters, whereas <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$Q$ </tex-math></inline-formula> -learning algorithm addresses real-time decisions based on real-time data. The problem is solved for various sets of houses. Results demonstrated that houses can gain more benefits when they are operating in the aggregate mode. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners:</i> Residential buildings besides commercial and public sectors are among the building sectors responsible for high-energy consumption. Numerous measures have been considered to construct more energy-efficient buildings, such as implementing new effective insulation materials and increasing the utilization ratio of sunlight. However, there is also a need for practical solutions to reduce the greenhouse gas emissions and avoid power peak from the residential sector. Under this situation, energy management system aggregator (EMSA) offers the opportunity to exploit the flexibility potential of various houses and other available distributed energy resources, promoting their participation in ancillary services and benefiting from rewards and lower energy bills. In this article, an innovative and comprehensive model predictive control-based scheduling optimization that considers uncertainties of renewable resources and weather conditions is formulated. It can be considered as a practical solution in order to optimally control the operation of a residential community. We proposed a curtailable demand response (DR), where customers agree to participate in DR programs defined by the EMSA in response to price changes.