Economic operation of an agent-based virtual storage aggregated residential electric-heating loads in multiple electricity markets

Abstract

The increasing penetration of intermittent renewable energy sources leads to a higher demand for flexible balancing service in the power system. Consequently, demand-side management techniques, such as utilizing flexible resources from residential electric-heating loads, provide a promising solution. However, due to various uncertainties such as stochastic weather and market prices, it is challenging to develop an economically efficient strategy. Hence, this paper proposes an economic operation optimization approach for residential electric-heating loads to develop optimal bidding and scheduling strategy in the multi-stage market. First, the dynamic characteristics of aggregated electric-heating loads are modeled as the virtual energy storage systems (VESS) to quantify the flexibility potential arising from the combination of building thermal inertia, user tolerance and thermal energy storage (TES). Subsequently, the joint energy and balancing service market is introduced to exploit the operating flexibility of VESS. A multi-agent deep reinforcement learning (MADRL) optimization framework is proposed to address the multi-objective problem at different decision frequency, which involves achieving lower economic costs while maintaining indoor comfort. Finally, we combine the DRL with imitation learning (IL) by introducing the behavior cloning mechanism to mitigate training process instability. It reduces the computational costs and eliminates the need for redundant additional structure. Comparative simulation experiments demonstrate the efficiency of data-driven optimization approach, which provides an adaptive, economic cost-effective and general solution for multi-type aggregators to activate their flexibility in dynamic markets.