An efficient and privacy-preserving algorithm for multiple energy hubs scheduling with federated and matching deep reinforcement learning

Abstract

As a significant paradigm change in reinforcement learning, federated learning (FL) has emerged to address the efficiency bottlenecks and privacy concerns of centralized training. However, the discrepancy of multi-energy demands among energy hubs (EHs) may cause undesirable instability of FL mechanisms. This study proposes a scheduling algorithm for adaptively controlling multiple EHs with combined heat and power, gas boiler, and electrical boiler. We first present the formulations of twin delayed deep deterministic policy gradient (TD3) to schedule the energy conversion under the time-varying price and demands of electricity, heat, and natural gas. The novelty of the proposed algorithm lies in developing a distributed TD3-agents training that consists of FL and matching-based agent-to-agent learning, namely matching learning (ML). Specifically, the FL is first introduced for collaborative training of EH's TD3 agents by sharing their semi-trained scheduling models. The ML innovatively proposed for addressing the training instability of FL is formulated as a matching game in which the poor-trained agents learn the scheduling knowledge from the agent who has analogous demand patterns and better reward. Simulation results display the advantage and feasibility of the proposed algorithm in terms of the speed of training convergence, energy consumption of equipment, and economic benefit.