This paper introduces Dyna-PINN, a novel physics-informed Deep Dyna-Q (DDQ) reinforcement learning (RL) approach, designed to address the data-intensive training requirements and model-agnostic nature of the conventional model-free RL methods. The DDQ approach blends model-based and model-free elements to enhance both learning and decision-making processes. By utilizing a physics-informed neural network (PINN) based model, our method enriches the learning process with physical information, enhancing the agent's planning capabilities and leading to faster learning compared to conventional model-free RL methods like Deep Q-Network (DQN) in scenarios with low-diversity training data availability. Our results demonstrate that Dyna-PINN has 50% greater sample efficiency than DQN and outperforms rule-based control in terms of thermal discomfort. Due to physics incorporation, the Dyna-PINN implements a more logical and interpretable control policy. It shows consistently good performance compared to all control variants across low-diversity data scenarios, i.e., 6 weeks of building data, and in higher-diversity data regimes, i.e., 6 months of building energy data, demonstrating the value of physics incorporation into the RL training. Additionally, we present two other DDQ-based techniques, RC-DDQ and NN-DDQ, exploring the synergy between neural networks and physical data in intelligent control designs for building energy systems. Rigorous controller testing is performed using the Building Optimization and Testing Framework (BOPTEST), a high-fidelity simulator that closely represents a real building's operation. Through comprehensive comparisons and realistic simulations, our study underscores the effectiveness of incorporating physics-informed approaches into RL-based control strategies, paving the way for more efficient and robust building energy management systems.
Read full abstract