Model-free control method based on reinforcement learning for building cooling water systems: Validation by measured data-based simulation

Abstract

In the domain of optimal control for building HVAC systems, the performance of model-based control has been widely investigated and validated. However, the performance of model-based control highly depends on an accurate system performance model and sufficient sensors, which are difficult to obtain for certain buildings. To tackle this problem, a model-free optimal control method based on reinforcement learning is proposed to control the building cooling water system. In the proposed method, the wet bulb temperature and system cooling load are taken as the states, the frequencies of fans and pumps are the actions, and the reward is the system COP (i.e., the comprehensive COP of chillers, cooling water pumps, and cooling towers). The proposed method is based on Q-learning. Validated with the measured data from a real central chilled water system, a three-month measured data-based simulation is conducted under the supervision of four types of controllers: basic controller, local feedback controller, model-based controller, and the proposed model-free controller. Compared with the basic controller, the model-free controller can conserve 11% of the system energy in the first applied cooling season, which is greater than that of the local feedback controller (7%) but less than that of the model-based controller (14%). Moreover, the energy saving rate of the model-free controller could reach 12% in the second applied cooling season, after which the energy saving rate gets stabilized. Although the energy conservation performance of the model-free controller is inferior to that of the model-based controller, the model-free controller requires less a priori knowledge and sensors, which makes it promising for application in buildings for which the lack of accurate system performance models or sensors is an obstacle. Moreover, the results suggest that for a central chilled water system with a designed peak cooling load close to 2000 kW, three months of learning during the cooling season is sufficient to develop a good model-free controller with an acceptable performance.