Heating, Ventilation, And Air Conditioning Control Research Articles

Towards various occupants with different thermal comfort requirements: A deep reinforcement learning approach combined with a dynamic PMV model for HVAC control in buildings

Reinforcement learning (RL) has great potential in achieving energy-efficient, comfortable and intelligent control of heating, ventilation and air conditioning (HVAC) systems. Although research on RL-based HVAC control has attracted increasing interest, current studies generally use simple building simulation as the environment to train agents, and the definition of thermal comfort is limited to a wide temperature range, which cannot meet the different thermal comfort requirements of various occupants. This study proposes a deep reinforcement learning (DRL) control framework based on the Dueling Deep Q-network (DQN) algorithm, combined with a self-designed environmental model and reward function, for HVAC control meeting different thermal comfort requirements. Specifically, based on the theory of building thermal dynamics, a nonlinear equation modified by experimental data is used for the environmental model that reflects the actual thermal change of building. Different thermal comfort requirements are considered and analysed through a dynamic predicted mean vote (PMV) model that focuses on the metabolic rate and clothing level of occupants. By systematically exploring different heating modes for occupants and control time intervals, the proposed framework demonstrates that heating energy consumption can be reduced by 4.8%-39.58% under various conditions compared to rule-based control. In addition, the study found that the HVAC control based on DRL has greater potential in saving energy when the heating demand of building is higher. Our study is helpful for researchers to make HVAC control more energy-efficient and user-friendly with the help of artificial intelligence.

A multi-objective energy management optimization for a hybrid electric bus covering an urban route

The development of electrified vehicles is a promising step toward energy savings, emissions reduction, environmental protection, and more sustainable economic growth. In the case of hybrid electric vehicles (HEVs), the energy management strategy (EMS) is essential for their efficiency and energy consumption. Typically, EMS employs rule-based strategies calibrated to general driving conditions. So, this paper proposes to calibrate the EMS of an urban hybrid electric bus that covers a particular route by taking advantage of past driving information. The EMS computes the percentage of the vehicle power demand that must be supplied by each of the sources (fuel and battery) and also controls the heating, ventilating and air conditioning (HVAC) system to achieve cabin thermal comfort. The proposed approach is based on employing an optimal solution by dynamic programing in a previous loop covered by the bus in the considered route. Then, the cost-to-go matrix is stored and used in the following trips by applying the one-step look-ahead rollout, taking profit from the similarities of the loops in the route. To compare and evaluate the performance of the proposed algorithm, a benchmark was carried out by employing the widespread equivalent consumption minimization strategy (ECMS) approach, combined with rule-based strategies in the HVAC control system. Finally, the pareto front presents the trade-off between cabin temperature control performance and total fuel consumption, allowing to compare and evaluate the different EMS calibrations.

Field test of machine-learning based mean radiant temperature estimation methods for thermal comfort-integrated air-conditioning control improvement and energy savings

Advanced control strategies for heating, ventilation, and air-conditioning (HVAC) systems aim to enhance both buildings' energy efficiency and occupant thermal comfort. Despite their potential, real-world demonstration studies on such advanced technologies are still lacking. This study presents a field demonstration of a real-time predicted mean vote (PMV)-based HVAC control strategy in a typical residential building under hot and dry climate conditions. This study introduces an advanced thermal comfort-based controller (TCC) for the PMV-based HVAC control. TCC continually assesses indoor and outdoor thermal conditions and adjusts the HVAC setpoint temperature to optimize real-time energy use while ensuring satisfactory indoor thermal comfort. Machine learning models for the PMV-based HVAC control system are employed to estimate a mean radiant temperature (MRT) point, which is one of the variables used to calculate real-time PMV values. Three machine learning models (i.e., linear regression, regression trees, and artificial neural network) are adopted in this study with non-stationary real-time input values, including times, pre-determined setpoints, and indoor and outdoor temperatures. The developed TCC is installed in a full-scale experimental house in Kuwait, which is under hot and dry climate conditions, to assess the house's indoor thermal comfort and AC energy efficiency performance. Results indicate that the TCC provides better thermal comfort performance compared to the non-TCC case, including up to a 60% improvement in PMV. The proposed TCC controlled by the machine learning methods demonstrates HVAC energy savings potential of over 20% while meeting the desired thermal comfort levels in the experimental building. These findings are expected to be valuable, as they can contribute to reducing cooling energy in residential buildings in hot and dry climate conditions.

Comparing the life cycle costs of a traditional and a smart HVAC control system for Australian office buildings

Many smart technologies have been introduced in buildings with the aim to reduce the energy and GHG emissions associated with their operation, particularly through improved control systems for regulating heating, ventilation and air conditioning (HVAC) equipment. Despite their energy saving potential, only a few studies have comprehensively assessed the costs associated with their practical implementation from a life cycle perspective. Accordingly, this study quantifies and compares the life cycle costs of a smart HVAC control system with that of a traditional control system, in the context of an Australian office building. For both systems, the required hardware are specified based on the characteristics of these systems and the layout of the serviced spaces in the reference building. The costs incurred over the period of assessment are quantified using the net present cost (NPC) approach. To evaluate the effects of these control systems on the operational energy costs of the building HVAC system, the control logics of both these systems are modelled through building energy simulations. The results show that, over the period of assessment, the smart control system incurred a higher total cost compared to the traditional control system. However, the findings from the simulations show that the HVAC energy cost savings achieved through the specification of the smart control system offset the additional cost incurred to deploy this system over the traditional control system. The smart control system resulted in HVAC operational cost savings between 9 % and 10 % compared to the traditional control system. Sensitivity analyses indicated that the total life cycle costs varied between −27 % and +50 %, with the discount rate and energy price increase rate being the most influential parameters.

Assessing the Impact of Climate Changes, Building Characteristics, and HVAC Control on Energy Requirements under a Mediterranean Climate

Assessing the Impact of Climate Changes, Building Characteristics, and HVAC Control on Energy Requirements under a Mediterranean Climate

Deep reinforcement learning for PID parameter tuning in greenhouse HVAC system energy Optimization: A TRNSYS-Python cosimulation approach

The control of indoor temperature in greenhouses is crucial as it directly impacts the crop's thermal comfort and the performance of heating, ventilation, and air-conditioning (HVAC) systems. Conventional feedback controllers, like on/off, can sometimes make HVAC system work at full capacity when only half that capacity is needed. In contrast, the proportional-integral-derivative (PID) controller, provides precise control based on its P, I, and D parameters. However, it lacks a formal design procedure for optimizing a specified objective function. Previous studies have utilized conventional PID tuning approaches to track room setpoint temperature for residential buildings, data centers, and office buildings, with limited research in greenhouse applications. To address this gap, this study proposes a flexible PID controller that employs a deep reinforcement learning (DRL) algorithm to optimize its parameters, by tracking the setpoints and energy consumption of a greenhouse planted with tomatoes. This approach is different from the typical method of using the trained RL agent directly in HVAC controls. Through a self-made TRNSYS-Python cosimulation framework, the DRL agent interacts directly and in real time with the greenhouse and its plants. Consequently, optimized PID parameters were established and tested in the simulated environment. The resulting performance, in terms of both energy consumption and its ability to maintain the crop’s comfort temperature, was compared with the simulated on/off and manually tuned PID controllers. Compared to the on/off baseline control, the proposed PID optimized parameters reduce energy use by 8.81% to 12.99% and the manually tuned PID parameters with the Ziegler-Nichols tuning method reduce energy use by 7.17 %. Additionally, the proposed method had a deviation of 2.07% to 3.13%, while the manually tuned PID controller and the on/off controller had deviations of 7.27% and 3.27%, respectively, from the minimum comfortable temperature. This study serves as a framework for improving the energy efficiency of greenhouse HVAC system operations.

A Systematic Review on the Use of AI for Energy Efficiency and Indoor Environmental Quality in Buildings

Global warming, climate change and the energy crisis are trending topics around the world, especially within the energy sector. The rising cost of energy, greenhouse gas (GHG) emissions and global temperatures stem from the over-reliance on fossil fuel as the major energy resource. These challenges have highlighted the need for alternative energy resources and urgent intervention strategies like energy consumption reduction and improving energy efficiency. The heating, ventilation, and air-conditioning (HVAC) system in a building accounts for about 70% of energy consumption, and a decision to reduce energy consumption may impact the indoor environmental quality (IEQ) of the building. It is important to adequately balance the tradeoff between IEQ and energy management. Artificial intelligence (AI)-based solutions are being explored for improving building energy performance without compromising IEQ. This paper systematically reviews recent studies on AI and machine learning (ML) for building energy management and IEQ by exploring common use areas, the methods or algorithms applied and the results obtained. The overall purpose of this research is to add to the existing body of work and to highlight energy-related AI applications in buildings and the related gaps. The result shows five common application areas: thermal comfort and indoor air quality (IAQ) control; energy management and energy consumption prediction; indoor temperature prediction; anomaly detection; and HVAC controls. Gaps involving policy, real-life scenario applications, and insufficient study of the visual and acoustic comfort areas are also identified. Very few studies take into consideration the need to follow IEQ standards in the selection process and positioning of sensors in AI applications for IEQ in buildings. This study reveals a need for more systematically summarized research.

A hardware-in-the-loop (HIL) testbed for cyber-physical energy systems in smart commercial buildings

In recent years, there has been a growing trend toward the development of smart buildings that rely on cyber-physical systems (CPS) to optimize occupant comfort, safety, and energy efficiency. To ensure the reliable and efficient operation of CPS with designed control strategies, it is important to evaluate their performance under various scenarios before deploying them in the real world. This is where a Hardware-in-the-loop (HIL) testbed designed for studying sensor and control-related studies in smart buildings can be highly valuable. With the growing threat of cyber-attacks and physical faults targeting smart buildings, it is essential to ensure the security of building operations. A HIL testbed can emulate cyber-attack and physical fault scenarios, allowing researchers to develop and test threat detection and mitigation algorithms. This enables researchers to identify potential issues and optimize the algorithms in a safe and controlled environment before they are deployed in real-world settings, reducing the risk of failures that can negatively impact occupant comfort, safety, and energy efficiency. Therefore, this paper developed a HIL testbed designed for cyber-physical energy systems (e.g. buildings automation system (BAS)) in smart commercial buildings. The HIL testbed is comprised of a real-time building and Heating, Ventilation, and Air-Conditioning (HVAC) emulator using Modelica-based dynamic models, a set of BAS controllers, and a BAS computer server. The data generation capability of the HIL testbed is demonstrated by tracking normal and faulty operating data in the BAS, as well as monitoring detailed network traffic in the local BAS network. This study further demonstrates the HIL testbed’s capability by conducting case studies on real-time physical fault and cyber-attack experiments using a Department of Energy (DOE) prototype commercial building. It is anticipated that the fully functional HIL testbed will be utilized for a variety of sensor and control-related studies, including but not limited to testing, developing, validating of different HVAC control strategies, fault detection & diagnosis, energy monitoring and analysis, cyber security study, etc.

Modelling building HVAC control strategies using a deep reinforcement learning approach

Heating, ventilation, and air-conditioning (HVAC) systems are responsible for a considerable proportion of total building energy consumption but are also vital for improved indoor temperature comfort, indoor air quality and well-being of building occupants. Thus, developing control strategies for HVAC systems is critical for the total life cycle of any building projects. Particularly, HVAC and building operations are not stationary but are filled with fuelled by environmental dynamisms and unexpected disruptions such as users' activities, weather conditions, occupancy rate, and operation of machinery and systems. This research aims to develop and propose a strategic control learning framework for HVAC systems using the deep reinforcement learning (DRL) approach. The results show that the proposed Phasic Policy Gradient (PPG) based method is more adaptive to changes in real building's environments. Notably, PPG performs better and more reliable than the conventional method for HVAC control optimization with about 2-14% in energy consumption reduction and indoor temperature comfort enhancement, along with a 66% faster convergence rate. Overall, our findings demonstrate that our proposed DRL approach is less resource intensive and much easier than the conventional approach in deriving solutions for HVAC control optimization driven by energy efficiency and indoor temperature comfort.

Real-world implementation of a cloud-based MPC for HVAC control in educational buildings

In their quest to enhance energy efficiency and carbon footprint management, many enterprises are turning their attention towards optimizing their facilities through amplified data monitoring and innovative control methods. In this context, this research highlights a practical solution in a commercial building located in the United States. It explains the creation, implementation, and costs of a universally applicable model predictive control (MPC) framework for the building’s heating, ventilation, and air conditioning (HVAC) system. The paper elaborates on both the hardware and software used to accumulate pertinent data and the formulation of the MPC system. This approach leverages cloud-based microservices and can be seamlessly integrated into current building management systems. In this regard, this paper presents three innovative strategies: Proportional–integral (PI)+Preheating, MPC, and occupancy-based control. These scenarios were designed to improve energy efficiency and thermal comfort significantly by reducing temperature fluctuations and adhering more closely to the setpoint temperature. The implementation of these strategies resulted in substantial energy savings, with reductions in energy consumption of 12.83%, 19.21%, and 14.98%, respectively.

Experimental study on the physiological parameters of occupants under different temperatures and prediction of their thermal comfort using machine learning algorithms

Real-time prediction of indoor thermal comfort has great potential in the development of a thermal comfort-driven control method for heating, ventilation, and air conditioning (HVAC) systems and the improvement of the indoor environment. This study measured physiological parameters (including electrocardiogram (ECG) signal, electroencephalogram (EEG) signal, and skin temperature (ST)) of 10 male and 10 female volunteers under three temperature conditions (22 °C, 25 °C, and 28 °C). A setup database containing 3600 sets of data was used to develop the thermal comfort prediction model with three machine learning algorithms: random forest (RF), back-propagation neural network (BP), and convolutional neural network (CNN). Significant differences were found in both the values and the impact indices of physiological indicators between males and females (p < 0.05). The RF algorithm had a significant advantage with an overall accuracy of 89.3 %. If only one factor is employed for machine learning training, using the RF algorithm and training with the ST dataset is the most appropriate method, with an accuracy of 81.8 %. The addition of all datasets can increase the modeling accuracy to 95.6 %. Taking into account the sample imbalance, the SMOTETomek method was applied. After applying the SMOTETomek method to balance the data, the model accuracy reached at 97.3 %. The comprehensive results showed that thermal comfort prediction for the mixed group of males and females by the RF algorithm using one hybrid model was feasible. The establishment of this model laid the foundation for thermal comfort-driven HVAC control methods and the improvement of indoor environments.

Occupant-centric HVAC and window control: A reinforcement learning model for enhancing indoor thermal comfort and energy efficiency

Occupant behavior plays a crucial role in enhancing indoor thermal comfort and achieving energy efficiency by influencing the operational modes of Heating, Ventilation, and Air Conditioning (HVAC) systems as well as windows. However, accurately quantifying the impact of occupant behavior on the indoor environment presents significant challenges in practical applications. This study introduces an innovative approach by leveraging the ASHRAE Global Building Occupant Behavior Database and harnessing the power of XGBoost in conjunction with Deep Q Networks (DQN) to construct a reinforcement learning model. This model enables precise prediction of the impact of occupant behavior on the indoor environment at the next time step under varying indoor-outdoor conditions, simultaneously targeting the dual objectives of indoor thermal comfort and energy conservation. By applying the XGB-DQN model in sample rooms of four international cities with distinct features, the results demonstrate a significant increase in indoor thermal comfort duration by 24 %, accompanied by a 24.7 % decrease in air conditioning usage compared to baseline models and actual occupant data. This research represents a pioneering effort in applying reinforcement learning techniques to accurately predict occupant behavior's impact on indoor environments, offering valuable insights for intelligent building design and energy management.

Demand Response and HVAC Controls in Smart Grid Integration

This extensive research paper meticulously investigates the intricate synergies between demand response strategies and Heating, Ventilation, and Air Conditioning (HVAC) controls within the dynamic realm of smart grid integration. As the global energy landscape undergoes transformative changes driven by technological advancements and sustainability imperatives, the study critically examines the symbiotic relationship between consumers, HVAC systems, and the smart grid. The paramount importance of this interaction is underscored, emphasizing its pivotal role in achieving elevated levels of energy efficiency, grid reliability, and sustainability. The research explores the multifaceted potential inherent in demand response programs and advanced HVAC controls, unraveling their collective capacity to reshape the energy ecosystem. It goes beyond the theoretical framework by delving into real-world applications, case studies, and cutting-edge technological advancements. By doing so, the study seeks to provide not only a theoretical understanding but also practical insights into the dynamic and evolving landscape of demand-side management. This investigation is poised to contribute significantly to the existing body of knowledge by illustrating how the harmonious coordination of consumer behaviors and HVAC systems can be strategically harnessed. This strategic collaboration aims to optimize energy consumption, mitigate peak loads, and contribute substantively to the establishment of a more sustainable, adaptive, and responsive energy infrastructure. As the global community grapples with the imperative to transition towards cleaner and more efficient energy systems, this research positions itself as a valuable resource for policymakers, researchers, and industry stakeholders seeking innovative solutions for a resilient and sustainable energy future.

Life cycle energy and greenhouse gas emissions of a traditional and a smart HVAC control system for Australian office buildings

In recent years, many novel smart technologies have been proposed to reduce the energy consumption and greenhouse gas (GHG) emissions attributed to the building sector. One such application of these technologies is the reduction of the Heating, Ventilation and Air Conditioning (HVAC) energy needs and GHG emissions using smart control systems. The environmental benefits of smart control systems during building operation have been explored in many studies, however, their embodied effects, associated with the extraction of raw materials, manufacturing and replacement are often overlooked. Accordingly, this study quantifies the life cycle energy needs and GHG emissions of a smart HVAC control system and assesses its potential for reducing the HVAC operational energy and GHG emissions in an Australian office building. A comparative assessment is performed, considering a traditional HVAC control system and an equivalent smart HVAC control system. The components of both the traditional and smart control systems are specified based on the characteristics of these systems as well as the layout of the serviced spaces in the reference building. The embodied energy and GHG emissions of both the traditional and smart control systems are quantified through a hybrid life cycle inventory (LCI) approach. To evaluate the effects of these control systems on the building HVAC operational energy, a building energy simulation is performed by applying the control logics of both systems. The results show that energy and GHG emissions savings in the HVAC operational offset the additional energy needs and GHG emissions of deploying the smart HVAC control system.

Development of an HVAC system control method using weather forecasting data with deep reinforcement learning algorithms

Heating, ventilation, and air conditioning (HVAC) systems account for a significant proportion of the energy consumption of a building. With the global energy demand increasing every year, optimal HVAC control methods that reduce energy consumption while maintaining the thermal comfort of the occupants have become more important. Weather forecasting data have thus attracted significant research attention for the optimization of building energy consumption in model-based studies. However, these HVAC control approaches have failed to consider the reduction in the predictive error of the incorporated weather model, leading to performance limitations. Accordingly, model-free reinforcement learning (RL)-based HVAC systems that can be controlled dynamically have been proposed, and these have exhibited significantly better performance than model-based methods. However, RL algorithms using weather forecasting data have not yet been assessed for use in optimal HVAC control for buildings. In this study, we propose WDQN-temPER, a hybrid HVAC control method combining a deep Q-network with a novel prioritized experience replay (PER) technique (referred to as temPER) and a gated recurrent unit (GRU) model. The GRU model predicts the future outdoor temperature and this is used as a state variable in the RL model. The proposed temPER algorithm facilitates the use of samples in the RL training process with larger changes in the outdoor temperature based on the predicted temperature. We experimentally demonstrate in EnergyPlus simulations that our proposed WDQN-temPER model outperforms a rule-based baseline model in terms of HVAC control, with energy savings of up to 58.79%.

Assessment methodology for dynamic occupancy adaptive HVAC control in subway stations integrating passenger flow simulation into building energy modeling

Heating, ventilation, and air conditioning (HVAC) systems in subway stations are often pre-scheduled to non-discriminatorily control large areas, ignoring uneven and dynamic thermal demand. Although building energy modeling (BEM) contributes to optimizing HVAC design and operation, it is limited in investigating dynamic occupancy adaptive HVAC control strategies in large open subway stations. This research formalized an integrated thermal control methodology (ITCM) that integrates passenger flow simulation and BEM to assess the dynamic occupancy adaptive HVAC control strategies for subway stations. To evaluate the applicability and effectiveness of ITCM, we developed a prototype coupling AnyLogic and EnergyPlus and carried out a simulation-based case study using a Hong Kong subway station. Based on the case study, we identified that ITCM enables subway station development and management teams to assess the comprehensive dynamic occupancy adaptive HVAC control strategies (e.g., decomposed thermal zones considering the layouts of diffusers and variable air volume boxes) in a timely and consistent manner. In addition, the case study results show that adaptive control with ceiling diffusers could save between 1.8% and 24.4% of cooling energy without considerably sacrificing thermal comfort.

Multi-indicator adaptive HVAC control system for low-energy indoor air quality management of heritage building preservation

Regenerated commercial heritage buildings employ heating, ventilation, and air conditioning (HVAC) systems that significantly increase their energy consumption, necessitating a methodology that supports low-energy operation to achieve a sustainable built environment. Poor indoor air quality (IAQ) management can cause irreversible damage to heritage buildings. However, there are risks of the destruction of inherent heritage values if energy-efficiency approaches are implemented without considering the visual impacts and multiple IAQ parameters in heritage buildings. To preserve heritage buildings, this study developed a multi-indicator adaptive ventilation control system for IAQ management using digital twin technology, which consisted of triggers and feedback. A digital representation of heritage buildings was established using Heritage Building Information Modelling (HBIM) with sensors to trigger adjustments in ventilation system settings. The sensor placement rules for IAQ monitoring and HVAC control of heritage buildings were demonstrated using computational fluid dynamics (CFD) simulations. The relationships among the HVAC inlet velocity, multiple IAQ parameters, and energy consumption were quantified using CFD and energy simulations. Simulation data were used to generate responsive charts for adaptive ventilation control, and the sensor data provided feedback to the ventilation system. The optimal ventilation system control strategy achieved up to 30% energy savings in the illustrative example. The proposed multi-indicator adaptive HVAC control system contributes significantly to the timely reduction of multiple air pollutants, IAQ parameter adjustments for the preservation of heritage buildings with minimal structural and visual impacts, and the need for more autonomous and energy-efficient HVAC systems.

Transfer learning for occupancy-based HVAC control: A data-driven approach using unsupervised learning of occupancy profiles and deep reinforcement learning

Model-free heating, ventilation, and air conditioning (HVAC) control systems have demonstrated promising potential for adjusting indoor setpoint temperature based on dynamic occupancy patterns in smart buildings. Although these control systems offer the advantage of not needing building or occupancy models, the involved trial-and-error learning process can cause considerable thermal discomfort for occupants, particularly during the initial learning period. Given the critical importance of thermal comfort, this limitation is a major barrier to the practical implementation of such systems. To address this challenge, the present study proposes a framework to enhance the learning process of the model-free HVAC controllers. Specifically, a transfer learning (TL) technique is adopted based on a similarity analysis of occupancy patterns using an unsupervised learning of occupancy profiles. This control framework leverages a k-means clustering algorithm with dynamic time warping to match the most similar households in terms of occupancy patterns within 26 residential units. The results demonstrate that the proposed method significantly improves the performance of the HVAC control system. It enhances the jumpstart performance and total rewards by nearly 25% and 5%, respectively, compared to a conventional model-free controller. Furthermore, it reduces the deviation period and mean temperature deviation by approximately 4% and 68%, respectively. Overall, this framework presents a promising approach to enhancing the performance and practicality of model-free HVAC control systems by reducing the thermal discomfort during the learning process.

An online reinforcement learning approach for HVAC control

Heating, Ventilation and Air Conditioning (HVAC) optimization for energy consumption reduction is becoming ever more a topic of the utmost environmental and energetic concerns. The two most employed methodologies for optimizing HVAC systems are Model Predictive Control (MPC) and Reinforcement Learning (RL). This paper compares three different RL approaches to HVAC optimization: one based on a black-box system identification model trained on historical data, one based on a white-box model of a building and one online method based on an imitation learning pretraining phase on historical data. The three approaches are compared with a literature baseline and an EnergyPlus baseline. Results show that the overall best method in terms of energy consumption reduction (65% decrease) and thermal comfort increase (25% increase) is the approach based on the white-box model. However, the proposed methodology, based on online and imitation learning, demonstrates remarkable efficiency, achieving comparable improvements in energy consumption after just a few months of online training, while maintaining thermal comfort at around the same level as the baseline. These results prove a direct online RL approach, which avoid the use of costly simulations, can provide a reliable and inexpensive solution to the problem of HVAC optimization.

MF^2: Model-free reinforcement learning for modeling-free building HVAC control with data-driven environment construction in a residential building

Reinforcement Learning (RL) has advanced energy-efficient control of building Heating, Ventilation and Air Conditioning (HVAC) systems. Constructing a suitable RL environment for buildings is a crucial challenge. Compared to widely-used simulation-based environments, data-driven approaches offer higher training efficiency but face convergence difficulties due to influential factors, limiting their current application.To explore data-driven construction of RL environments for building HVAC systems, this study proposes two strategies for controlling room temperature setpoints in a residential building. XGBoost and Long Short-Term Memory Network (LSTM) are trained for energy consumption and room temperature prediction. One strategy predicts parameters for on-off states, while the other for power-on states. The XGBoost models are integrated into an OpenAI Gym environment. The first strategy achieves 0.8634 R2 and 0.2423 Root Mean Squared Error (RMSE) for energy consumption prediction. The R2 of room air temperature models are approximately 0.99 and the RMSE are lower than 0.31. The second strategy achieves 0.9181 R2 and 0.1042 RMSE for energy consumption prediction and similar performance for room temperature prediction. Deep Q-learning (DQN) and Deep Deterministic Policy Gradient (DDPG) algorithms are separately trained using these environments. Results show that the first strategy fails to induce the correct training of RL models, while the second strategy successfully induces a useable DDPG model for controlling building HVAC systems but fails to induce a useable DQN model. We analyze the reasons behind these observations. Compared to the original room temperature setpoint method, the DDPG-based HVAC control logic achieves a 10.06% energy-saving effect while ensuring comfort.