Comparing hyperparameter tuning methods in machine learning based urban building energy modeling: A study in Chicago

Abstract

In recent years, many studies have applied machine learning (ML) in urban building energy models (UBEMs) to understand building energy issues in cities. However, previous studies had limited in-depth discussions on hyperparameter tuning, an essential aspect of model training. To address this gap, this study examines the impact of different hyperparameter tuning strategies on the development of ML-based UBEMs, specifically for residential buildings in Chicago. Utilizing a comprehensive dataset that encompasses building, urban, household, and energy management factors, and annual electricity usage across 2,078 residential block groups in Chicago for the year 2010, this study employs five widely used supervised machine learning (SML) algorithms. Through three experiments, the study found that gradient boosting decision trees models generally outperform others. However, hyperparameter tuning can significantly enhance the performance of all tested SML models. The comparison of three search methods, grid search, random search, and Bayesian search, revealed that they have similar tuning performance, but random search stands out for its effectiveness, speed, and flexibility. Moreover, the study demonstrates that the suggested search space for hyperparameters yields better results compared to the commonly used search space in most cases. Interestingly, altering the search budget appeared to have minimal impact on the tuning effectiveness. Employing repeated cross-validation methods helped mitigate the issues associated with random data splits in the training process, leading to more reliable tuning outcomes. Further investigations into the scalability of training datasets and the evaluation of models using alternative accuracy metrics and considerations of model inductive bias provided additional insights. These findings informed recommendations regarding hyperparameter tuning strategies, optimal sample sizes, and model evaluation techniques. Overall, this research contributes to a deeper understanding of the role of hyperparameter tuning in the development of ML-based UBEMs, offering valuable insights for energy-efficient urban planning and policymaking toward sustainable city development.