Climate plays an important role in heating energy consumption owing to the direct relationship between space heating and changes in meteorological conditions. To quantify the impact, the Transient System Simulation Program software was used to simulate the heating loads of office buildings in Harbin, Tianjin, and Shanghai, representing three major climate zones (i.e., severe cold, cold, and hot summer and cold winter climate zones) in China during 1961–2010. Stepwise multiple linear regression was performed to determine the key climatic parameters influencing heating energy consumption. The results showed that dry bulb temperature (DBT) is the dominant climatic parameter affecting building heating loads in all three climate zones across China during the heating period at daily, monthly, and yearly scales (R 2 ≥ 0.86). With the continuous warming climate in winter over the past 50 years, heating loads decreased by 14.2, 7.2, and 7.1 W/m2 in Harbin, Tianjin, and Shanghai, respectively, indicating that the decreasing rate is more apparent in severe cold climate zone. When the DBT increases by 1 °C, the heating loads decrease by 253.1 W/m2 in Harbin, 177.2 W/m2 in Tianjin, and 126.4 W/m2 in Shanghai. These results suggest that the heating energy consumption can be well predicted by the regression models at different temporal scales in different climate conditions owing to the high determination coefficients. In addition, a greater decrease in heating energy consumption in northern severe cold and cold climate zones may efficiently promote the energy saving in these areas with high energy consumption for heating. Particularly, the likely future increase in temperatures should be considered in improving building energy efficiency.